Tools and Scripts#

This page documents the repository-level scripts that create and inspect datasets in the current format. They live in scripts/ of the spora-io repository and are not part of the installed package, so run them from the repository root. The dataset-creation scripts are listed in pipeline order; Convert New Datasets explains how they fit together.

Before running these tools, point spora_io to the datasets root:

export SPORA_DATASETS_DIR=/path/to/datasets

compute_tissue_masks and convert_dataset_to_omezarr take --dataset_name. compute_standardization_stats and compute_tiling accept both --dataset-name and --dataset_name, and compute_tiling also accepts --tile_size for --tile-size.

Compute Tissue Masks#

scripts/compute_tissue_masks.py writes a boolean mask for every image of every resolution found in the dataset:

<dataset>/segmentations/<resolution>/tissue_masks/<tissue_id>.npz

python -m scripts.compute_tissue_masks --dataset_name my_dataset

The script reads both .zarr and .ome.zarr stores and picks the method by modality:

  • H&E: the HEST tissue-segmentation model (DeepLabV3-ResNet50, MahmoodLab/hest-tissue-seg on Hugging Face), applied in overlapping 4096-pixel windows, followed by dilation and hole filling.

  • IMC, CODEX, CyCIF: Otsu thresholding of a log-compressed maximum projection over the QC-passing channels, followed by morphological opening and closing.

H&E directories are processed first and existing masks are never overwritten, so paired modalities of a tissue share the H&E-derived mask, and reviewed masks saved beforehand are kept. ihc and mibi directories are currently not processed.

The script loads the segmentation model on CUDA when it starts, including for multiplex-only datasets. It requires PyTorch, torchvision, SciPy, and scikit-image; downloading the checkpoint requires huggingface_hub.

Convert to OME-Zarr#

scripts/convert_dataset_to_omezarr.py turns the plain .zarr stores written during conversion into the OME-Zarr images read by the loaders:

python -m scripts.convert_dataset_to_omezarr --dataset_name my_dataset

For each <tissue_id>.zarr below <modality>/<resolution>/images/ of the he, ihc/ihc_*, imc, codex, cycif, and mibi directories, it

  • moves a root-level array to the array path 0 and makes the store an OME-Zarr 0.5 image group with c, y, x axes;

  • transposes channel-last RGB arrays to channel-first;

  • writes omero channel labels: R, G, B for H&E and IHC, and for multiplex images the names of the channels marked available for that tissue in channels_per_tissue.parquet;

  • renames the store to <tissue_id>.ome.zarr.

Errors are reported per image and the remaining images are still converted; check the output for failures and rerun after fixing them. H&E and IHC images that already have an .ome.zarr counterpart are skipped. The converter does not resample; the recorded scale metadata is not used by the loaders.

Utility Functions for Dataset Creation#

scripts/dataset_creation_utils/ contains the functions used by conversion scripts, such as create_folderstructure, save_np_as_zarr, save_rgb_as_zarr, hash_patient, and save_np_as_npz. They are summarized in Convert New Datasets; examples/create_hoefflin2026spatial.py shows them in a complete conversion script.

Compute Multiplex Standardization Statistics#

scripts/compute_standardization_stats.py computes parquet-backed standardization statistics for multiplex modalities. Outputs are written under:

<dataset>/<modality>/<resolution>/standardization/<method>/uq_<upper_quantile>_<quantile_level>/

For example, --method quantile_clipping --upper-quantile 0.99 --quantile-level image writes to:

<dataset>/<modality>/1_0mpp/standardization/quantile_clipping/uq_0.99_image/

The suffix encodes the quantile level, not the mean/std level. This is intentional: means and standard deviations are computed after the selected quantile clipping transform, so uq_0.99_image and uq_0.99_global can have different *_means.parquet and *_stds.parquet values even when --stats-level is the same.

Minimal command:

python -m scripts.compute_standardization_stats \
    --dataset-name schurch2020coordinated \
    --modality codex \
    --method quantile_clipping \
    --quantile-level image \
    --stats-level global \
    --upper-quantile 0.99 \
    --resolution 1.0

Log-compressed standardization:

python -m scripts.compute_standardization_stats \
    --dataset-name schurch2020coordinated \
    --modality codex \
    --method quantile_clipping_log1p \
    --quantile-level image \
    --stats-level global \
    --upper-quantile 0.99 \
    --resolution 1.0 \
    --overwrite

Important arguments:

  • --dataset-name: dataset folder under SPORA_DATASETS_DIR.

  • --modality: one of codex, cycif, imc, or mibi.

  • --method: quantile_clipping or quantile_clipping_log1p.

  • --quantile-level: image or global for quantile computation. This value is encoded in the output directory and in the standardization value. For example, --quantile-level image writes uq_0.99_image while --quantile-level global writes uq_0.99_global.

  • --stats-level: image or global for mean/std computation.

  • --upper-quantile: upper clipping quantile, usually 0.99.

  • --lower-quantile: optional lower clipping quantile.

  • --resolution: image resolution in microns per pixel.

  • --overwrite: recompute even if all expected files already exist.

Settings for Model Families#

Different models expect different preprocessing. The table below lists the settings used by the spora models. For each method you need, compute the statistics with both --stats-level global and --stats-level image.

Settings used by the spora models#

Model

--method

--quantile-level

standardization

VirTues

quantile_clipping_log1p

image

quantile_clipping_log1p/uq_0.99_image

VirTues-M2

quantile_clipping

image

quantile_clipping/uq_0.99_image

The spora-bench dataset configurations use the same values in their standardization field.

Compute Tiling#

scripts/compute_tiling.py computes tile coordinates from shared tissue masks and writes parquet outputs under:

<dataset>/tiling/<resolution>/<tiling_method>/<tile_size>_tile_coordinates.parquet
<dataset>/tiling/<resolution>/<tiling_method>/<tile_size>_tile_stats.parquet

Minimal command:

python -m scripts.compute_tiling \
    --dataset-name schurch2020coordinated \
    --tile-size 224 \
    --resolution 1.0

Common production-style command:

python -m scripts.compute_tiling \
    --dataset-name schurch2020coordinated \
    --tile-size 224 \
    --resolution 1.0 \
    --tiling-method default \
    --stride 224 \
    --tolerance 0.95 \
    --coverage-goal 0.95 \
   --min-gain-ratio 0.1 \
   --overwrite

Fixed-grid tiling is also available. It lays a regular grid over the padded mask extent and keeps tiles whose tissue fraction is at least 1 - tolerance. Because padded grid tiles can extend past the image edge, grid outputs must be saved under a non-default strategy name; default is reserved for the fast no-padding tile-loading path.

python -m scripts.compute_tiling \
    --dataset-name schurch2020coordinated \
    --tile-size 224 \
    --resolution 1.0 \
    --tiling-method grid_stride224 \
    --grid \
    --stride 224 \
    --tolerance 0.85 \
    --overwrite

This writes:

<dataset>/tiling/1_0mpp/grid_stride224/224_tile_coordinates.parquet
<dataset>/tiling/1_0mpp/grid_stride224/224_tile_stats.parquet

Important arguments:

  • --dataset-name: dataset folder under SPORA_DATASETS_DIR.

  • --tile-size: square tile size in pixels.

  • --resolution: mask resolution in microns per pixel; 1.0 maps to 1_0mpp.

  • --tiling-method: output subdirectory under tiling/<resolution>/. When --grid is used this must not be default.

  • --grid: use padded fixed-grid tiling instead of adaptive greedy tiling. Edge grid tiles that cross the image boundary are represented by their original top-left row/col and are padded with zeros during dataset loading.

  • --stride: candidate lattice stride in pixels. If omitted, defaults to tile_size // 2 for adaptive tiling and tile_size for grid tiling.

  • --tolerance: maximum invalid/background fraction allowed inside a tile. For grid tiling, this means a tile is kept when its tissue fraction is at least 1 - tolerance.

  • --coverage-goal: target foreground coverage before adaptive stopping can trigger.

  • --min-gain-ratio: minimum marginal new foreground area required after coverage_goal has been reached.

  • --overwrite: recompute even if output parquet files already exist.

Channel TUI#

scripts/marker_viz.py opens a terminal UI for browsing multiplex channel metadata across datasets. It reads channels.parquet files for multiplex modalities, shows marker names and UniProt IDs, and highlights likely nuclear channels such as DAPI, Hoechst, Iridium, DNA stains, and Histone H3.

Install the UI dependencies if needed:

pip install textual rich pandas pyarrow

Run:

PYTHONPATH=. python scripts/marker_viz.py

Use this when checking marker naming consistency, UniProt coverage, quality-control flags, or nuclear marker annotations across cohorts.

Dataset Inventory TUI#

scripts/datasets_viz.py opens a terminal UI for dataset-level inspection. It summarizes available modalities, metadata rows, image counts, tile counts by tile size, available resolutions, and the standardization statistics of multiplex modalities.

Install dependencies:

pip install textual rich pandas pyarrow

Run:

PYTHONPATH=. python scripts/datasets_viz.py

Use this after curation or migration to quickly inspect whether datasets expose the expected modalities, tiling files, and standardization outputs.

Tile Viewer#

scripts/tile_viz.py starts a small local web application (FastAPI) that draws the tiles of one dataset, tile size, and modality on the image; for multiplex modalities, up to five markers can be selected. Install fastapi and uvicorn, run for example

PYTHONPATH=. python scripts/tile_viz.py my_dataset 224 he --mask-only

and open http://localhost:8000 in a browser.