Convert New Datasets#

This guide converts a spatial proteomics or histopathology imaging dataset from any file format into the spora [data] format. Every image is first written as a channel-first .zarr array with a small dataset-specific script, using the conversion helpers of spora-io. Dedicated scripts then generate the remaining components: tissue masks, the OME-Zarr stores, standardization statistics, and tile coordinates.

Step	Output	Tool
1	`.zarr` images, tissue metadata, and channel metadata	Your own conversion script
2	Tissue masks	`scripts/compute_tissue_masks.py`
3	`.ome.zarr` images with channel labels	`scripts/convert_dataset_to_omezarr.py`
4	Standardization statistics (spatial proteomics only)	`scripts/compute_standardization_stats.py`
5	Tile coordinates	`scripts/compute_tiling.py`
6	(Optional) A checked dataset	The spora-io dataset loaders

Each step first explains what to do and then shows the commands that created hoefflin2026spatial, a public dataset of CODEX images with H&E sections registered to the DAPI channel. Use it as a template for your own dataset.

Before You Begin#

Install spora-io as described in Installation and work from a clone of the repository: the conversion helpers and scripts live in scripts/ and are not part of the installed package. Run all commands from the repository root and point SPORA_DATASETS_DIR to the directory that will contain your dataset. Use pandas 2.2 or newer.

The conversion helpers import scikit-image. The H&E helpers and the tissue-mask script also import PyTorch, torchvision, and SciPy, and the tissue-mask script loads its H&E segmentation model on a CUDA device at start-up, also for datasets without H&E images.

Example. Download the raw hoefflin2026spatial data:

Download the images from the BioImage Archive study S-BIAD2840 and rename its Files folder to examples/hoefflin2026spatial_raw.
Download the sample metadata from Zenodo record 18378034, export the spreadsheet to CSV, and save it as examples/hoefflin2026spatial_raw/sample_metadata.csv.

The conversion script of this example additionally imports OpenCV (cv2) and tifffile.

export SPORA_DATASETS_DIR=/path/to/datasets

1. Convert the Dataset Images#

Decoding the source files is dataset-specific, so step 1 is a script you write yourself. Use the helpers in scripts/dataset_creation_utils/ wherever possible, for example to create the folder structure, write NumPy arrays as .zarr, and pseudonymize patient IDs. The script writes three things; Dataset Format describes each file in detail.

Images. One (C, H, W) array per tissue and resolution, saved as <modality>/<mpp>/images/<tissue_id>.zarr, for example codex/1_0mpp/images/<tissue_id>.zarr. H&E and IHC images have three RGB channels on the 0–255 scale. Multiplex images contain only the channels measured for that tissue, in the order of the cohort panel. The helpers store the array as given; check axis order, plane selection, and pixel spacing against the acquisition metadata, and resample to the target resolution before writing.

Tissue metadata. metadata/tissues.parquet with one row per tissue and modality, and at least the columns tissue_id, patient_id, and modality. Keep tissue_id as a regular column, not only as the index. Images of different modalities share a tissue_id only when they show the same tissue; for paired tiles, register them to the same pixel grid before writing.

Channel metadata (multiplex only). <modality>/channels.parquet lists the cohort panel in order: the channel names with their UniProt IDs and, optionally, QC and nuclear-marker flags. <modality>/channels_per_tissue.parquet records which channels are present in each image as a boolean matrix, with tissue IDs as the index and the channel names as columns, in panel order.

Conversion helpers in `scripts/dataset_creation_utils`#
Helper	Purpose
`utils_mx.create_folderstructure`	Creates `metadata/`, `<modality>/<mpp>/`, and `segmentations/<mpp>/tissue_masks/` under `SPORA_DATASETS_DIR` from a `{modality: [mpp, ...]}` mapping.
`utils_mx.save_np_as_zarr`	Writes a `(C, H, W)` NumPy array as a sharded Zarr v3 array (256 px chunks, 1024 px shards). Overwrites existing stores.
`utils_he.save_rgb_as_zarr`	Writes an RGB image in the same layout, moving a trailing channel axis to the front.
`utils_mx.hash_patient`	Maps a source patient identifier to a stable eight-letter pseudonym.
`utils_mx.save_np_as_npz`	Saves a boolean mask as `np.savez_compressed(path, mask=...)`.

Example. examples/create_hoefflin2026spatial.py converts the raw data:

PYTHONPATH=. python examples/create_hoefflin2026spatial.py

For every CODEX OME-TIFF, the script checks that all files share the same 70-channel panel, derives a pseudonymous patient_id and a tissue ID of the form hoefflin2026spatial_<patient>_<specimen>, and writes the CODEX stack and the registered H&E image as .zarr at the native 0.5 mpp and, after 2× downsampling, at 1.0 mpp, together with tissue masks of the H&E image. It then writes codex/channels.parquet with UniProt IDs and DAPI as the nuclear marker, metadata/tissues.parquet with one codex and one he row per tissue, and codex/channels_per_tissue.parquet, in which every channel is available for every tissue.

2. Create Tissue Masks#

compute_tissue_masks.py segments the tissue from the background and saves one boolean mask per image and resolution under segmentations/<mpp>/tissue_masks/. It picks the method by modality: the HEST segmentation model for H&E images, and Otsu thresholding of the QC-passing channels for spatial proteomics images. H&E images are processed first and existing masks are never overwritten, so paired modalities of a tissue share the H&E-derived mask, and masks you saved yourself are kept. IHC and MIBI directories are not processed; save their masks with utils_mx.save_np_as_npz.

Example. The script also saved the H&E masks in step 1, so only the remaining tissues are segmented:

python -m scripts.compute_tissue_masks --dataset_name hoefflin2026spatial

3. Convert the Images to OME-Zarr#

convert_dataset_to_omezarr.py adds the channel names to the image metadata and saves each image as an OME-Zarr 0.5 store on Zarr v3, renamed from <tissue_id>.zarr to <tissue_id>.ome.zarr. The multiplex channel names come from channels_per_tissue.parquet, so this step requires that step 1 wrote the images and the availability matrix correctly: an image whose number of planes does not match its available channels is reported and left unconverted.

Example:

python -m scripts.convert_dataset_to_omezarr --dataset_name hoefflin2026spatial

4. Compute Standardization Statistics#

Spatial proteomics images need standardization statistics; H&E and IHC images do not. Pass the modality and the standardization method to compute_standardization_stats.py and compute the statistics twice: once at the global level (--stats-level global) and once at the image level (--stats-level image). Different models need different methods and quantiles; Standardization lists the settings of each model family.

Example. The statistics for VirTues:

python -m scripts.compute_standardization_stats --dataset-name hoefflin2026spatial \
  --modality codex --method quantile_clipping_log1p --quantile-level image \
  --stats-level global --resolution 1.0
python -m scripts.compute_standardization_stats --dataset-name hoefflin2026spatial \
  --modality codex --method quantile_clipping_log1p --quantile-level image \
  --stats-level image --resolution 1.0

For VirTues-M2, run the same two commands with --method quantile_clipping.

5. Generate Tile Coordinates#

compute_tiling.py computes tile coordinates from the tissue masks with a tiling algorithm optimized for maximally diverse tissue coverage. Run it for every tile size your models use; the coordinates are written to tiling/<mpp>/default/<tile_size>_tile_coordinates.parquet. Tools and Scripts describes the options, including fixed-grid tiling.

Example:

python -m scripts.compute_tiling --dataset-name hoefflin2026spatial \
  --tile-size 224 --resolution 1.0

6. Check the Dataset (Optional)#

Load a few tissues with the spora-io loaders, for example in a notebook. Load them with kind="complete" and preprocess=False, so that no channel is filtered and no intensity is transformed, and check that the channel mapping is correct: plot several channels and compare them with the source files. Structural checks cannot detect a mislabelled channel plane.

Example:

from spora_io import ComposedImagingDataset

ds = ComposedImagingDataset(
    name="hoefflin2026spatial", modalities=["he", "codex"],
    resolution=1.0, tile_size=224,
    modality_kwargs={"codex": {"standardization": "quantile_clipping_log1p/uq_0.99_image"}},
)
tissue_id = ds.get_tissue_ids()[0]
codex = ds.get_unimodal_tissue(tissue_id, modality="codex", kind="complete", preprocess=False)
print(codex.image.shape, list(codex.channel_names)[:5])

Common problems#
Symptom	Check
`KeyError: tissue_id`	Save tissue IDs as a regular metadata column, not only as the Parquet index.
Unexpected `include_groups` argument	Use pandas 2.2 or newer.
Channel-count mismatch	Check the panel order, the column order of the availability matrix, and the planes actually stored.
Empty multiplex image after filtering	Inspect the QC flags and UniProt IDs; first load with `kind="complete"`.
Image or tile files not found	Check the dataset root, modality directory, resolution, tile size, and tiling strategy.
Missing or incompatible statistics	Match the loader’s standardization to the method and quantile level used in step 4.

Other Resolutions and Standardizations#

Your dataset is now in the spora [data] format. To prepare another resolution or standardization, pass the corresponding flags (for example --resolution or --method) to steps 4 and 5; hoefflin2026spatial also contains 0.5 mpp images, prepared with --resolution 0.5. Run a script with --help to see all of its options.