Benchmarks Overview

spora [bench] consists of six benchmark pipelines. The Task Zoo lists all datasets and configurations for which results are reported in the paper.

Benchmark pipelines

Task	Pipeline module	benchmarks_config	Wrapper method
Cross-cohort Instance Segmentation	run_segmentation_tasks	Not used	predict_instance_segmentation
In-cohort Cell Phenotyping	run_cell_probe_tasks	Required	compute_cell_tokens
Cross-cohort Cell Annotation	run_cell_annotation_tasks	Required	predict_cell_types
Tissue-level Prediction	run_tissue_level_tasks	Required	embed_tissue
H&E-to-MPX Virtual Staining	run_he_virtual_staining_tasks	Not used	predict_markers_from_he
MPX-to-MPX Virtual Staining	run_mpx_virtual_staining_tasks	Not used	predict_marker

All modules live in spora_bench.benchmarks and are launched as described in Configuration. The wrapper methods are those a model must implement to take part; see Add New Models. To define new tasks, see Add New Tasks.

The specialized cell-typing tools astir and MAPS have their own pipelines, run_astir and run_maps; see Model Zoo.

Data Splits

Evaluation uses the split column of each dataset’s metadata/tissues.parquet, as provided by spora [data]. The cell phenotyping and tissue-level pipelines train on the patients marked train and evaluate on those marked test. Segmentation, cell-type annotation, and MPX-to-MPX virtual staining evaluate on the test tissues without training. H&E-to-MPX virtual staining scores randomly sampled tiles of the dataset.