spora [bench]#
spora [bench] evaluates foundation models for spatial proteomics with six standardized tasks at three biological scales, from marker co-localization to patient-level pathology, on the harmonized cohorts of spora [data].
- [data]harmonized datasets
- [io]load and sample
- [bench]evaluate models
Three Scales, Six Tasks#
Each task assesses a distinct property of a model’s representations, from the subcellular to the tissue level.
Assesses whether a model captures marker co-localization by predicting held-out marker images from the remaining markers or from H&E.
Assesses whether cell representations encode phenotype and generalize to cohorts not seen during training.
Assesses whether tissue representations carry clinically relevant signal for grading, subtyping, and treatment response.
Running a Benchmark#
Each pipeline is specified by three configuration files: a model config, a datasets
config, and a benchmarks config. The example evaluates VirTues with linear probes for
cell phenotyping on the hoch2022multiplexed dataset. Before running it, set the
output directory, the dataset path, and the model checkpoint in the respective configs.
python -m spora_bench.benchmarks.run_cell_probe_tasks \
model_config=configs/models/virtues.yaml \
datasets_config=configs/datasets/virtues/hoch2022multiplexed.yaml \
benchmarks_config=configs/benchmarks/cell_probe/hoch2022multiplexed.yaml
Results are written to output_dir/virtues/results/. Several benchmarks can be run
in a single call by passing wildcards or lists of configs; see Configuration.
The Model Zoo#
spora [bench] includes foundation models, baselines, and specialized tools, each
instantiated from its model config. Further models can be integrated by implementing
SporaModelWrapper; see Model Zoo and Add New Models.
Explore the Docs#
Installation of spora [io] and spora [bench] and setup of the environment.
Path settings, dataset configs, and single or batched benchmark runs.
The six benchmark pipelines, their modules, and the wrapper methods they call.
Foundation models, baselines, and specialized tools included in spora [bench].