Configuration

Every benchmark pipeline is a Python module under spora_bench.benchmarks, configured by YAML files and launched from the repository root.

Configuration Files

Pipelines combine four types of configuration files:

Config	Location	Contents
base_config	configs/base_config.yaml	Global parameters: output_dir and random_seed.
model_config	configs/models/	Everything required to instantiate the target model.
datasets_config	configs/datasets/<model>/	Dataset storage locations and standardization settings.
benchmarks_config	configs/benchmarks/<task>/	Which tasks to run for each dataset.

The base config is always loaded. The other three are passed on the command line as key=value arguments; additional individual keys passed the same way override the merged configuration, for example model.checkpoint_path=/path/to/ckpt.pt.

Initial Configuration

All configurations needed to run the benchmarks reported in the paper are provided in configs/. Only a few path variables need to be updated:

1. Set output_dir in configs/base_config.yaml. Results are written to subdirectories of output_dir/<model_name>/.

2. In every dataset config, for example configs/datasets/virtues/cords2023cancer.yaml, set path to the root directory of the respective dataset.

3. In the model config, for example configs/models/virtues.yaml, set the model-specific paths: checkpoint_path and marker_embeddings_dir for VirTues; checkpoint_path and marker_metadata_path for KRONOS.

Some models require their own dataset configs to account for differences in standardization. For KRONOS, use the dataset configs in configs/datasets/kronos/.

Dataset Configs

spora [bench] is built on spora [data] and spora [io]. Any dataset in spora [data] can be integrated by writing a dataset config in the following format:

datasets:
  <dataset_name>:
    name: <dataset_name>
    path: <path/to/dataset>
    modality: <modality>
    standardization: quantile_clipping_log1p/uq_0.99_image
    resolution: 1.0
    filter_list: ['gaussian_blur']
    use_mean_std: <True/False>
    disable_quantile_mask: <True/False>

Field	Meaning
modality	Multiplex modality key, e.g. imc, codex, cycif, or mibi.
standardization	Path to the standardization statistics relative to the modality directory, e.g. quantile_clipping_log1p/uq_0.99_image for 99 % clipping and a log1p transform. See the spora [io] standardization settings.
resolution	Resolution in micrometres per pixel; default 1.0.
filter_list	Image filters applied by the spora [io] loader: gaussian_blur and/or median_filter, or []. The provided configs use ['gaussian_blur'].
use_mean_std	Whether to apply mean/std normalization; depends on the standardization.
disable_quantile_mask	Option to keep channels with zero quantiles and/or zero standard deviations, which are filtered out otherwise.

For examples, see configs/datasets/virtues/.

Starting a Benchmark

The general form is:

python -m spora_bench.benchmarks.<pipeline> \
  model_config=<path-to-model-config> \
  datasets_config=<path-to-datasets-config> \
  benchmarks_config=<path-to-benchmarks-config>

Cell instance segmentation and both virtual staining pipelines need no benchmarks_config. Benchmarks Overview lists the pipelines and links to a page for each task.

Batched Submission

datasets_config and benchmarks_config accept a single path, a path with wildcards, or a list of paths; matching files are loaded in sorted order and merged. Quote wildcards so that the pipeline, not the shell, expands them. For instance, to run all cell phenotyping tasks for VirTues:

python -m spora_bench.benchmarks.run_cell_probe_tasks \
  model_config=configs/models/virtues.yaml \
  datasets_config="configs/datasets/virtues/*.yaml" \
  benchmarks_config="configs/benchmarks/cell_probe/*.yaml"

Lists use brackets without spaces, for example datasets_config=[configs/datasets/virtues/cords2023cancer.yaml,configs/datasets/virtues/cords2024cancer.yaml].

Outputs

Results are saved per model to output_dir/<model_name>/results/. Each pipeline page describes the files it writes.