Cross-cohort Cell Annotation

In practice, new datasets are generated without prior labels, and annotating them requires phenotyping methods that generalize across cohorts with varying measured markers. The pipeline run_cell_annotation_tasks assesses the cross-cohort generalization of such annotation methods: given a tissue and its reference cell instance mask, the model predicts a cell type for every cell.

Important

To assess cross-cohort generalization, it is the responsibility of the user to ensure that the annotation method has not seen the target datasets during supervised training.

Running the Benchmark

python -m spora_bench.benchmarks.run_cell_annotation_tasks \
  model_config=<path-to-model-config> \
  datasets_config=<path-to-datasets-config> \
  benchmarks_config=<path-to-benchmarks-config>

Batched submission of datasets and benchmarks is possible via lists or wildcards.

Outputs

For each task, a classification report, a bootstrapped classification report (1,000 resamples), and a confusion matrix over the test tissues are saved to output_dir/<model_name>/results/.

Benchmark Config

datasets:
  <dataset_name>:
    benchmarks:
      cell_annotation:
        <task_name>:
          label_col: <column_name>
          class_order: <list of classes (by name) predicted by the method, in the order of the predicted class indices>
          excluded_classes: <list of classes (by name) to exclude from evaluation>

class_order maps the integer class IDs returned by predict_cell_types to label names, so it must match the method. For example, configs/benchmarks/cell_annotation/cords2023cancer.yaml:

datasets:
  cords2023cancer:
    benchmarks:
      cell_annotation:
        cell_type_atlas:
          label_col: cell_type_atlas
          class_order: ['None', 'B cell', 'CD4 T cell', 'CD8 T cell',
                        'Fibroblasts /Stroma', 'Myeloid', 'Tumor', 'Unknown',
                        'Vessel / Endo.']
          excluded_classes: ['None', 'Unknown']

The datasets and configs used in the paper are listed in the Task Zoo.

MAPS is evaluated on these tasks with its own pipeline; see Model Zoo.