Adding a Dataset

For new datasets, follow Convert New Datasets. See Dataset Format for the current on-disk contract and Tools and Scripts for tiling and statistics commands.

Implement a Custom Standardizer

Custom standardization classes live in spora_io/utils/dataset/standardize.py.

Use BaseStandardizer if the transform does not need saved parquet statistics. Use StatsBackedStandardizer if the transform should load quantiles, means, or standard deviations from <modality>/<resolution>/standardization/<standardization>.

Example stats-backed standardizer:

class SqrtQuantileClippingStandardizer(StatsBackedStandardizer):
    def _transform(
        self,
        x_t: torch.Tensor,
        upper_t: torch.Tensor,
        lower_t: torch.Tensor,
    ) -> torch.Tensor:
        x_t = torch.clamp(x_t, min=lower_t, max=upper_t)
        x_t = (x_t - lower_t) / (upper_t - lower_t + 1e-8)
        return torch.sqrt(torch.clamp(x_t, min=0.0))

Register the class in build_standardizer:

if method == "quantile_clipping_sqrt":
    return SqrtQuantileClippingStandardizer(
        spec=spec,
        use_mean_std=use_mean_std,
        **kwargs_common,
    )

If the method needs the standard statistics script to generate files, also add the method name to VALID_METHODS in scripts/compute_standardization_stats.py. The directory name written by the script must match the method name used in build_standardizer.

After registering the class, test it by loading one tissue manually:

ds = MultiplexImagingDataset(
    name="my_dataset",
    modality="imc",
    resolution=1.0,
    tile_size=None,
    standardization="quantile_clipping_sqrt/uq_0.99_image",
)
tissue = ds.get_tissue(ds.get_tissue_ids()[0])
print(tissue.image.shape)