feat(dynacell): grouped multi-condition eval driver

.github/workflows/test.yml

@@ -145,14 +145,16 @@ jobs:
         # tiny iohub fixture + prebuilt mask cache — no eval extras (no
         # cellpose, transformers, cubic) needed since target_name=er +
         # require_complete_cache=true short-circuit segmenter and feature
-        # extractor loads.
+        # extractor loads. tests/test_evaluation_grouped.py drives the
+        # multi-condition driver against the same cache-only fixture.
         run: |
           uv run --frozen pytest \
             tests/test_benchmark_config_composition.py \
             tests/test_submit_benchmark_job.py \
             tests/test_runtime.py \
             tests/test_evaluation_pipeline_parallel.py \
             tests/test_evaluation_pipeline_parallel_cpu.py \
+            tests/test_evaluation_grouped.py \
             -v
         working-directory: applications/dynacell
 

applications/dynacell/CLAUDE.md

@@ -67,8 +67,56 @@ Under `executor=process`, workers return their slice of the timing log inside `F
 
 ### Reality check for process mode
 
-- The parent still loads the seg model once at `pipeline.py:171` (used for checkpoint pre-warm side-effect under `process` mode + the seg model itself under `serial` mode).
+- The parent still loads the seg model once via `load_eval_models(config)` at the start of `evaluate_predictions` in `pipeline.py` (used for checkpoint pre-warm side-effect under `process` mode + the seg model itself under `serial` mode).
 - `precompute_deep_features` (the upfront batched feature pre-fill) stays in the parent — single-pass over positions with the `DeepFeatureBatcher` cross-FOV amortization. Parallelizing precompute is a separate plan.
 - Workers cannot share open iohub plate handles or torch modules across the pickle boundary; each worker re-opens plates on first FOV.
 - `ProcessPoolExecutor.shutdown(wait=False, cancel_futures=True)` cancels queued futures only — in-flight workers continue to completion (~minutes if mid-cellpose). Ctrl-C may need `scancel` to fully release GPU memory.
 - Tests for the runtime module + FovResult pickle contract live in `applications/dynacell/tests/test_runtime.py` and `tests/test_evaluation_pipeline_parallel.py`. An end-to-end serial-vs-process parity test on a real iohub fixture is a follow-up (see plan `.claude/plans/eval-parallelism.md` §C5).
+
+### Grouped multi-condition eval
+
+When evaluating the same `(model, organelle)` across multiple I/O variants — typically the three A549 treatment plates (`mock`, `denv`, `zikv`), but also any case where the same trained model is scored on multiple datasets — use `dynacell evaluate-grouped`. The driver loads `SuperModel` + `DinoV3` + `DynaCLR` + `CELL-DINO` once, then loops over the conditions calling `evaluate_predictions(merged_cfg, models=...)` so the per-condition cold-start (~30–90 s of weight load + checkpoint warmup) is paid once total instead of N times.
+
+```yaml
+# applications/dynacell/configs/.../eval_grouped_a549_mantis_er.yml
+defaults:
+  - eval_grouped
+  - _self_
+
+target_name: er           # MUST be set at the base; conditions cannot override it
+compute_feature_metrics: true
+feature_extractor:
+  dinov3: { ... }         # shared across all conditions
+  dynaclr: { checkpoint: /path/to/dynaclr.ckpt, ... }
+  celldino: { weights_path: /path/to/celldino.ckpt, ... }
+
+conditions:
+  - name: a549_mock
+    io: { pred_path: ..., gt_path: ..., gt_cache_dir: ..., pred_cache_dir: ..., cell_segmentation_path: ... }
+    save: { save_dir: /path/to/out/a549_mock }
+  - name: a549_denv
+    io: { ... }
+    save: { save_dir: /path/to/out/a549_denv }
+  - name: a549_zikv
+    io: { ... }
+    save: { save_dir: /path/to/out/a549_zikv }
+```
+
+Invoke:
+```sh
+uv run dynacell evaluate-grouped -c eval_grouped_a549_mantis_er
+```
+
+Constraints (enforced at runtime by `_check_grouped_field_invariants`):
+- Per-condition overlays may freely override `io.*`, `save.*`, `runtime.*`, `limit_positions`, `force_recompute.*`, and carry a `name` label.
+- Overlays MUST NOT change `target_name`, `feature_extractor.*`, `compute_feature_metrics`, or `use_gpu` — those gate model loading and must be identical across conditions. Run such variants separately.
+- Each condition independently honors `_final_metrics_cache_valid` — if a condition's CSV/NPY already exist AND both `force_recompute.all=false` and `force_recompute.final_metrics=false`, the driver skips it and loads the cached outputs. Either flag set to `true` bypasses the cache.
+
+Process-mode caveat: under `runtime.executor=process`, the parent's shared `EvalModels` is passed to `evaluate_predictions(..., models=...)` so the parent-side pre-warm is amortized, but each condition still spawns a fresh `ProcessPoolExecutor` whose workers re-load their own model copies. Use `executor=serial` to maximize the amortization benefit. The driver tiers its message based on the cache mode:
+
+- `executor=process` + `require_complete_cache=true` + `n_conditions > 1` → mild **note**: workers re-init per condition but skip `prepare_segmentation_model` (returns None) and don't instantiate extractors, so the cost is just N pool spawns.
+- `executor=process` + `require_complete_cache=false` + `n_conditions > 1` → loud **WARNING**: each condition's worker pool independently loads SuperModel + DINOv3 + DynaCLR + CELL-DINO. Total waste is ~30-90 s × `runtime.fov_workers` × `n_conditions`. Fix: switch to `executor=serial` so the parent's pre-loaded models are reused across conditions; reserve `process` for per-FOV parallelism within a *single* condition.
+
+For an A40 / single-GPU interactive node where you'd run serial anyway, this is the default win.
+
+Tests: `applications/dynacell/tests/test_evaluation_grouped.py` validates byte-equal parity against sequential per-condition runs on the same cache-only fixture used by `test_evaluation_pipeline_parallel_cpu.py`, plus rejection cases for empty `conditions` and forbidden model-loading-field overrides.

applications/dynacell/src/dynacell/__main__.py

@@ -19,6 +19,7 @@
 
 _HYDRA_COMMANDS: dict[str, tuple[str, str, str]] = {
     "evaluate": ("dynacell.evaluation.pipeline", "evaluate_model", "eval"),
+    "evaluate-grouped": ("dynacell.evaluation.pipeline", "evaluate_model_grouped", "eval"),
     "precompute-gt": ("dynacell.evaluation.precompute_cli", "precompute_gt", "eval"),
     "report": ("dynacell.reporting.cli", "generate_report", "report"),
 }

applications/dynacell/src/dynacell/evaluation/_configs/eval_grouped.yaml

@@ -0,0 +1,34 @@
+defaults:
+  - eval
+  - _self_
+
+# Per-condition overlays for grouped multi-condition eval. Each entry is a
+# deep merge applied to the base config above. Use this to run e.g. all three
+# A549 treatment conditions (mock / denv / zikv) in one process, amortizing
+# the segmenter + DINOv3 + DynaCLR + CELL-DINO load.
+#
+# Each overlay may freely change ``io.*``, ``save.*``, ``runtime.*``,
+# ``limit_positions``, ``force_recompute.*``, and optionally carry a ``name``
+# label used in logs + the returned result tuple. Overlays MUST NOT touch
+# ``target_name``, ``feature_extractor.*``, ``compute_feature_metrics``, or
+# ``use_gpu`` — those gate model loading, which is shared. See
+# applications/dynacell/CLAUDE.md "Grouped multi-condition eval".
+#
+# Example:
+# conditions:
+#   - name: a549_mock
+#     io:
+#       pred_path: /path/to/mock_pred.zarr
+#       gt_path:   /path/to/mock_gt.zarr
+#       cell_segmentation_path: /path/to/mock_seg.zarr
+#       gt_cache_dir:   /path/to/mock_gt_cache
+#       pred_cache_dir: /path/to/mock_pred_cache
+#     save:
+#       save_dir: /path/to/eval_out/a549_mock
+#   - name: a549_denv
+#     io: { ... }
+#     save: { save_dir: /path/to/eval_out/a549_denv }
+#   - name: a549_zikv
+#     io: { ... }
+#     save: { save_dir: /path/to/eval_out/a549_zikv }
+conditions: []

applications/dynacell/src/dynacell/evaluation/model_loader.py

@@ -0,0 +1,193 @@
+"""Shared model-loading entry point for the dynacell eval pipeline.
+
+A single source of truth for instantiating the segmenter + deep-feature
+extractors used by ``evaluate_predictions`` and the precompute-gt CLI.
+Lifting the load out of the pipeline lets the grouped multi-condition
+driver build models once and reuse them across conditions that differ
+only in I/O paths; the precompute-gt CLI shares the same code path with
+its own per-extractor gates (``build.masks`` / ``build.dinov3`` / ...).
+
+The dataclass carries the **identity tags** (model name, checkpoint sha
+sources) that ``init_cache_context`` needs, so callers don't have to
+recompute them per condition.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+from omegaconf import DictConfig
+
+
+@dataclass
+class EvalModels:
+    """Bundle of pre-loaded segmenter + feature extractors with their identity tags.
+
+    Attributes
+    ----------
+    seg_model : Any | None
+        SuperModel returned by ``prepare_segmentation_model``. May be None
+        when ``io.require_complete_cache=true`` short-circuits the load
+        or when ``LoadFlags.masks=False``.
+    dinov3, dynaclr, celldino : Any | None
+        Deep feature extractor instances. Each is None when its
+        ``LoadFlags`` gate is False, or (for celldino) when its
+        ``weights_path`` is unset.
+    dinov3_model_name, dynaclr_ckpt_path, dynaclr_encoder_cfg,
+    celldino_weights_path : str | dict | None
+        Identity tags consumed by ``init_cache_context``. Stay None when
+        the corresponding extractor was not loaded.
+    """
+
+    seg_model: Any | None
+    dinov3: Any | None
+    dynaclr: Any | None
+    celldino: Any | None
+    dinov3_model_name: str | None
+    dynaclr_ckpt_path: str | None
+    dynaclr_encoder_cfg: dict[str, Any] | None
+    celldino_weights_path: str | None
+
+
+@dataclass(frozen=True)
+class LoadFlags:
+    """Per-model gate for :func:`load_eval_models`.
+
+    Lets ``precompute-gt`` (per-extractor flags ``build.masks`` /
+    ``build.dinov3`` / ``build.dynaclr`` / ``build.celldino``) and
+    ``evaluate-predictions`` (all extractors gated together by
+    ``compute_feature_metrics``) share one loader.
+    """
+
+    masks: bool = True
+    dinov3: bool = False
+    dynaclr: bool = False
+    celldino: bool = False
+
+    @classmethod
+    def for_evaluate(cls, config: DictConfig) -> LoadFlags:
+        """Flags matching the ``evaluate_predictions`` call path.
+
+        Masks always on (``prepare_segmentation_model`` handles its own
+        ``require_complete_cache`` short-circuit). Extractors gated as a
+        group by ``config.compute_feature_metrics``; celldino additionally
+        soft-skips inside the loader when its ``weights_path`` is null.
+        """
+        ext_on = bool(config.compute_feature_metrics)
+        return cls(masks=True, dinov3=ext_on, dynaclr=ext_on, celldino=ext_on)
+
+
+def load_eval_models(config: DictConfig, *, flags: LoadFlags | None = None) -> EvalModels:
+    """Instantiate the segmenter + deep feature extractors from ``config``.
+
+    No GPU serialization wrapping — callers (worker setup) wrap with
+    ``gpu_serialization_lock`` as needed. The parent of
+    ``evaluate_predictions`` calls this unwrapped today, so this entry
+    preserves that exact pattern.
+
+    Parameters
+    ----------
+    config : DictConfig
+        Resolved eval config. Reads ``target_name``,
+        ``feature_extractor.{dinov3,dynaclr,celldino}``,
+        and (transitively) ``io.require_complete_cache``.
+    flags : LoadFlags, optional
+        Per-model gates. Defaults to :meth:`LoadFlags.for_evaluate`,
+        which preserves the historical ``evaluate_predictions``
+        behavior. Pass a custom :class:`LoadFlags` to load a subset
+        (e.g. for precompute-gt where ``build.dinov3`` / etc. are
+        toggled independently).
+
+    Returns
+    -------
+    EvalModels
+        All model handles + identity tags. Each slot is None when its
+        flag is off; celldino additionally soft-skips when its
+        ``weights_path`` is null even with the flag on.
+    """
+    from dynacell.evaluation.pipeline_cache import resolve_dynaclr_encoder_cfg
+    from dynacell.evaluation.segmentation import prepare_segmentation_model
+    from dynacell.evaluation.utils import (
+        CellDinoFeatureExtractor,
+        DinoV3FeatureExtractor,
+        DynaCLRFeatureExtractor,
+    )
+
+    if flags is None:
+        flags = LoadFlags.for_evaluate(config)
+
+    seg_model = prepare_segmentation_model(config) if flags.masks else None
+
+    dinov3_model_name: str | None = None
+    dynaclr_ckpt_path: str | None = None
+    dynaclr_encoder_cfg: dict[str, Any] | None = None
+    celldino_weights_path: str | None = None
+    dinov3 = None
+    dynaclr = None
+    celldino = None
+
+    if flags.dinov3:
+        dinov3_model_name = config.feature_extractor.dinov3.pretrained_model_name
+        dinov3 = DinoV3FeatureExtractor(dinov3_model_name)
+    if flags.dynaclr:
+        dynaclr_config = config.feature_extractor.dynaclr
+        dynaclr_ckpt_path = str(dynaclr_config.checkpoint)
+        dynaclr_encoder_cfg = resolve_dynaclr_encoder_cfg(config)
+        dynaclr = DynaCLRFeatureExtractor(
+            checkpoint=dynaclr_config.checkpoint,
+            encoder_config=dynaclr_encoder_cfg,
+        )
+    if flags.celldino:
+        celldino_cfg = config.feature_extractor.celldino
+        if celldino_cfg.weights_path is not None:
+            celldino_weights_path = str(celldino_cfg.weights_path)
+            celldino = CellDinoFeatureExtractor(
+                weights_path=celldino_weights_path,
+                img_size=int(celldino_cfg.img_size),
+                patch_size=int(celldino_cfg.patch_size),
+            )
+
+    return EvalModels(
+        seg_model=seg_model,
+        dinov3=dinov3,
+        dynaclr=dynaclr,
+        celldino=celldino,
+        dinov3_model_name=dinov3_model_name,
+        dynaclr_ckpt_path=dynaclr_ckpt_path,
+        dynaclr_encoder_cfg=dynaclr_encoder_cfg,
+        celldino_weights_path=celldino_weights_path,
+    )
+
+
+def _identity_kwargs(models: EvalModels) -> dict[str, Any]:
+    """Identity tags forwarded into ``init_cache_context``."""
+    return {
+        "dinov3_model_name": models.dinov3_model_name,
+        "dynaclr_ckpt_path": models.dynaclr_ckpt_path,
+        "dynaclr_encoder_cfg": models.dynaclr_encoder_cfg,
+        "celldino_weights_path": models.celldino_weights_path,
+    }
+
+
+def init_cache_contexts(config: DictConfig, models: EvalModels) -> tuple[Any, Any]:
+    """Build ``(gt_ctx, pred_ctx)`` cache contexts using ``models``'s identity tags.
+
+    Replaces the four-call-site duplication in ``evaluate_predictions``
+    and ``_worker_setup`` where the same six kwargs flowed into
+    ``init_cache_context(..., side="gt")`` then again into
+    ``init_cache_context(..., side="pred")``.
+    """
+    from dynacell.evaluation.pipeline_cache import init_cache_context
+
+    kwargs = _identity_kwargs(models)
+    gt_ctx = init_cache_context(config, side="gt", **kwargs)
+    pred_ctx = init_cache_context(config, side="pred", **kwargs)
+    return gt_ctx, pred_ctx
+
+
+def init_gt_cache_context(config: DictConfig, models: EvalModels) -> Any:
+    """Build only the GT cache context — used by ``precompute-gt`` which is GT-only."""
+    from dynacell.evaluation.pipeline_cache import init_cache_context
+
+    return init_cache_context(config, side="gt", **_identity_kwargs(models))