Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
fd555c1
feat(dynacell/eval): add GT-only CP feature selection
alxndrkalinin Sep 26, 2026
da1f1b4
fix(dynacell/eval): score CP metrics in one shared per-target GT space
alxndrkalinin Sep 26, 2026
ed47573
feat(dynacell/tools): build per-target CP references from GT caches
alxndrkalinin Sep 26, 2026
e9eed94
fix(dynacell/eval): keep feature-less caches reusable without a CP hash
alxndrkalinin Sep 26, 2026
0940b5e
fix(dynacell/eval): scale CP per test set and pin each scaler to its …
alxndrkalinin Sep 26, 2026
6d7f70d
fix(dynacell/eval): mask the cross-condition CP probe with the target…
alxndrkalinin Sep 26, 2026
307f9de
fix(dynacell/tools): score lite-sim CP in the pipeline's reference space
alxndrkalinin Sep 26, 2026
4be7b53
test(dynacell/eval): pin CP scaler selection to each condition's dataset
alxndrkalinin Sep 26, 2026
f4b4153
fix(dynacell/eval): hash only what moves CP numbers in the reference
alxndrkalinin Sep 26, 2026
409f4cf
fix(dynacell/eval): refuse a non-lite CP scaler with no GT-cache stamp
alxndrkalinin Sep 26, 2026
659fa00
feat(dynacell/tools): verify a CP reference's GT-matrix hashes agains…
alxndrkalinin Sep 26, 2026
ca2de55
refactor(dynacell/eval): resolve an eval dir's CP space from its side…
alxndrkalinin Sep 26, 2026
3cbd4f1
docs(dynacell/eval): point the CP criteria comments at the reference
alxndrkalinin Sep 26, 2026
add4af7
docs(dynacell/eval): numpy docstrings for check_n_cells and save_metrics
alxndrkalinin Sep 26, 2026
cb52c00
fix(dynacell/eval): refuse final-metrics cache reuse after a GT re-cache
alxndrkalinin Sep 26, 2026
babe080
fix(dynacell/eval): bind the final-metrics stamp to the dataset and i…
alxndrkalinin Sep 26, 2026
b7b8615
fix(dynacell/eval): bypass the final-metrics cache on a partial posit…
alxndrkalinin Sep 26, 2026
6adf7a5
fix(dynacell/eval): check the GT store's positions before reusing CP …
alxndrkalinin Sep 26, 2026
9ae416d
fix(dynacell/eval): gate CP scoring on the GT cells' content, not bui…
alxndrkalinin Sep 26, 2026
d25ee66
fix(dynacell/eval): check CP cache columns by name against the reference
alxndrkalinin Sep 26, 2026
5c40ac7
test(dynacell/eval): score CP offset and contraction through the real…
alxndrkalinin Sep 26, 2026
26be0e4
test(dynacell/eval): pin lite staging to the parent scaler with pred …
alxndrkalinin Sep 26, 2026
4426fad
fix(dynacell/eval): let a cross-condition probe failure fail the grou…
alxndrkalinin Sep 26, 2026
9abbf13
fix(dynacell/eval): stamp only this dataset's CP space, not the whole…
alxndrkalinin Sep 26, 2026
5fc5ef8
fix(dynacell/eval): name benchmark.dataset_ref when a CP eval has none
alxndrkalinin Sep 26, 2026
f69f325
docs(dynacell/tools): show the feature-less limit_positions smoke run
alxndrkalinin Sep 26, 2026
d6dc5d5
test(dynacell/eval): check CP scaler binding with the benchmark's dat…
alxndrkalinin Sep 26, 2026
5f6b26f
docs(dynacell/eval): document that GLCM+ cosine cannot see contractio…
alxndrkalinin Sep 26, 2026
cdd0410
fix(dynacell/eval): resolve a relative CP reference override to an ab…
alxndrkalinin Sep 26, 2026
d87a029
fix(dynacell/eval): integrity-protect the CP reference's content-gate…
alxndrkalinin Sep 26, 2026
3387467
docs(dynacell/eval): pair the README smoke-run examples with feature …
alxndrkalinin Sep 26, 2026
5c6b607
fix(dynacell/eval): gate CP scoring on GT moments within tolerance, n…
alxndrkalinin Sep 26, 2026
bb7b4ab
fix(dynacell/eval): run the CP content gate before the FOV loop when …
alxndrkalinin Sep 26, 2026
4076b28
fix(dynacell/eval): have eval-dir sidecars pin their dataset's CP spa…
alxndrkalinin Sep 26, 2026
0c2fb98
docs(dynacell/eval): describe the per-dataset CP binding and toleranc…
alxndrkalinin Sep 26, 2026
72b2464
fix(dynacell/eval): check CP column names in every CP cache writer
alxndrkalinin Sep 26, 2026
6c2ee7b
fix(dynacell/eval): bind a cached CP space to its fit position set
alxndrkalinin Sep 26, 2026
23109df
docs(dynacell/tools): describe --verify's tolerant gate in the builde…
alxndrkalinin Sep 26, 2026
f85d472
fix(dynacell/eval): gate the GT and write the CP sidecar when no pred…
alxndrkalinin Sep 26, 2026
7cd6403
fix(dynacell/tools): make --verify compare the cache's position set w…
alxndrkalinin Sep 26, 2026
374a3e6
docs(dynacell/eval): state the CP moments gate's measured margins and…
alxndrkalinin Sep 26, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions applications/dynacell/src/dynacell/evaluation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ End-to-end evaluation pipeline for virtual staining predictions against fluoresc
- `io.gt_path` — fluorescence ground truth (channel: `io.gt_channel_name`)
- `io.cell_segmentation_path` — *optional* precomputed cell segmentation HCS OME-Zarr. Required when `compute_feature_metrics=true` or when building CP/DINOv3/DynaCLR/CELL-DINO cache entries. Position layout must match GT/pred 1:1.
- `io.gt_cache_dir`, `io.pred_cache_dir` — *optional* artifact cache directories; must be distinct. See [Caches](#caches).
- `feature_metrics.cp.reference_path` — the target's CP (GLCM+) reference, required when `compute_feature_metrics=true`. `null` (default) resolves to `DATA_ROOT/cp_reference/<target_name>.json` for full and lite evals alike. The reference holds one feature mask chosen on GT cells only, pooled over iPSC + A549 mock/denv/zikv, and one GT mean/std per test set (keyed by `benchmark.dataset_ref.dataset`; HEK gets its own, lite reuses its parent's). Every eval applies the mask and its dataset's scaler to pred AND GT. The eval refuses a partial position walk, and refuses to score unless its staged GT CP cells match the reference's fit for that dataset: the same GT-finite cell count exactly, and every CP feature's raw GT mean/std within a relative tolerance of 1e-6 (`GT_MOMENT_RTOL`). A re-cache that reproduces the cells passes, including a GPU recompute (reproducible only to ~1e-15) and a CPU-vs-GPU recompute (up to ~4e-8, measured). It is a moments gate, not a content hash: a change that moves any feature's GT mean/std by more than the tolerance fails, but permuting values within a column, moving cells between blocks, or a single-cell change below about n × 1e-6 std passes. When the GT CP cache is already complete, the check also runs before the FOV loop. Build the reference from existing GT caches with `tools/build_cp_reference.py --target <target>`; `--verify` re-applies the same check to the current caches. `metrics_provenance.json` stamps the per-dataset `cp_space_sha256`: the mask, feature names, recipe, and that dataset's scaler, recorded GT position set, GT cell count and GT moments. A rebuild therefore invalidates only the eval dirs whose own dataset's CP space changed; adding or refitting another dataset leaves them reusable. The `cp_selected_feature_mask.json` sidecar records the same per-dataset hash.
- GLCM+ (CP) space in two lines: one feature mask selected on GT cells only, pooled over the target's iPSC + A549 mock/denv/zikv test sets; then each test set's own GT mean/std standardizes pred and GT alike, so offsets and scale errors relative to that test set's GT stay visible.
- GLCM+ median cosine is invariant to contraction toward the GT mean: for `pred = mean + 0.5·(GT − mean)`, every pred/GT cell pair is parallel after the shared scaler and the median cosine is exactly 1.0000, while KID catches it (reviewer-measured on real cells: KID 0.194 vs a pred==GT floor of −0.041; reproduced on synthetic cells in `cp_reference_test.py`). Cosine measures direction only; over-smoothing toward the mean shows up in KID, not cosine.

## Quick start

Expand All @@ -35,7 +38,7 @@ uv run dynacell evaluate \
save.save_dir=/hpc/.../eval_fnet3d_sec61b
```

Add `compute_feature_metrics=true` to enable feature metrics. Smoke test on a subset of FOVs with `limit_positions=N`.
Add `compute_feature_metrics=true` to enable feature metrics. Smoke test on a subset of FOVs with `limit_positions=N compute_feature_metrics=false`: CP feature metrics are scored in a reference fit on the full GT cell set, so a partial position walk with feature metrics on raises.

## Submission tooling

Expand Down Expand Up @@ -109,7 +112,7 @@ Canonical leaves at `configs/benchmarks/virtual_staining/<org>/<model>/<train_se
uv run dynacell evaluate leaf=er/celldiff/ipsc_confocal/eval__ipsc_confocal
```

Coverage: `(er, membrane, mito, nucleus) × (celldiff, unetvit3d)`. CLI overrides apply on top (e.g. `limit_positions=1` for smoke).
Coverage: `(er, membrane, mito, nucleus) × (celldiff, unetvit3d)`. CLI overrides apply on top (e.g. `limit_positions=1 compute_feature_metrics=false` for smoke).

## Caches

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,13 @@ feature_metrics:
enabled: true
levels: 32
distances: [1]
# Per-target CP reference: one GT-only feature mask (pooled over iPSC + A549
# mock/denv/zikv) + one GT mean/std per test set (lite -> its parent's), applied to
# BOTH pred and GT so every model is scored in the same CP space. Keyed by
# benchmark.dataset_ref.dataset. null -> registry default
# DATA_ROOT/cp_reference/<target_name>.json (lite evals use it too).
# Build with applications/dynacell/tools/build_cp_reference.py --target <target>.
reference_path: null
# In-focus slab for the 2D deep-feature crops + per-cell similarity. When
# enabled, the max-Z projection is restricted to a band centered on the phase
# focus plane (read from focus_slice zattrs; write via precompute-gt build.focus)
Expand Down
Loading
Loading