You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(dynacell/eval): score GLCM+/CP metrics in one GT-fit feature space - #499
The GLCM+/CP feature metrics (KID, FID, cosine, at both dataset level and per row) were scored in a feature space that depended on the model.
Per-dir feature subset.select_features(gt, pred) ran once per eval dir, on GT and prediction cells pooled together. In the paper scope, each (organelle, test set) had 6–8 distinct subsets across its 8 model cells, keeping 7–14 of 22 features.
Per-side z-score._zscore_per_side standardised prediction and GT each by its own mean and std. That hid every per-feature offset or scale gap, including uniform over-smoothing. The per-(FOV, t) path _cp_dropzero_zscore did the same.
Deep extractors were never affected.
Fix
Per-target CP reference at DATA_ROOT/cp_reference/<target>.json:
Feature mask: one per target, selected on GT cells only, pooled over iPSC and A549 mock/denv/zikv. The criteria constants are unchanged.
Scalers: one per test set, with mean and std fit on that test set's GT only and applied identically to prediction and GT. HEK gets its own scaler but is not in the mask fit. Lite datasets reuse their parent's scaler, via lite_of.
Std floor: 1e-3 × the pooled std, with floored features recorded. None fire today.
Why per-test-set scalers: a pooled scaler squeezes within-dataset spread. Between-dataset share of variance reaches eta² 0.91 (nucleus glcm_homogeneity), and some kept features had only 0.02 sd of within-dataset spread.
Fit provenance, unhashed and checked at eval: per-dataset positions, n_cells, GT cache dir and cp_features.built_at, and the GT-matrix sha256.
At eval start, the run raises if the position set differs: it must be equal for full datasets and a subset for lite ones.
It raises if the GT cache was re-cached, or if the dataset-level GT cell count differs.
build_cp_reference.py --verify recomputes the GT-matrix hashes.
Reference hash covers only what moves numbers: mask, scalers, floors, the lite→parent map, the recipe and the criteria.
It is stamped in metrics_provenance.json from the reference actually scored.
_final_metrics_cache_valid rejects a different hash when feature metrics are on. Dirs with feature metrics off are untouched.
Other consumers moved to the reference:cross_condition_probe (its per-plate MAD stays deliberate) and the lite tools (lite_checks, lite_subset_sim). select_features and the per-side z-score are deleted.
Builder:tools/build_cp_reference.py reads existing GT CP caches only. It writes atomically and needs --force to replace a reference with a different hash.
Rollout
The 4 references are already built in the registry and pass --verify (exit 0):
target
sha256
nucleus
59726576…
membrane
f4b60510…
er
ba687e63…
mitochondria
4cf6c089…
After merge, any eval with feature metrics on raises if its target's reference is missing.
All 34 (target, dataset) pairs used by eval leaves are covered.
All existing CP-bearing caches recompute on next eval, because their stamps have no hash.
Tests
Full dynacell suite in the eval venv, with worktree packages/*/src and applications/dynacell/src first on PYTHONPATH: 2 failed, 3569 passed, 164 skipped, 5 deselected.
The 2 failures (test_load_eval_models_pins_morphem_to_the_configured_revision and test_check_cubic_pin_rejects_a_mismatch) also fail on the base commit and pass in isolation, so they are order-dependent.
Discrimination: an independent reviewer mutated the code, and each mutation fails at least one test:
a per-side z-score in the transform
a per-side z-score only in the per-row path
one pooled scaler
selecting the wrong dataset's scaler
removing the position or cell-count check
a wrong lite parent
Over-smoothing (pred = 0.5*GT + 7): the old transform gives cosine 1.0 and KID −0.04, identical to GT scored against itself. The new transform gives cosine 0.41 and KID 8.27.
select_features pools GT with one model's prediction, so the kept CP
subset moves with the model being evaluated. select_gt_features applies
the same variance + |r|>0.9 criteria to GT rows alone; it is the
selector the per-target CP reference is fit with. select_features stays
for cross_condition_probe, now sharing the pruning helper.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
CP (GLCM+) KID/FID/cosine were computed in a model-dependent space: each
eval dir chose its own feature subset from pooled GT+pred cells, then
z-scored pred and GT each by its OWN mean/std. Every model got a
different mask (er/ipsc: fnet3d_paper kept 11 features,
fcmae_vscyto3d_pretrained 10), and any per-feature offset or scale
error -- e.g. uniform over-smoothing -- was standardized away.
Every eval now loads the target's CP reference (cp_reference.py): a
keep-mask chosen on GT cells only, pooled over iPSC + A549
mock/denv/zikv, plus one mean/std fit on that GT. Both are applied to
pred AND GT, at dataset level and per row. The linear probe gets the
masked, unscaled arrays (it scales per fold). The per-FOV drop of
target-all-zero columns existed only to keep the per-side z-score
finite, so it is gone. The dim-mismatch StaleCacheError moves into
CPReference.
- feature_metrics.cp.reference_path (null -> DATA_ROOT/cp_reference/
<target>__3d.json, shared by lite and full evals).
- Missing reference -> FileNotFoundError naming the build command;
reference of another CP recipe (version/norm/GLCM) -> StaleCacheError.
Both are checked before any model load.
- The reference's content hash is stamped in metrics_provenance.json,
and _final_metrics_cache_valid refuses a cache stamped with another
hash, or with none, so old CP values cannot be silently reused.
- cp_selected_feature_mask.json now records the reference path + hash.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
build_cp_reference.py fits a target's CP reference on iPSC + A549
mock/denv/zikv GT cells. HEK and the movie sets are excluded. Each
dataset ref is resolved as an eval resolves it (apply_dataset_ref on
eval.yaml). The GT cache is opened via init_cache_context with
require_complete_cache=true, so a cache of another store or CP recipe
is refused. CP arrays are read read-only; any missing (position,
timepoint) raises, and features are never computed. Non-finite cells
are dropped with drop_paired_nonfinite_rows. The JSON records per-
dataset cache paths, cell counts and a sha256 of the stacked GT matrix.
Real er/3d build (to scratch): 7162 cells from 4 caches, 12/22
features kept, ~15 s wall.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
metrics_provenance_matches required a cp_reference_sha256 key in every
stamp, so a compute_feature_metrics=false dir -- which scores no CP
value -- was invalidated just for predating the CP reference. The key
is now required only when the caller passes a hash, i.e. when the
config computes feature metrics.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
…GT cells
The pooled GT scaler squeezed within-dataset spread: between-dataset
offset dominates the pooled variance (eta^2 up to 0.91, measured by the
reviewer), leaving 0.02-0.13 sd on kept features. The reference (schema
2) keeps one GT-only mask pooled over iPSC + A549 mock/denv/zikv. It now
holds one GT mean/std per test set, keyed by
benchmark.dataset_ref.dataset, applied to pred and GT alike.
- The builder fits a scaler for every dataset a grouped leaf of the
target evaluates, HEK included; HEK stays out of the mask fit. Lite
datasets reuse the parent named by their split's
selection_criteria.lite_of and are never refit.
- A dataset std below 1e-3 x the pooled GT std is floored and the
floored features are recorded; a zero pooled std is still refused.
- Each scaler records its positions, n_cells, GT-matrix sha256 and GT
cache cp_features.built_at. An eval refuses a GT re-cache up front, a
partial walk (limit_positions / exclude_fov_names) on a non-lite set,
and a GT CP cell count that differs from the fit.
- The caller loads the bound space once and threads it to
evaluate_predictions and save_metrics, so the stamp is the reference
the run scored with, never a re-read.
- Atomic write (tmp + os.replace). A different existing reference is
replaced only with --force; an identical hash is a no-op.
- One cp_space(config) derives identity and names for eval and builder,
reading the config keys directly. The dimension parameter is gone; the
registry path is DATA_ROOT/cp_reference/<target>.json.
- Per-row CP goes through _cp_row_features. Tests assert the staged and
per-row arrays equal the dataset-scaler transform (a per-side mutation
fails them), and cover the _worker_run_fov handoff.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
…'s reference
The infection probe pruned CP columns with select_features on each
(condition pair, source) cohort, so its feature subset moved with the
model and the pair. It now applies the keep-mask of the CP reference
named in the eval dirs' cp_selected_feature_mask.json, hash-checked. It
raises when the two dirs were scored in different references. The
per-plate MADScaler is unchanged; a comment now says it removes per-
condition offsets by design.
select_features has no caller left, so it and its tests are removed;
the GT-only selector inlines the pruning helper. The determinism test
now pins the exact expected mask.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
lite_checks (check E) and lite_subset_sim re-implemented the retired
CP transform: the sidecar keep_mask, then a per-side z-score over each
cell subset. Both now read the reference path, hash and dataset from
cp_selected_feature_mask.json and apply the pipeline's transform: the
reference mask plus that dataset's GT scaler (the parent's for lite).
The transform no longer depends on the cell subset, so the
--cp-frozen-norm variant is gone. The tests fit a real reference
whose scaler differs from the eval cells' own statistics, and assert
the result differs from the per-side z-score.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
Nothing checked which scaler an eval was bound to: making
eval_cp_space return the first scaler passed every test. The new tests
fit a reference with three sets whose scalers differ plus one lite set.
They drive the real eval_cp_space and the grouped per-condition
driver, and assert that each condition gets its own dataset's
mean/std, a lite set gets its parent's, the stamp is the scoring
reference, and an unknown dataset is refused. The first-scaler mutation
now fails all three.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
The content hash covered fit provenance too (GT-cache built_at,
cache dirs, positions, cell counts, GT-matrix hashes), so rebuilding
after a harmless re-cache changed the hash and invalidated every
stamped final-metrics cache although no scaler moved.
Schema 3 splits the payload. The hashed part is recipe identity,
criteria, feature names, keep-mask, per-dataset mean/std/floored
features, and the lite -> parent map. Provenance moves to an unhashed
`fit` section, which the eval checks still read. write_cp_reference
rewrites same-hash, new-provenance references without --force, so the
checks see the new built_at. It stays a no-op when both are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
check_gt_cache skipped whenever no built_at was recorded, so a
non-lite reference built without one silently disabled GT re-cache
detection. The skip now applies only to lite datasets (their scaler is
the parent's); a non-lite dataset without a recorded built_at raises.
The test fixture now writes a real cp_features entry (full CP identity
plus built_at) into the eval's GT cache and records it in the
reference, as the builder does for real caches. The re-cache test
moves that stamp instead of hand-editing the reference.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
…t the caches
The docstrings claimed the per-dataset gt_matrix_sha256 made a GT
re-cache "detectable against the artifact", but nothing ever
recomputed it. build_cp_reference.py --verify now re-reads each
non-lite dataset's GT CP cells, compares sha256 and cell count with the
reference's fit section, and exits 1 on a mismatch.
check_gt_cache now documents its limit. built_at is a best-effort
re-cache signal: _merge_manifests takes it from the last flusher, and
cache merging is unchanged here. check_n_cells catches count changes;
a value-only change is caught only by --verify.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
…car in one place
cross_condition_probe and lite_subset_sim each re-implemented
"sidecar -> reference". They both read the reference file once just
to learn its target_name and passed it back to load_cp_reference,
which then checked the target against itself.
cp_reference.sidecar_cp_space reads the reference path and sha256 from
the sidecar, loads the reference, and verifies that sha256 still
matches. It binds the space to the recorded dataset, so lite gets its
parent's scaler. All three consumers use it, and the sidecar filename
is a single constant shared with the pipeline.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
feature_select's constants comment and active_cp_feature_names still
described the per-eval cp_selected_feature_mask sidecar as where the
criteria and column schema must not drift. Both now live in the CP
reference (the criteria block and the recipe identity).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
This PR standardizes CP/GLCM+ metrics in a GT-fitted feature space with shared references, dataset scalers, provenance validation, and updated tooling.
Changes:
Adds CP reference building, hashing, verification, and cache provenance.
Applies shared CP transforms across evaluation and analysis tools.
Adds tests and documentation for references, scaling, and validation.
The cache-reuse path compared only the CP reference hash, so a GT CP
cache re-cached after the reference was built still let the next eval
reuse CP rows scored against the old GT. Run the same check_gt_cache
guard the scoring path runs before accepting the cache.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
…ts GT cells
The stamp held only the whole-reference hash, which deliberately leaves
fit provenance unhashed. Two stale caches could still be reused (both
from Copilot review on #499). A GT re-cache that changed the cell
matrix while mean/std/mask stayed the same reused CP rows scored on the
old cells. A save_dir scored for dataset A was also accepted for
dataset B of the same reference.
One mechanism fixes both. DatasetCPSpace.binding_sha256 hashes the
reference sha256, dataset, scaler dataset, lite flag and the GT-matrix
sha256 recorded in the fit section. That is the dataset's own for a
non-lite set; a lite set uses its own if recorded, else the parent's
plus its recorded cache built_at. metrics_provenance.json stamps it as
cp_space_sha256 and keeps cp_reference_sha256 for audit.
metrics_provenance_matches compares cp_space_sha256 when feature
metrics are on. save_metrics takes the scoring DatasetCPSpace itself,
so both hashes come from the space that scored the run.
A rebuild that moves only built_at keeps the binding, so harmless
re-caches still invalidate nothing. Existing registry references need
no rebuild: their fit section already records every GT-matrix sha256.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
This copies the configured reference path verbatim into the sidecar. With a relative feature_metrics.cp.reference_path, the eval can load the file from its current working directory, but sidecar_cp_space() later resolves that path relative to whatever directory lite_checks/cross_condition_probe was launched from, so post-hoc consumers cannot reopen the scored reference. Store a normalized absolute path (or define and apply a stable sidecar-relative path convention) before writing the sidecar.
…ion walk
The cache-hit path returned the cached CP rows without running
check_positions, so an eval with limit_positions or io.exclude_fov_names
could reuse metrics scored over a different GT position set. Treat a
partial walk as a cache miss so the scoring path's position check runs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
Lite final-cache provenance cannot detect a value-only rewrite of the lite CP cache. lite_entry records only the lite cache's built_at, and binding_sha256 falls back to the parent's GT hash plus that timestamp when own_gt_matrix_sha256 is absent. If the lite cache is rewritten with the same cell count and unchanged/stale built_at (the code documents this as a possible miss), _final_metrics_cache_valid produces the same binding and reuses CP metrics computed from the old lite GT cells. Record and compare a lite GT-matrix digest (and count), or conservatively disable final-cache reuse for lite datasets.
…rows
The cache-hit path accepted cached CP metrics without comparing the GT
store's current positions with the reference's fit positions, so a GT
store extended or replaced since the fit could hand back rows scored on
another cell set. Run the scoring path's check_positions on the GT
store before accepting the cache.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
…metrics off
A partial position walk now raises when CP feature metrics are on, so
the two `limit_positions` smoke examples would fail as written.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
Re-review at 5f6b26f, plus cdd0410 and d87a029: changes requested (one blocker, one important)
Everything that reads the real registry was run from a d87a0293 snapshot; the mutation runs used 5f6b26f8, since d87a029 touches none of the mutated lines. The suite of touched files passes: 242 at d87a029, 0 failures.
#
Verdict
Evidence
1b manifest-merge built_at
FIXED
No gate reads built_at any more.
1c same-count GT change
FIXED
The e2e test raises before any write; disabling the gate fails 3 tests.
row order behind the GT sha
OK
Keys are sorted before hashing, so executor and position order don't matter; an unsorted mutation is caught.
4 mask by name
FIXED
Reordering names without a version bump raises for new and old caches (9 tests fail).
5 e2e coverage
FIXED
All 3 previous survivors now fail 3–4 tests; putting the per-side z-score back fails 4.
6 lite refit
FIXED
7 probe except
FIXED
8 per-dataset hash
FIXED
Adding a dataset to the real er.json left all 8 per-dataset hashes unchanged.
1a force_recompute.all on GPU
NOT FIXED
See F1.
F1 (blocking): the exact-sha content gate can't survive a GPU recompute
cp_regionprops on GPU is not bit-reproducible. Five runs on an A40 gave five different hashes, differing by about 1e-15; three runs on CPU gave one hash.
On the real er.json, swapping one cached a549-mantis-sec61b-mock block for an A40 recompute raises StaleCacheError.
So any force_recompute.gt_cp / all / --overwrite eval on GPU (i.e. production, and the planned re-eval campaign) breaks that dataset for every later eval. It only fails after the whole FOV loop has rewritten the shared GT cache.
The e2e fixture sets use_gpu: False, so the test suite can't see it.
Proposed fix. The question the gate must answer is "does the recorded scaler still describe these GT cells?", so compare statistics within a tolerance rather than bits:
positions equal and n_cells equal (exact);
per feature, max |mean_staged − mean_fit| / std_fit ≤ 1e-6 and max |std_staged / std_fit − 1| ≤ 1e-6, computed on the same staged GT-finite rows.
A same-count value change of any consequence moves these by orders of magnitude more than 1e-15; please show that with the existing 1c test. Keep the sha in the record for audit only. Add a GPU-nondeterminism test by perturbing the staged rows by ~1e-12 relative, which must pass, next to the 1c case, which must raise.
F2 (important): a stale sidecar hash after a reference rebuild
Item 8 keeps a dir reusable across unrelated rebuilds, because its per-dataset hash is unchanged.
But the sidecar still records the whole-reference hash. sidecar_cp_space raises ValueError, and after item 7 nothing catches it, so the grouped run fails. Reused dirs are never re-scored, so the stale sidecar persists.
d87a029 makes whole-reference changes more frequent. Reproduced at both commits; not live yet, since all 458 sidecars on disk are pre-PR.
Fix: the sidecar records and checks the same per-dataset cp_space_sha256 as the stamp.
Nits
F3: the README still says that a rebuilt reference invalidates the metrics cache, and that an identical re-cache passes. Both are now false; please update.
F4: the name check is per manifest, not per slot. A writer that skips the check (precompute_cli.py) could restamp the names over old-order slots. This only happens after a reorder without a version bump, which a test already fails, so it's low priority.
9ae416d moved the re-cache check to after the FOV loop, so a run that will fail computes everything first. The tolerance gate from F1 makes that failure rare; if it's cheap, also check positions and n_cells before the loop.
Noted, not blocking
Code outside the grouped run will raise KeyError on pre-PR sidecars until the re-eval campaign has passed through them: the probe command line (--eval_dirs), lite_checks.py, lite_subset_sim.py, and dynacell's rerun_pix2pix_probes.py. Loud is right here, since the campaign re-scores everything.
…ot an exact sha
GPU cp_regionprops is not bit-reproducible. The reviewer measured 5
distinct hashes in 5 A40 runs (|diff| ~1e-15); here two runs of one
block differ by 8.9e-16. So the exact-sha content gate failed every
GPU force_recompute.all / gt_cp / --overwrite, after the FOV loop had
already rewritten the shared GT cache, poisoning the dataset until a
--force rebuild (F1, blocker, #499 re-review).
The gate now requires an exact position set and GT-finite cell count,
plus per-feature tolerance on the staged GT's raw mean/std over ALL CP
columns against the fit's recorded moments: |dmean| <= 1e-6*std_fit and
|std/std_fit - 1| <= 1e-6. A constant feature (std_fit == 0) must stay
within 1e-12*(1+|mean_fit|). The tolerances are named constants.
- Each fit record (lite from its own caches) stores gt_moments. They are
hashed gate fields; gt_matrix_sha256 is audit-only (unhashed).
- --verify applies the same gate, via the shared gt_cells_problems, and
reports an exact-sha difference as information.
- cp_space_sha256 hashes the recorded count + moments instead of the GT
sha, so GPU jitter never moves a binding.
- Tests: a 1e-12 relative jitter of every GT cell passes end to end. The
same-count value change still raises. CUDA-only tests recompute one
block twice on the GPU, and run an e2e force_recompute.gt_cp with
use_gpu=True against a CPU-built reference. Both pass on the A40. The
GPU fixtures leave out the single-voxel cell: cuCIM's GPU regionprops
raises a TypeError on a one-voxel region. Schema 5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
…the GT cache is complete
The gate ran only after the FOV loop, so a doomed run (GT changed since
the fit) spent the whole loop first (ordering nit, #499 re-review).
When the GT CP cache already holds every slot and this run does not
recompute GT CP, the cells the loop will score are known up front.
The gate is applied to them right after the position check. The
post-staging gate stays authoritative (for recomputes and partially
cached stores). A test shows a stale complete cache failing before any
FOV is processed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
…ce, not the reference
The cp_selected_feature_mask.json sidecar pinned the whole-reference
sha256, while the final-metrics stamp is per dataset. After any rebuild
(a dataset added, a --force for F1), cache-valid dirs kept their old
sidecar and sidecar_cp_space raised. Since the probe no longer swallows
errors, that failed every grouped re-run with a reused condition, and
the lite tools too (F2, #499 re-review).
The sidecar now records reference_path, dataset and that dataset's
cp_space_sha256 (binding), built by one cp_sidecar_payload shared by the
pipeline and the tests. The whole-reference sha is kept as audit.
sidecar_cp_space loads the current reference, binds the recorded
dataset and compares bindings. The cross-condition probe requires one
reference path and one mask across its dirs. Test: an old sidecar
loads after a rebuild that adds an unrelated dataset, and raises once
its own scaler changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
…e gate accurately
The reference_path bullet said a rebuilt reference invalidates the
final-metrics cache (the binding is per dataset now), and that an
identical re-cache passes an exact-hash gate (false on GPU) (F3, #499
re-review). It now describes the count + moments tolerance gate and its
early run on a complete cache, --verify, and the per-dataset
cp_space_sha256 in both the provenance stamp and the sidecar.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
The name check ran per manifest at eval start, not per write. A writer
that skipped it (precompute-gt) could restamp cp_feature_names over
slots of the old column order, and the mixed cache then passed (F4,
#499 re-review). fov_cp_features, the one writer the eval and
precompute-gt share, now runs check_cp_cache_feature_names against the
running code's column names before any write. A test simulates a
reordered live column list and shows the precompute-style write refused
with the recorded names untouched.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
cp_space_sha256 covered the GT cell count and moments but not the
position set, so a reference rebuilt on other positions with the same
count, moments and scaler kept the binding and let cached metrics from
the old positions be reused.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
…r docstring
--verify fails on the count/moment gate and reports an exact GT-matrix
sha difference as information only (GPU extraction is not
bit-reproducible); the module docstring still promised exit 1 on any
sha mismatch.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
mask_fit is returned unchanged here, so its gt_matrix_sha256 remains part of the reference content hash even though _AUDIT_FIT_KEYS documents matrix hashes as audit-only. A harmless GPU recache can change this pooled digest at ~1e-15 while leaving the mask, pooled moments, scalers, and metric values unchanged; rebuilding then produces a different reference hash and requires --force unnecessarily. Exclude gt_matrix_sha256 from fit["mask_fit"] as well, or otherwise keep this audit field out of payload_sha256.
… CP rows remain
When every prediction CP row was non-finite, the dataset-level block
skipped CP staging, which also skipped the GT content gate and the
cp_selected_feature_mask.json sidecar. Run both whenever feature metrics
are on; CP metrics stay NaN in that case.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
…ith the fit
--verify checked only cell count and moments, so a cache rebuilt from a
different position set with the same count and distribution reported
OK despite the eval's exact position check.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
Round-6 verification: code APPROVED, with one required data step before any GT-recompute eval
Verified at 72b2464 on an A40. 72b2464e..23109df0 changes no verdict: 6c2ee7b only adds the position set to the per-dataset hash, and the F2 probe and tests give identical results at 23109df.
Item
Verdict
Evidence (MEASURED)
F1 GPU jitter
Fixed
a549-mantis-sec61b-mock fov0000, all 10 t, recomputed 3× on the A40 and 1× on CPU through the real cache writer, into scratch. Max Δ was 6.5–6.8e-15 on GPU and 3.0e-8 on CPU. The exact sha differed every time; the new gate passes all 4. The 3 CUDA tests pass on the A40, and an exact-sha mutant fails all 3.
Discrimination
OK, with known limits
×1.01 on one column of one block: 21/22 caught (the miss is an all-zero column). Swapping two features within a cell: 231/231 caught. Smallest single-cell change caught: 1.9e-3 σ (n_cells × 1e-6). Swapping two cells within one feature is not caught, but it can't change the scaler either.
F2 rebuild
Fixed
Changing an unrelated dataset keeps the hash, the cache and the sidecar valid. Changing or refitting this dataset's scaler raises.
F3, F4
Fixed
Nits below.
Tests
248 passed, 0 failed, 0 skipped
16 PR test files on the A40, at both heads.
F5 (data, not code): the ER/mito references were fit on GT CP computed from clipped GT
8 A549 GT CP datasets have blocks the current code can't reproduce:
mito: tomm20 denv, denv-lite, mock and mock-lite (1 each).
Cached values match CP of the clipped GT to 7.6e-15. The clean GT differs by 1.4–1.7.
This is the in-place clipping fixed in a5b1547. When segmentation ran first on a cold cache, the GT CP cached in that run was computed from the clipped view. Masks are unaffected.
The other 26 datasets reproduce (t0 and t1 scanned everywhere).
Consequence:
the er/mito scalers encode clipped GT for those datasets;
evals that reuse the cache score against wrong GT CP;
any force_recompute.gt_cp eval rewrites the cache, then fails the gate, and blocks the dataset until a --force rebuild.
--verify can't catch this, because it re-reads caches and never recomputes.
Required before the re-eval campaign, not before the merge:
Re-cache GT CP for every ER and mito dataset, all timepoints, full and lite. The t0/t1 scan found 8 affected, but only a full recompute proves the rest clean.
Rebuild er.json and mitochondria.json with --force.
Show that --verify, plus a recompute spot check, pass.
Nits
F6: the README's "a real value change fails" overstates the gate. It is a moments gate: within-feature cell swaps pass, and so do changes below about n_cells·1e-6 σ. Also list cell count and positions among what the per-dataset hash covers.
F7: the CPU-vs-GPU full-dataset Δ is 4.3e-8, 23× under the 1e-6 tolerance. The code comment claiming "≥ 3 decades" is wrong; correct it. The margin itself is fine.
F8: the 23109df docstring says --verify checks the position set; it doesn't. Fix the text or the check.
F4 leftover: the name check is skipped under forced gt_cp combined with limit_positions.
Merge once Copilot is clean and CI is green. The nits can go in a follow-up.
… blind spots
Review nits F6-F8: the README overclaimed that any real value change
fails and omitted the position set and cell count from the binding; the
GT_MOMENT_RTOL comment claimed three decades of margin where a CPU-vs-GPU
recompute leaves ~23x; the builder docstring still said the eval hashes
the stacked cells.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
binding_sha256 fingerprints only the GT row count and per-feature mean/std, while the recorded gt_matrix_sha256 is omitted. A changed canonical GT matrix can preserve all univariate moments (for example, a row reordering or changed cross-feature pairing), so check_gt_cells() and final-cache validation accept it; this can reuse dataset metrics or change aligned per-row cosine without invalidating the stamp. Include a canonical matrix fingerprint in the binding/content gate, using a tolerance-aware representation if exact GPU jitter must remain acceptable.
…cosine (#501)
* feat(dynacell/eval): record a checked CP z-clip and its GT survey in the reference
The shared-space z will be clipped before KID/FID/cosine, so the clip
value has to be justified by the GT and never touch it. A read-only
survey of every GT cell of all 4 registry references (lite included),
each in its own test set's scaler, gives a global max |z| of 26.55 (er,
a549-mantis-sec61b-denv, kurtosis). The per-target p99.99 is <= 14.7.
CP_Z_CLIP = 50 leaves a 1.9x margin.
fit_cp_reference re-runs that survey on every build. It records z_clip
and the survey summary (global max, p99.99, max dataset/feature) in the
hashed criteria, and each dataset's own max/p99.99/feature in its fit
record. It refuses to build if any GT |z| exceeds the clip, so the
margin is a checked invariant. The per-dataset binding includes z_clip
but not the target-wide survey, so adding a dataset still leaves other
bindings unchanged. Schema 6.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
* fix(dynacell/eval): clip the shared CP space to +-z_clip before KID/FID/cosine
Through the cubic KID kernel, one heavy-tailed predicted feature set
the metric's magnitude. On the post-#499 nucleus-lite pilot, pred
glcm_ASM sat at z up to 239 in the iPSC GT scaler, driving A549->iPSC
Dataset_CP_KID to 8e5-4e9 while in-domain was 5-57. As the paper owner
decided, the reference-transformed z is now clipped to +-z_clip on
both sides (DatasetCPSpace.transform_clipped), at dataset level and
per row, and in the lite tools that mirror the pipeline. The
cross-condition probe (mask + MADScaler) and the linear probe
(masked, unscaled) do not use the transformed space and are unchanged.
Tests:
- Without any |z| > c, the staged arrays and KID/FID/cosine are
bit-identical to the unclipped space.
- A pred cell at z=1000 scores exactly like one at the clip. Unclipped
it moves KID by > 1e3x the capped change.
- The per-row inputs equal the dataset-level staged rows.
Removing the clip from the pipeline fails the last two.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
* feat(dynacell/eval): report the fraction of pred CP cells the z-clip touches
Clipping discards magnitude, so how much it discarded is recorded next
to the metrics it affects:
- Dataset_CP_clip_frac (feature_metrics.csv, beside Dataset_CP_KID):
the fraction of pred CP cells with any kept feature beyond +-z_clip.
- CP_clip_frac: the same per (FOV, t) row. NaN without pred cells.
- The sidecar's "clip" block: z_clip, the dataset fraction and the
per-feature fraction of pred cells clipped.
DatasetCPSpace.clip_fraction computes all three from the unclipped
transform. _final_metrics_cache_valid refuses a CP feature cache
lacking either column, since those rows were scored unclipped. Tests:
an e2e case whose checkerboard cells leave the band checks all three
fractions against a direct count, and a cache missing either column is
recomputed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
* docs(dynacell/eval): document the CP z-clip and its clip-fraction columns
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
* fix(dynacell/eval): clip CP z at 20 with a per-dataset GT clip-fraction bound
The paper owner set c = 20, based on an offline sweep over c in {5, 8,
10, 15, 20, 30, 50, none} on the nucleus-lite pilot. The cross-domain
model order is identical for all c >= 20, and the in-domain order holds
at every c except one near-tie (zikv, c = 8). At c = 20 some GT cells
are clipped (3 of 36,019 registry GT cells; max |z| 26.55), so "no GT
cell clipped" no longer holds as the builder invariant.
- CP_Z_CLIP = 20.
- The builder now refuses when any dataset's gt_clip_frac exceeds
CP_GT_CLIP_FRAC_MAX = 1e-3. gt_clip_frac is the fraction of that
dataset's GT cells with any |z| > c in its own scaler, lite included.
The highest measured value is 9.5e-4 (er,
a549-mantis-sec61b-denv-lite: 1 of 1054).
- Each dataset records gt_clip_frac, max |z| and its feature. The
hashed criteria record c, the bound, a rank_stability note and the
survey summary (now including max_gt_clip_frac).
- Tests: a dataset beyond the bound is refused; one GT cell beyond the
clip in 5000 (2e-4) is accepted and recorded.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
* docs(dynacell/eval): document c = 20, the GT clip bound and the KID-magnitude caveat
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
* fix(dynacell/eval): enforce the GT clip-fraction bound on non-lite datasets only
With the 1e-3 bound on a ~1000-cell lite set, one extra GT cell beyond
the clip bricks every build of the target. Lite GT cells are a subset
of the parent's, scored in the parent's scaler, so a lite set's
clipped-cell count is at most its parent's. The enforced parent bound
therefore covers it, and the lite fraction is only small-sample noise.
Lite datasets still record gt_clip_frac and max |z|, now marked
`enforced: false`, and the survey's max_gt_clip_frac covers enforced
datasets only.
Test: a lite set at 2e-3 builds while its parent at 2e-4 passes the
bound; once the parent is over the bound, the build refuses.
Re-enforcing lite fails the test. README states why lite is exempt.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
* docs(dynacell/eval): state up front that the GT clip bound is enforced on full test sets only
The README said the bound applied to every test set, lite included, and
only later that lite sets are exempt.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
* fix(dynacell/eval): write the CP sidecar's clip fractions as strict JSON
With no finite pred CP rows the clip fractions are NaN, which json.dumps
wrote as a bare NaN token that strict JSON parsers reject. Record null
instead and dump with allow_nan=False.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
* fix(dynacell/eval): summarize the GT z-clip survey over full test sets only
The survey pooled lite rows, which are subsets of their parents'. So
"3 of 36,019 GT cells beyond 20" counted the er denv outlier twice
(denv and denv-lite), and the quoted 9.5e-4 was an unenforced lite
value. The criteria survey summary now covers full (non-lite) test
sets only: max |z|, p99.99, max dataset/feature, the enforced
max_gt_clip_frac, n_cells and n_cells_beyond_clip. The registry gives
2 of 23,970 distinct GT cells beyond 20, with an enforced max of 5.3e-4
(er a549-mantis-sec61b-denv and -mock). The per-dataset records, lite
included, are unchanged. The README and the CP_Z_CLIP comment quote the
corrected numbers, with lite 9.5e-4 only as the unenforced record. A
test checks that a lite copy of a parent outlier is not counted twice.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
* test(dynacell/eval): pin z_clip and the GT clip bound in every CP binding
The hashed-criteria test checked the whole-reference payload_sha256,
not binding_sha256, the stamp eval reuse compares (#501 review). So
dropping z_clip or gt_clip_frac_max from the binding passed. The new
test changes each in criteria, re-hashes, and asserts binding_sha256
moves for every dataset, lite included.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
* test(dynacell/tools): make the lite tools' CP tests see the z-clip
The lite tools' test helper spelled out the CP transform without the
clip, and no fixture exceeded c. So reverting lite_checks or
lite_subset_sim to the unclipped transform passed (#501 review). The
helper now clips with the reference's z_clip. Both fixtures add a few
pred cells far beyond the clip, and assert the unclipped KID diverges
by > 10x.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
* fix(dynacell/eval): refuse a CP reference whose GT-only mask keeps no feature
An all-constant GT pool left keep_mask empty, and the clip survey then
failed on an argmax over an empty array with an unhelpful error.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
---------
Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…#516)
_import_pipeline_with_stubs replaces dynacell.evaluation.metrics with a
module listing only the names pipeline imports. #499 added a
CP_FEATURE_NAMES_BY_VERSION import in pipeline_cache and #500 added a
round_device_dependent_cp_columns import in cp_reference. The stub
lacked both, so the test raised ImportError whenever it ran before any
other test had imported the real modules -- alone, or first on an xdist
worker. Every top-level metrics import in the package is now stubbed.
Claude-Session: https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA
Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The GLCM+/CP feature metrics (KID, FID, cosine, at both dataset level and per row) were scored in a feature space that depended on the model.
select_features(gt, pred)ran once per eval dir, on GT and prediction cells pooled together. In the paper scope, each (organelle, test set) had 6–8 distinct subsets across its 8 model cells, keeping 7–14 of 22 features._zscore_per_sidestandardised prediction and GT each by its own mean and std. That hid every per-feature offset or scale gap, including uniform over-smoothing. The per-(FOV, t) path_cp_dropzero_zscoredid the same.Deep extractors were never affected.
Fix
DATA_ROOT/cp_reference/<target>.json:lite_of.cp_features.built_at, and the GT-matrix sha256.build_cp_reference.py --verifyrecomputes the GT-matrix hashes.metrics_provenance.jsonfrom the reference actually scored._final_metrics_cache_validrejects a different hash when feature metrics are on. Dirs with feature metrics off are untouched.cross_condition_probe(its per-plate MAD stays deliberate) and the lite tools (lite_checks,lite_subset_sim).select_featuresand the per-side z-score are deleted.tools/build_cp_reference.pyreads existing GT CP caches only. It writes atomically and needs--forceto replace a reference with a different hash.Rollout
--verify(exit 0):59726576…f4b60510…ba687e63…4cf6c089…Tests
packages/*/srcandapplications/dynacell/srcfirst on PYTHONPATH:2 failed, 3569 passed, 164 skipped, 5 deselected.test_load_eval_models_pins_morphem_to_the_configured_revisionandtest_check_cubic_pin_rejects_a_mismatch) also fail on the base commit and pass in isolation, so they are order-dependent.pred = 0.5*GT + 7): the old transform gives cosine 1.0 and KID −0.04, identical to GT scored against itself. The new transform gives cosine 0.41 and KID 8.27.🤖 Generated with Claude Code
https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF