fix(dynacell/eval): invalidate pred caches and metrics when the prediction changes by alxndrkalinin · Pull Request #511 · Biohub/VisCy

alxndrkalinin · 2026-09-30T18:45:13Z

Summary

The dynacell eval's prediction-side cache (io.pred_cache_dir) and the final-metrics reuse gate (_final_metrics_cache_valid) never checked what the prediction contained. A re-predict into the same pred_path silently reused masks, instances, CP and deep features built from the old voxels.

This has now bitten twice: pix2pix3d instance-AP on 2026-08-01, and 29 paper conditions on 2026-09-30.

On 09-25, 25 stores were re-predicted: 24 UNetViT3D A549 stores that predated the tiling fix, plus membrane VSCyto3D A549→iPSC.
The 09-26 re-eval forced only final_metrics. Dice, instance Dice/AP, CP and deep KID came from July caches, while the pixel metrics were fresh.
Signature: in the lite-vs-full comparison, nucleus UNetViT3D iPSC→A549 has PCC max abs diff 0, but Dice median 0.017 and max 0.57.

Those 29 conditions are being recomputed separately, with every pred_* flag forced. This PR removes the cause.

Provenance is now per position, because staleness is per position:

cache.prediction_sources(plate, channel) gives each prediction position a source:
- marker: sha256_12 of the pred channel's completion marker (fix(dynacell): correct resume, cache, inference, and monitoring behavior #497), with provenance-only fields stripped by marker_identity, now public in viscy-utils;
- written_ns: mtime of the channel's first stored chunk, located through zarr's own chunk-key encoding (v2/v3, sharded or not). Later timepoints are tried when t=0 is all fill.
- The chunk date catches code-only re-predicts, which leave the marker unchanged: same checkpoint and settings. The tiling fix was one.
- Metadata mtimes are deliberately ignored, so another channel's marker write, or focus_slice metadata under pred_is_source, does not invalidate.
Every pred-side write records entry["sources"][pos]: masks, instances, CP, the deep-feature FOV path and the DeepFeatureBatcher flush. A read hits only on an exact match; otherwise it is a per-position miss.
- An interrupted rebuild (TIMEOUT or preempt) therefore resumes on the positions it never reached.
- Under require_complete, a stale position raises through the existing miss path.
- _merge_manifests unions positions and sources per position, with memory winning only for positions this process wrote.
Entries written before this PR are upgraded once, eagerly, at context init. A position is kept iff its chunk predates the entry's loaded built_at; an undatable position (blank channel) is rebuilt once.
metrics_provenance.json records prediction_sources_sha256_12, taken before scoring on both the single-leaf and grouped paths. The gate rejects a mismatch, and an old sidecar is dated the same way.
GT side unchanged.

Blast radius on production (measured with this branch's helpers)

Over all 1304 grouped store/cache/save_dir conditions (experiments/2026-09-30_repredict-stale-pred-cache/blast_radius_v4.tsv):

category	conditions
with a pred cache, 0 errors	1044
… would be invalidated	31 stores, 2488 positions
… of those in the independent chunk-level audit's stale set	31 / 31, 0 beyond
saved final metrics invalidated	0
no cache and no sidecar yet (never scored)	31
store missing	229

The 31 are the 29 being recomputed plus 2 spotlight_v2 buckets that already force pred_*. So merging triggers no mass re-eval.

Behavior changes to note

A missing prediction store now raises FileNotFoundError when the pred cache is enabled, at the final-metrics gate, and at save. Before, a deleted store's saved metrics were silently reused.
The one-time legacy upgrade rewrites manifest.yaml, so the cache dir must be writable. Eval cache dirs here are group-writable.
write_metrics_provenance and save_metrics take a required prediction_sources_sha256_12; the three internal callers are updated.

Known residuals (documented in docstrings/README)

Legacy only: metrics written after a re-predict but from stale caches cannot be told apart by date. Today's production audit found exactly these 29, all being recomputed, and grouped leaves force final_metrics: true anyway.
A zarr v2 .-flat store whose chunks lie past a 4096-entry scan window reads as undatable, so it is rebuilt once. The current writer only writes v3.
A store with no markers (485 of 1054) is identified by chunk date alone.

Review

Two independent review passes. Pass 1 found that a store-wide stamp mis-handles interrupted rebuilds (reproduced with the real fov_masks/flush_manifest), code-only re-predicts and multi-channel stores, which led to this per-position design. Pass 2 found the blank-t=0 and late-position cases, both fixed here.
The first blast-radius scan caught two real bugs, both fixed and tested: zarr v2 .zattrs, and chunks rewritten under unchanged metadata.

Tests

New tests/test_evaluation_pred_cache_identity.py, 43 tests over zarr v2 and v3. Each flaw was reintroduced one at a time and fails at least one test:
- interrupted rebuild, code-only re-predict and multi-channel dating;
- metadata-only writes, legacy upgrade and blank channels;
- require_complete, concurrent-writer merge and the digest taken before scoring;
- late positions under exclude_fov_names.
Existing tests updated for the new contract: fake pred paths replaced with real tiny stores, slots filled through the real writer, and scan_nan_microssim_caches_test updated for the new signature.
Separate commit: test_load_eval_models_pins_morphem_to_the_configured_revision now patches by import path. It also failed on base in a full tests/ run, an order dependence on live_pipeline_module().
Local runs, eval venv cpdino-eval-cubic090a2, separate processes:

suite	result
`applications/dynacell/tests`	3509 passed, 171 skipped, 0 failed
`applications/dynacell/src/dynacell/evaluation`	172 passed
`applications/dynacell/tools`	226 passed
CI list (main `.venv`)	2055 passed, 13 failed. All 13 are `cubic.segmentation` ImportErrors from that venv's cubic 0.7.0; the same 13 fail on base.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF

… test load_eval_models imports dynacell.evaluation.utils at call time. Once an earlier test in the same process re-imports dynacell.evaluation.* (live_pipeline_module, first reached via test_evaluation_cp_end_to_end), the module object this file bound at collection time is no longer the one in sys.modules. Patching the attribute on that object then misses. The test failed in a full tests/ run on base 203fb3f too; patching by import path targets whichever module load_eval_models imports. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF

…ction changes The pred-side artifact cache keyed its identity on pred_path and the channel only, and the final-metrics gate ignored the prediction too, so a re-predict into the same path silently reused masks, instances, CP and deep features built from the old voxels. It happened twice. On 2026-09-25, 25 stores were re-predicted; the 09-26 re-eval forced final_metrics only, and 29 paper conditions scored Dice, AP, CP and KID from July caches while their pixel metrics were fresh. Provenance is now per position, since staleness is per position: - prediction_sources() gives each prediction position a source: the sha256_12 of the pred channel's completion marker (#497, provenance fields stripped by the now-public marker_identity), plus the mtime of the channel's first stored chunk, located via zarr's chunk-key encoding. The chunk date catches code-only re-predicts that leave the marker unchanged. Metadata mtimes are ignored, so another channel's marker write does not invalidate. - Every pred-side write records entry["sources"][pos]; a read hits only on an exact match, else it is a per-position miss. An interrupted rebuild resumes on the positions it never reached. _merge_manifests unions sources per position across processes. - Entries written before this are upgraded once at context init: a position is kept iff its chunk predates the entry's built_at, and an undatable (blank) position is rebuilt once. - metrics_provenance.json records prediction_sources_sha256_12, taken before scoring. The gate rejects a mismatch, and an old sidecar is dated the same way. Measured against production (1304 grouped conditions): the upgrade invalidates exactly the 31 stores a chunk-level audit found stale (2488 positions), nothing else, and no saved metrics. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF

Copilot

Copilot review overview

🔵 Needs a closer look

Cache migration, concurrency merging, and filesystem-sensitive Zarr provenance affect production metric correctness and warrant final human review.

Review effort: Balanced
Findings: None

What changed in this PR

Adds prediction-aware provenance to Dynacell evaluation, preventing stale prediction artifacts and final metrics from being reused after in-place re-prediction.

Changes:

Fingerprints prediction sources per position using completion markers and chunk timestamps.
Validates and migrates prediction-side cache manifests, including concurrent writes.
Stamps final metrics with prediction provenance and adds extensive Zarr v2/v3 regression coverage.

File	Description
`packages/viscy-utils/src/viscy_utils/prediction_metadata.py`	Exposes marker identity normalization.
`packages/viscy-utils/tests/test_prediction_metadata.py`	Tests provenance-field stripping.
`applications/dynacell/src/dynacell/evaluation/cache.py`	Implements prediction-source fingerprints.
`applications/dynacell/src/dynacell/evaluation/pipeline_cache.py`	Enforces per-position cache provenance.
`applications/dynacell/src/dynacell/evaluation/pipeline.py`	Validates and stamps final metrics.
`applications/dynacell/src/dynacell/evaluation/provenance.py`	Adds prediction digest to sidecars.
`applications/dynacell/src/dynacell/evaluation/provenance_test.py`	Updates provenance tests.
`applications/dynacell/src/dynacell/evaluation/instance_metrics_test.py`	Updates metrics-save contract.
`applications/dynacell/src/dynacell/evaluation/README.md`	Documents automatic invalidation.
`applications/dynacell/tests/test_evaluation_pred_cache_identity.py`	Adds comprehensive identity regressions.
`applications/dynacell/tests/test_pipeline_cache.py`	Updates cache fixtures and assertions.
`applications/dynacell/tests/test_evaluation_timepoint_guard.py`	Adds prediction provenance fixtures.
`applications/dynacell/tests/test_evaluation_pipeline.py`	Tests final-metrics reuse with provenance.
`applications/dynacell/tests/test_evaluation_extractors.py`	Stabilizes module-path patching.
`applications/dynacell/tests/test_evaluation_cp_scaler_selection.py`	Creates fingerprintable prediction stores.
`applications/dynacell/tests/test_evaluate_compose.py`	Uses a real prediction store.
`applications/dynacell/tools/scan_nan_microssim_caches_test.py`	Updates provenance API usage.
`applications/dynacell/tools/generate_spotlight_v2_eval_configs.py`	Clarifies forced recomputation behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…#514) * perf(dynacell/eval): read and write cache manifests with libyaml The manifest is saved and reloaded after every FOV, and #511's per-position sources made it several times larger. OmegaConf round-trips it through its own node tree plus the pure-Python YAML emitter. PyYAML's C loader and dumper produce byte-identical files and equal dicts. On a real 6-family x 100-position manifest with 600 sources, save+load drops from 0.299 s to 0.022 s. Interpolation resolution is dropped with OmegaConf; no production manifest contains `${`. The pure-Python classes remain the fallback for a PyYAML without libyaml. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF * refactor(dynacell/eval): share one JSON sha256_12 helper in the cache The marker digest, the prediction-sources digest and the encoder-config hash each built the same key-sorted JSON payload and truncated sha256 by hand. json_sha256_12 does it once; digests are unchanged (checked on a real marker, a real 100-position source snapshot and a non-JSON encoder value, which default=str still serializes). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF * refactor(dynacell/eval): drop the flat zarr v2 chunk-key fallback scan The bounded scandir for `.`-separated keys served only flat zarr v2 layouts, and every evaluated non-v3 prediction store uses `/` separators, so nothing reaches it. A flat store now resolves only its exact chunk key; when that chunk is absent the position reads as unwritten, which rebuilds a legacy cache entry over it once. The `/` prefix fallback and the exact-key lookup for every layout stay. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF * refactor(dynacell/eval): read single positions through prediction_sources prediction_source duplicated prediction_sources for one position; a positions= argument covers it. The on-demand read for a position that appeared after the snapshot now takes the store and channel from the manifest's pred identity, which init_cache_context already seeds and validates, so _CacheContext no longer carries its own copy of them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF * refactor(dynacell/eval): date legacy caches and sidecars by one rule The legacy manifest upgrade and the final-metrics gate each spelled out "a position is verified iff its first stored chunk exists and is no newer than the legacy record's time". source_predates states it once; the upgrade checks the missing-built_at case once instead of per position, and drops an isinstance guard on the dict it just wrote. The upgrade also no longer builds per-family identities it discards: _artifact_checks splits into _artifact_entries (kind, label, manifest keys) and _artifact_identity, which only the param check needs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF * refactor(dynacell/eval): check the prediction stamp in the sidecar reader metrics_provenance_matches already reads metrics_provenance.json; the final-metrics gate then read it a second time, without the is_file() guard, to compare the prediction digest. It now takes the current prediction_sources and does both checks from one read, with the same semantics: the stamped digest must match, and an unstamped sidecar is judged by date through source_predates against its mtime. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF * refactor(dynacell/eval): walk manifest entries through one helper The five cache writers each repeated _update_manifest_entry, then _record_position, then mark_manifest_dirty; _record_write does the three. _record_position absorbs _add_position's separate manifest walk and uses _manifest_leaf, and _source_current takes the entry's keys instead of every caller walking to the entry first, which drops its isinstance guards on values that are always dicts. _instance_keys names the instance-label entry once for the writer, instance_cache_hit and the family list. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF * refactor(dynacell/eval): require the written set when merging manifests flush_manifest, the only caller, always passes the positions this process wrote, and the source merge is only correct with them: a defaulted empty set would let the on-disk copy win for positions this process just rebuilt. Making the argument required keeps a future caller from dropping it silently. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF * perf(dynacell/eval): read the prediction sources once per condition evaluate_model and the grouped driver read every position's sources to stamp the metrics, then init_cache_context read them all again for the pred cache. The caller's snapshot now flows through evaluate_predictions and init_cache_contexts into the pred context, which copies it (it adds positions read on demand); the stamp hashes the same snapshot. The final-metrics gate and process-executor workers still read their own. The save_metrics / write_metrics_provenance kwarg becomes prediction_digest: prediction_sources_sha256_12 shadowed the function of that name. The sidecar key is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF * docs(dynacell/eval): correct the prediction-source notes The README still dated a position by its t=0 chunk; it is the earliest stored timepoint since blank outputs store none at t=0. The feature prefetch comment named force_recompute, but the guard is now any position that is rewritten anyway. Comments that restated the _source_current call go, and the legacy-dating docstrings keep their reasoning without the campaign's dates and counts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF * refactor(dynacell/eval): share the store-file predicates in cache tests Four places in the prediction-identity tests re-derived "zarr metadata vs chunk file" by hand; _is_metadata and _chunks now define it once. The pipeline cache-reuse test builds its prediction store with make_hcs_plate like the other eval tests, dropping its own iohub setup. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF * fix(dynacell/eval): write cache manifests exactly as OmegaConf did The plain libyaml SafeDumper differs from OmegaConf.save in three ways that matter here: it leaves a str such as the hex sha12 marker 4e1234567890 unquoted, which OmegaConf's loader (and any tool that reads a manifest with it) parses as a float; it emits &id001 aliases when two entries share one object, as ctx.spacing does; and it escapes non-ASCII text. A dumper subclass now ignores aliases and single-quotes any str that parses as a bool, int or float, as OmegaConf's own representer does, and allow_unicode is set. A new test pins the output byte-for-byte to OmegaConf.save on those cases. Found in the internal review of this branch; all 1224 production manifests were already byte-identical, since none holds such a str yet. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF * fix(dynacell/eval): replace cache manifests atomically save_manifest truncated the manifest in place while init_cache_context reads it without the manifest lock, so an eval starting during another job's flush, or after a writer died mid-dump, read a torn file. OmegaConf turned an empty read into an empty cache; the libyaml loader on this branch raises StaleCacheError instead. Writing to a sibling temp file and renaming it over the manifest leaves readers the old file or the new one. The lock files live under .locks/, so the rename does not disturb locking, and cache dirs are setgid with default ACLs, so the replacement keeps the group and permissions. Found in the internal review of this branch. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF * test(dynacell/eval): pin that on-demand positions stay out of the caller's snapshot The metrics digest is computed from the snapshot the caller read before scoring, and the pred context adds positions that appear later to its own copy. The old test only checked identity (is not); it now adds a position on demand and checks the caller's dict is unchanged. Dropping the dict() copy in init_cache_context makes it fail. Found in the internal review of this branch. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF * fix(dynacell/eval): create the manifest temp file exclusively under a random name The temp name was predictable (host + pid) and opened with "w", which truncates and follows whatever already sits at that path. Cache dirs are group-writable, so another process could plant a symlink there and have the writer truncate a file it points to; a stale temp could also collide after pid reuse. The name is now a uuid4 and the file is opened with "x" (O_CREAT | O_EXCL), which refuses any existing path, symlinks included. On that refusal the path is left alone rather than unlinked. Permissions still come from umask plus the directory's default ACL, unlike mkstemp's fixed 0600. A test plants a symlink at the temp name and checks that save_manifest raises, the target is untouched and the symlink remains; with "w" it does not raise. Addresses Copilot review on #514. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF --------- Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(viscy-utils): add the segmentation-aware and background losses Port the Spotlight-motivated losses from dynacell-models so they live in the shared package before any application consumes them: SegAuxDice (seg_aux.py) with the SAUNA-map and soft-clDice weight maps (seg_aux_maps.py), the segmenter-consistency loss (seg_consistency.py), the Track H background low-pass target (background_lowpass.py), and SpotlightLoss(return_components=True). Defaults keep SpotlightLoss's existing behavior; no package imports an application. Source on dynacell-models: 40de793 (#498), 6c28a31 (#521), 0cbeda1 (#524), 050f553 (#530). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-utils): build foreground masks with CLAHE, median and Otsu Port the Spotlight fg-mask recipe from dynacell-models. generate_fg_masks now binarizes a CLAHE-equalized, 5^3-median-filtered volume at an Otsu threshold picked on that volume after a sigma=5 Gaussian; compute_otsu uses the same full-resolution smoothing, so the otsu_grid_spacing parameter is gone from generate_normalization_metadata and VisCyTrainer.preprocess. The median and Gaussian run through cubic on the GPU when cubic and a device are present, else SciPy; cubic stays an optional import. torch is imported lazily so that light callers of write_meta_field do not pay for it. Behavior change: Spotlight masks and Otsu thresholds written by `viscy preprocess` differ from main's. Source on dynacell-models: 40de793 (#498), d087d96 (#510). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-utils): make prediction writes resumable, identified and write-once Port the HCSPredictionWriter and prediction_metadata work from dynacell-models: - z_reduction="center" and z_planes (slab-only predicts write only the planes they predicted, and record them); - per-FOV completion markers and the checkpoint/settings identity that predict resume and the eval caches key on (prediction_metadata.py); - refuse writes into an existing store when a FOV outside the run has unmarked prediction channels, or when the array would grow; - assemble each volume in memory and write it once on a background thread (bytes unchanged); - raise NotImplementedError when trainer.world_size > 1, instead of letting ranks race on one store. Source on dynacell-models: 2b848a6, 1be4466, 74fa18a (#497), 8e41498 (#511), 996adb4 (#526), 6c83fff (#531), 8e74a7e (#532). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-utils): guard every fit with checkpoint and health callbacks Port the training guardrails from dynacell-models. VisCyCLI now injects, on every `fit`: - a latest-weights ModelCheckpoint that owns last.ckpt (the monitored checkpoint gets save_last=False), so last.ckpt always means the latest epoch, inheriting the leaf's write cadence; - MonitorHealthCheck, which raises when the monitored metric is non-finite or bit-identical for 10 validation epochs; - OptimizerHealthCheck, which raises after 5 consecutive non-finite losses or on GradScaler collapse. Also adds the opt-in LinearLRCooldown callback and raises the CLI subprocess test timeout to 120 s (it timed out on main). Behavior change: this applies to cytoland, dynaclr and dynacell fits. Source on dynacell-models: 20e54db, daa7477, bc0f444, 74fa18a (#497), 92693ae, 2f9ef31 (#528). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): keep SSIM finite under mixed precision; lazy exports Port two viscy-utils changes from dynacell-models: - ssim_25d / ms_ssim_25d: clamp the variances, bound the covariance by Cauchy-Schwarz through a safe sqrt, floor the stability constants symmetrically, and run MS-SSIM outside autocast, so 16-mixed training cannot NaN the gradient. SSIM/MS-SSIM loss values change slightly for MixedLoss users. - viscy_utils/__init__ resolves its exports lazily (PEP 562), keeping torch and lightning off light import paths. Source on dynacell-models: a556bc9 (#484), 130e034, cee4e41, 272051c, 59343ad, 8b2413f (#509). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-data): slab-only and read-once predict, uint8 foreground masks Port the viscy-data side of the dynacell-models predict and Spotlight work: - HCSDataModule.predict_z_planes and SlidingWindowDataset z_planes / read_z_planes, so a 2D model predicts only its focus slab; - cache_volume: with num_workers == 0 predict reads each (position, t) volume once and, when every transform is pointwise, normalizes it once; - foreground masks are read in their stored dtype (uint8) and cast to float on the device in on_after_batch_transfer. Behavior change: every app's predict at num_workers=0 takes the read-once path. Source on dynacell-models: 40de793 (#498), 996adb4 (#526), 6c83fff (#531), 8e74a7e (#532). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-models): per-axis CellDiff patches and mask-conditioning channels Port from dynacell-models: - patch_size may be a per-axis (z, y, x) tuple through the patch embedding, transformer unpatchify and ViT bottleneck, which lets CELLDiffNet run Z-preserving at Z=1 for the 2D track (an int is still accepted); - CELLDiffNet(cond_channels=1) adds extra conditioning input channels (default keeps the old single-channel net). Source on dynacell-models: 0126fe6, 0617ed7, 40de793 (#498). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(viscy-models): non-uniform ODE time grids for CellDiff sampling Port the time_schedule option of the CellDiff ODE integrators from dynacell-models: uniform (default, unchanged), cosine, or a Beta(p, q) grid packed toward the data end, which is what lets the DynaCell-lite recipe sample in 14 velocity evaluations. scipy joins the celldiff extra for betaincinv. Source on dynacell-models: 19765e7 (#520). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-models): add the MorphEm feature extractor Port MorphEmModel from dynacell-models, the fifth deep-feature extractor of the DynaCell evaluation. transformers is imported lazily and the trust_remote_code model is pinned to a Hub commit. Source on dynacell-models: d3f6422 (#479), 4918d2e, 13cb336, c9c6914. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-models): let the PatchGAN discriminator preserve the Z axis Port preserve_z from dynacell-models, so the 3D PatchGAN runs on Z=1 inputs for pix2pix2d (default off, unchanged). Source on dynacell-models: c2d0481. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-models): UNet3D downsample_z for FNet-2D; reject non-5D input Port from dynacell-models: - UNet3D(downsample_z=True): False keeps Z through the encoder, which is how FNet2D runs on single planes (default unchanged); - UNet3DBase.forward raises on non-5D input, and a comment flags the ConvTranspose checkerboard artifact. Source on dynacell-models: a556bc9 (#484), 83a1d69, 16b1d31. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-transforms): stop BatchedRandAffined blanking singleton axes Port 9640537 from dynacell-models: a size-1 spatial axis (Z=1 for 2D models) gets an identity row in the affine, instead of being resampled off the volume and zeroed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(viscy-transforms): mark the sampled normalizers pointwise NormalizeSampled and MinMaxSampled set pointwise = True, which lets the read-once predict path in viscy-data normalize a whole volume once instead of once per window. Source on dynacell-models: 8e74a7e (#532). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(viscy-transforms): make the Gaussian-noise statistics test deterministic Port 3409560 (#513) from dynacell-models. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build: parallel test runs with pytest-xdist Port dfcf9a5 (#512) from dynacell-models: pytest-xdist joins the test group, and a root conftest.py pins OMP/MKL/OpenBLAS to one thread under each xdist worker. Without the pin every worker's torch sizes its pool to all cores, which made `-n 11` 10x slower than it should be and timed out subprocess tests. CLAUDE.md documents the usage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: register pytest markers at the root and make them strict Port the marker half of c50300f (#489) from dynacell-models. strict_markers = true turns an unregistered marker into a collection error, so a typo'd marker can no longer silently decorate nothing. hpc_integration moves from the cytoland and dynaclr conftests to the root: an addinivalue_line registration only holds when that conftest is in the invocation's ancestry. dynaclr drops its own testpaths so the root ini applies. requires_data is registered too: main's applications/dynacell/tools/generate_grouped_eval_configs_test.py uses it, and strict markers would otherwise drop that module at collection. (dynacell-models no longer uses the marker; the dynacell PR that follows removes the registration with it.) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build: put the dynacell hf_demo on the pytest pythonpath Port the root half of 7b7cb43 (#523) from dynacell-models so the HF demo's tests import its app module. The directory already exists on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(dynaclr): join annotations on the full fov_name Port d1ea492 from dynacell-models. The orchestrated linear-classifier fixture joined on the first fov_name component, which failed 7 tests on main once pytest collects dynaclr's colocated *_test.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(cytoland): correct the Spotlight recipe comments Port the cytoland recipe comment fixes from 40de793 (#498): the preprocess flags take explicit values (`--compute_otsu true --compute_fg_masks true`), fg_threshold and nonzero_threshold are inert while fg_mask_key is set, and the Otsu-centred target is what makes 0 the FG/BG boundary. Comments only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(cytoland): install the VCP tutorials' unpublished deps from GitHub Port the tutorial fix from d3f6422 (#479): cytoland and viscy-utils are not on PyPI yet, so the tutorials install them from the GitHub monorepo; their siblings resolve from PyPI. Adds the jupytext-paired notebooks (no outputs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: cut CLAUDE.md content the code or global memory already states Port 93ac234 from dynacell-models: drop the repo-layout tree, setup and common-command blocks, lint/docstring rules (pyproject.toml is the source) and the generic coding-philosophy sections. The DynaCell job runbook from the same file arrives with the dynacell PR, since it documents tools that only exist there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: run the viscy-utils tests in their own job viscy-utils had no CI job on main or on dynacell-models: four of its test files ran inside the dynacell job and the rest (losses, metrics, meta_utils, optimizer health, CLI, lazy imports) ran nowhere. Mirror the viscy-data job on ubuntu-latest for Python 3.12 and 3.13, and make the gate job wait for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(dynacell): run the launcher, engine and prediction-writer suites Port the test-dynacell-configs job from dynacell-models, minus the eight test files that only arrive with the dynacell PR: sync viscy-utils' test group with dynacell, and add test_submit_benchmark_batch, test_engine, test_pipeline_cache, feature_metrics_test and the viscy-utils prediction-writer, prediction-metadata, monitor-health and LR-cooldown tests. The dynacell-models PR target is not carried over. Source on dynacell-models: 33d2fee, 74fa18a (#497), 2d1ed32 (#506), 4450efd (#518), 2798dee (#519), 2f9ef31 (#528). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build: re-lock against the ported packages and main's dynacell Start from dynacell-models' uv.lock and re-resolve with `uv lock` against this tree, whose applications/dynacell/pyproject.toml is still main's. Against dynacell-models' lock this drops the ten dynacell-only packages (dinov3, etils, ftfy, jsonpath-rw, mlcroissant, pandas-stubs, ply, rdflib, submitit, termcolor) and resolves cubic back to 0.7.0; the dynacell PR restores them. Against main's lock it adds pytest-xdist and moves boto3/botocore 1.43.0 -> 1.43.20, cellpose 4.1.1 -> 4.2.1.1, pyarrow 19.0.1 -> 24.0.0, s3transfer and tracksdata; torch and lightning are unchanged. `uv lock --check` exits 0 (main's own lock does not). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): declare torchvision, which evaluation.metrics imports viscy_utils.evaluation.metrics imports torchvision.ops at module level, and losses.MixedLoss imports that module, but viscy-utils never listed torchvision. Installs only worked because another workspace member pulled it in. The new test-utils CI job syncs viscy-utils alone and failed to collect six test modules with ModuleNotFoundError. The lock gains only the dependency edge; no version moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(dynacell): keep main's batch-launcher test and the writer tests out Two corrections to the test-dynacell-configs job: - main's tests/test_submit_benchmark_batch.py resolves leaves under /hpc and fails on a runner (5 PermissionErrors in CI). The dynacell-models version that runs there arrives with the DynaCell PR, which adds it back. - the four viscy-utils test files (prediction writer, prediction metadata, monitor health, LR cooldown) now run in the test-utils job on both Python versions, so the dynacell job no longer repeats them or syncs viscy-utils' test group for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(viscy-utils): share the fit-callbacks lookup between CLI injectors _inject_checkpoint_guardrails and _inject_optimizer_health_guard each repeated the same walk from the config to fit's trainer.callbacks plus the already-injected check; the `else config` branch of that walk could never run behind the `subcommand != "fit"` return. Both now call _fit_callbacks_without. _callback_field collapses its two identical branches, and the monitored checkpoint's init_args are kept from the search loop instead of being looked up twice. No behavior change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(viscy-utils): drop MonitorHealthCheck's unused check_finite flag Nothing sets check_finite (the CLI injects monitor and patience only, and no config or test passes it), so the callback always checks finiteness; it now does so with math.isfinite instead of building a tensor for one float. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(viscy-utils): stop describing fg masks as built from Otsu thresholds generate_fg_masks does not read the stored otsu_threshold; it binarizes a CLAHE-equalized, median-filtered volume. Its summary line and VisCyTrainer.preprocess's compute_fg_masks entry still said "from Otsu thresholds". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build: leave the hf_demo pytest path to the DynaCell PR Nothing on this branch imports from applications/dynacell/examples/ hf_demo: its only test, build_demo_data_test.py, arrives with the DynaCell PR, which adds the pythonpath entry back with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): guard a fit that sets no callbacks Trainer.callbacks defaults to None, and the CLI injectors returned early unless they found a list, so a fit that set no callbacks ran without OptimizerHealthCheck, the guard meant for every fit. The shared lookup now gives such a fit an empty list to append to (None and [] mean the same to Trainer). The checkpoint guardrails still need a monitored ModelCheckpoint and remain a no-op there. Reported by Copilot on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(viscy-utils): say what the AMP skip counter actually counts The counter resets only when GradScaler grows its scale, so successful optimizer steps can sit between the reductions it counts; the error message claimed every step was skipped "in a row". Describe it as reductions without growth in between, which is what it detects: the backward overflowing faster than growth_interval clean steps can restore the scale. Behavior unchanged. Reported by Copilot on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): resolve viscy_utils.callbacks names lazily VisCyCLI now injects MonitorHealthCheck and OptimizerHealthCheck into every fit by class path through viscy_utils.callbacks, whose __init__ eagerly imported OnlineEvalCallback and with it scikit-learn, which only the eval extra installs. A fit on a base install would therefore fail before training. The package now resolves its exports lazily (PEP 562), like viscy_utils itself. The new test blocks sklearn and imports the health callbacks: the eager __init__ fails it with ModuleNotFoundError, the lazy one passes. Reported by Copilot on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(cytoland): install viscy-data from GitHub in the VCP tutorials The tutorials install viscy-utils from the GitHub monorepo and let the siblings resolve from PyPI. The writer they predict with now reads SlidingWindowDataset.z_planes, which the PyPI viscy-data 0.1.0 does not have (checked in its wheel), so predict would fail with AttributeError in on_predict_start. Install viscy-data from GitHub too, in the .py sources and their paired notebooks. Found by /code-review on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): return an empty CLAHE result when the clip range is zero foreground_mask_from_volume guards a constant volume, but a blank volume with a few hot pixels (under 0.2% of voxels) has equal 0.1/99.9 percentiles too, and equalize_clahe then divided by zero and fed NaN to CLAHE (RuntimeWarning: divide by zero, reproduced on a (20, 64, 64) volume). equalize_clahe now returns zeros there, which the mask's existing flat-histogram guard turns into an empty mask, as for a constant volume. Volumes with any contrast are unchanged. Found by /code-review on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): keep a step-cadence leaf's last.ckpt on its step trigger The injected latest-weights ModelCheckpoint takes over save_last from the monitored one but copied only every_n_epochs (defaulting to 1). A leaf that checkpoints every N steps, typical for max_steps runs with long epochs, would get last.ckpt only at epoch boundaries, so a preempted run resumed from much older weights. The injected callback now copies the monitored callback's every_n_train_steps or train_time_interval when set; epoch-cadence leaves are unchanged. No current leaf combines a monitored checkpoint with a step trigger. Found by /code-review on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): take last.ckpt from every monitored checkpoint The guardrail turned save_last off only on the first monitored ModelCheckpoint. A leaf with two monitored checkpoints (say loss and a validation metric) kept a second last.ckpt writer next to the injected latest-weights callback, so last.ckpt would alternate between "latest" and "that monitor's cadence". All monitored callbacks now give it up; the first still names the monitor, directory and cadence. No current leaf has two monitored checkpoints. Found by /code-review on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): write AnnData zarr without switching pandas string inference off anndata 0.12.6 (capped below 0.12.9, which needs pandas<3) cannot write pandas 3's default Arrow-backed strings to zarr. The workaround was global: write_embedding_dataset set pd.options.future.infer_string = False in the caller's process, and the test conftest set it for the whole session. Tests therefore never ran under the configuration users have, and the other writers (EmbeddingSnapshot's rewrite, annotation convert) worked only after the embedding writer had flipped the option. write_anndata_zarr converts obs/var Arrow strings (columns, categories, index) to object just before writing, reusing the conversion append_to_anndata_zarr already had; all three writers and the tests use it, and both global switches are gone. New tests pin a round trip under the default, that plain write_zarr still fails (so the TODO is dropped deliberately once anndata 0.13 lands), and that the embedding writer leaves the option alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(dynaclr): drop the session-wide pandas string-inference switch dynaclr's pytest_configure set pd.options.future.infer_string = False for the whole pytest session, so in a repo-wide run every package's tests ran with pandas 3's default off (it hid the viscy-utils guard added in the previous commit). split_embeddings flipped it in the caller's process too. Both go; the eleven test writes and the split writer use viscy_utils' write_anndata_zarr instead. The cellanome embedding scripts keep their own switch: each owns its process and has no test to check a change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc --------- Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* feat(viscy-utils): add the segmentation-aware and background losses Port the Spotlight-motivated losses from dynacell-models so they live in the shared package before any application consumes them: SegAuxDice (seg_aux.py) with the SAUNA-map and soft-clDice weight maps (seg_aux_maps.py), the segmenter-consistency loss (seg_consistency.py), the Track H background low-pass target (background_lowpass.py), and SpotlightLoss(return_components=True). Defaults keep SpotlightLoss's existing behavior; no package imports an application. Source on dynacell-models: 40de793 (#498), 6c28a31 (#521), 0cbeda1 (#524), 050f553 (#530). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-utils): build foreground masks with CLAHE, median and Otsu Port the Spotlight fg-mask recipe from dynacell-models. generate_fg_masks now binarizes a CLAHE-equalized, 5^3-median-filtered volume at an Otsu threshold picked on that volume after a sigma=5 Gaussian; compute_otsu uses the same full-resolution smoothing, so the otsu_grid_spacing parameter is gone from generate_normalization_metadata and VisCyTrainer.preprocess. The median and Gaussian run through cubic on the GPU when cubic and a device are present, else SciPy; cubic stays an optional import. torch is imported lazily so that light callers of write_meta_field do not pay for it. Behavior change: Spotlight masks and Otsu thresholds written by `viscy preprocess` differ from main's. Source on dynacell-models: 40de793 (#498), d087d96 (#510). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-utils): make prediction writes resumable, identified and write-once Port the HCSPredictionWriter and prediction_metadata work from dynacell-models: - z_reduction="center" and z_planes (slab-only predicts write only the planes they predicted, and record them); - per-FOV completion markers and the checkpoint/settings identity that predict resume and the eval caches key on (prediction_metadata.py); - refuse writes into an existing store when a FOV outside the run has unmarked prediction channels, or when the array would grow; - assemble each volume in memory and write it once on a background thread (bytes unchanged); - raise NotImplementedError when trainer.world_size > 1, instead of letting ranks race on one store. Source on dynacell-models: 2b848a6, 1be4466, 74fa18a (#497), 8e41498 (#511), 996adb4 (#526), 6c83fff (#531), 8e74a7e (#532). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-utils): guard every fit with checkpoint and health callbacks Port the training guardrails from dynacell-models. VisCyCLI now injects, on every `fit`: - a latest-weights ModelCheckpoint that owns last.ckpt (the monitored checkpoint gets save_last=False), so last.ckpt always means the latest epoch, inheriting the leaf's write cadence; - MonitorHealthCheck, which raises when the monitored metric is non-finite or bit-identical for 10 validation epochs; - OptimizerHealthCheck, which raises after 5 consecutive non-finite losses or on GradScaler collapse. Also adds the opt-in LinearLRCooldown callback and raises the CLI subprocess test timeout to 120 s (it timed out on main). Behavior change: this applies to cytoland, dynaclr and dynacell fits. Source on dynacell-models: 20e54db, daa7477, bc0f444, 74fa18a (#497), 92693ae, 2f9ef31 (#528). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): keep SSIM finite under mixed precision; lazy exports Port two viscy-utils changes from dynacell-models: - ssim_25d / ms_ssim_25d: clamp the variances, bound the covariance by Cauchy-Schwarz through a safe sqrt, floor the stability constants symmetrically, and run MS-SSIM outside autocast, so 16-mixed training cannot NaN the gradient. SSIM/MS-SSIM loss values change slightly for MixedLoss users. - viscy_utils/__init__ resolves its exports lazily (PEP 562), keeping torch and lightning off light import paths. Source on dynacell-models: a556bc9 (#484), 130e034, cee4e41, 272051c, 59343ad, 8b2413f (#509). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-data): slab-only and read-once predict, uint8 foreground masks Port the viscy-data side of the dynacell-models predict and Spotlight work: - HCSDataModule.predict_z_planes and SlidingWindowDataset z_planes / read_z_planes, so a 2D model predicts only its focus slab; - cache_volume: with num_workers == 0 predict reads each (position, t) volume once and, when every transform is pointwise, normalizes it once; - foreground masks are read in their stored dtype (uint8) and cast to float on the device in on_after_batch_transfer. Behavior change: every app's predict at num_workers=0 takes the read-once path. Source on dynacell-models: 40de793 (#498), 996adb4 (#526), 6c83fff (#531), 8e74a7e (#532). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-models): per-axis CellDiff patches and mask-conditioning channels Port from dynacell-models: - patch_size may be a per-axis (z, y, x) tuple through the patch embedding, transformer unpatchify and ViT bottleneck, which lets CELLDiffNet run Z-preserving at Z=1 for the 2D track (an int is still accepted); - CELLDiffNet(cond_channels=1) adds extra conditioning input channels (default keeps the old single-channel net). Source on dynacell-models: 0126fe6, 0617ed7, 40de793 (#498). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(viscy-models): non-uniform ODE time grids for CellDiff sampling Port the time_schedule option of the CellDiff ODE integrators from dynacell-models: uniform (default, unchanged), cosine, or a Beta(p, q) grid packed toward the data end, which is what lets the DynaCell-lite recipe sample in 14 velocity evaluations. scipy joins the celldiff extra for betaincinv. Source on dynacell-models: 19765e7 (#520). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-models): add the MorphEm feature extractor Port MorphEmModel from dynacell-models, the fifth deep-feature extractor of the DynaCell evaluation. transformers is imported lazily and the trust_remote_code model is pinned to a Hub commit. Source on dynacell-models: d3f6422 (#479), 4918d2e, 13cb336, c9c6914. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-models): let the PatchGAN discriminator preserve the Z axis Port preserve_z from dynacell-models, so the 3D PatchGAN runs on Z=1 inputs for pix2pix2d (default off, unchanged). Source on dynacell-models: c2d0481. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-models): UNet3D downsample_z for FNet-2D; reject non-5D input Port from dynacell-models: - UNet3D(downsample_z=True): False keeps Z through the encoder, which is how FNet2D runs on single planes (default unchanged); - UNet3DBase.forward raises on non-5D input, and a comment flags the ConvTranspose checkerboard artifact. Source on dynacell-models: a556bc9 (#484), 83a1d69, 16b1d31. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-transforms): stop BatchedRandAffined blanking singleton axes Port 9640537 from dynacell-models: a size-1 spatial axis (Z=1 for 2D models) gets an identity row in the affine, instead of being resampled off the volume and zeroed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(viscy-transforms): mark the sampled normalizers pointwise NormalizeSampled and MinMaxSampled set pointwise = True, which lets the read-once predict path in viscy-data normalize a whole volume once instead of once per window. Source on dynacell-models: 8e74a7e (#532). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(viscy-transforms): make the Gaussian-noise statistics test deterministic Port 3409560 (#513) from dynacell-models. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build: parallel test runs with pytest-xdist Port dfcf9a5 (#512) from dynacell-models: pytest-xdist joins the test group, and a root conftest.py pins OMP/MKL/OpenBLAS to one thread under each xdist worker. Without the pin every worker's torch sizes its pool to all cores, which made `-n 11` 10x slower than it should be and timed out subprocess tests. CLAUDE.md documents the usage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: register pytest markers at the root and make them strict Port the marker half of c50300f (#489) from dynacell-models. strict_markers = true turns an unregistered marker into a collection error, so a typo'd marker can no longer silently decorate nothing. hpc_integration moves from the cytoland and dynaclr conftests to the root: an addinivalue_line registration only holds when that conftest is in the invocation's ancestry. dynaclr drops its own testpaths so the root ini applies. requires_data is registered too: main's applications/dynacell/tools/generate_grouped_eval_configs_test.py uses it, and strict markers would otherwise drop that module at collection. (dynacell-models no longer uses the marker; the dynacell PR that follows removes the registration with it.) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build: put the dynacell hf_demo on the pytest pythonpath Port the root half of 7b7cb43 (#523) from dynacell-models so the HF demo's tests import its app module. The directory already exists on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(dynaclr): join annotations on the full fov_name Port d1ea492 from dynacell-models. The orchestrated linear-classifier fixture joined on the first fov_name component, which failed 7 tests on main once pytest collects dynaclr's colocated *_test.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(cytoland): correct the Spotlight recipe comments Port the cytoland recipe comment fixes from 40de793 (#498): the preprocess flags take explicit values (`--compute_otsu true --compute_fg_masks true`), fg_threshold and nonzero_threshold are inert while fg_mask_key is set, and the Otsu-centred target is what makes 0 the FG/BG boundary. Comments only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(cytoland): install the VCP tutorials' unpublished deps from GitHub Port the tutorial fix from d3f6422 (#479): cytoland and viscy-utils are not on PyPI yet, so the tutorials install them from the GitHub monorepo; their siblings resolve from PyPI. Adds the jupytext-paired notebooks (no outputs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: cut CLAUDE.md content the code or global memory already states Port 93ac234 from dynacell-models: drop the repo-layout tree, setup and common-command blocks, lint/docstring rules (pyproject.toml is the source) and the generic coding-philosophy sections. The DynaCell job runbook from the same file arrives with the dynacell PR, since it documents tools that only exist there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: run the viscy-utils tests in their own job viscy-utils had no CI job on main or on dynacell-models: four of its test files ran inside the dynacell job and the rest (losses, metrics, meta_utils, optimizer health, CLI, lazy imports) ran nowhere. Mirror the viscy-data job on ubuntu-latest for Python 3.12 and 3.13, and make the gate job wait for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(dynacell): run the launcher, engine and prediction-writer suites Port the test-dynacell-configs job from dynacell-models, minus the eight test files that only arrive with the dynacell PR: sync viscy-utils' test group with dynacell, and add test_submit_benchmark_batch, test_engine, test_pipeline_cache, feature_metrics_test and the viscy-utils prediction-writer, prediction-metadata, monitor-health and LR-cooldown tests. The dynacell-models PR target is not carried over. Source on dynacell-models: 33d2fee, 74fa18a (#497), 2d1ed32 (#506), 4450efd (#518), 2798dee (#519), 2f9ef31 (#528). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build: re-lock against the ported packages and main's dynacell Start from dynacell-models' uv.lock and re-resolve with `uv lock` against this tree, whose applications/dynacell/pyproject.toml is still main's. Against dynacell-models' lock this drops the ten dynacell-only packages (dinov3, etils, ftfy, jsonpath-rw, mlcroissant, pandas-stubs, ply, rdflib, submitit, termcolor) and resolves cubic back to 0.7.0; the dynacell PR restores them. Against main's lock it adds pytest-xdist and moves boto3/botocore 1.43.0 -> 1.43.20, cellpose 4.1.1 -> 4.2.1.1, pyarrow 19.0.1 -> 24.0.0, s3transfer and tracksdata; torch and lightning are unchanged. `uv lock --check` exits 0 (main's own lock does not). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(dynacell): DynaCell v1.1 benchmark configs from dynacell-models Port every DynaCell config path of dynacell-models verbatim: the model leaves (baselines, A549/joint/HEK/brightfield, the 2D track with its seed replicates and warmup-matched refit, the temporal and patch-vs-augmentation ablations, CELL-Diff-R2 and the DynaCell-lite leaves), the grouped and instance-AP eval buckets of the paper models, the shared overlays, train/predict sets and launcher profiles, and the deletion of 155 stale config files. The leaves are generated; review the generators and guard tests in the code commits, not each leaf. Spotlight v2 arm leaves, their four fit overlays and their 231 spotlight_v2_* eval buckets are left out; they follow in their own PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(dynacell): DynaCell v1.1 application code from dynacell-models Port every DynaCell code path of dynacell-models verbatim (226 non-merge commits since 2026-06-16; PRs #479-#530): the evaluation pipeline (cpdino/MorphEm/focus-aware 2D, focus2d slab metrics, foreground-limited metrics, MicroMS3IM, scale-invariant pixel metrics, CP shared space, provenance and cache invalidation), the paths registry, preprocessing/registry/croissant/distribution, the launcher and stall watchdog, the lite benchmark and CELL-Diff fast inference, the 2D track, the HF demo, docs, and the engine, celldiff_wrapper and mask_conditioning modules exactly as on dynacell-models. The three LightningModules keep their Spotlight training hooks (seg_aux, seg_consistency, mask_mode, target_bg_lowpass). Every hook defaults off, the bit-identity tests guard that, and only Spotlight leaves turn them on; two DynaCell-paper 2D checkpoints store seg_aux_weight in their hparams, so a hook-free engine could not resume them. mask_conditioning.py comes along because engine.py and celldiff_wrapper.py import it. The files that import the Spotlight leaf generator are trimmed in the commits that follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build(dynacell): lock the dynacell eval, croissant and preprocess deps Take dynacell-models' uv.lock verbatim, which this tree's pyprojects now match (`uv lock --check` exits 0). Against the shared-packages lock it moves cubic 0.7.0 -> 0.9.0a4 and adds dinov3 (pinned commit), etils, ftfy, jsonpath-rw, mlcroissant, pandas-stubs, ply, rdflib, submitit and termcolor. A fresh `uv lock` would instead float dinov3 to its current HEAD and bump mlcroissant/pandas-stubs, so the verbatim lock keeps the versions the paper evaluations ran with. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: drop the requires_data marker registration Its only user was main's generate_grouped_eval_configs_test.py, which the DynaCell port replaced: the live-data tests there are now marked slow. With strict markers, nothing needs the registration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(dynacell): run the release-path, provenance and watchdog suites Restore the eight dynacell test files the shared-packages PR left out of the test-dynacell-configs job, now that they exist: test_evaluation_microssim_calibration, segmentation_test, test_distribution, test_croissant, test_main_cli_routing, test_evaluation_fid_provenance, provenance_test and watch_stalled_jobs_test. The job now matches dynacell-models'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: add the runbook for jobs that hang while holding an allocation Port the DynaCell job runbook of the root CLAUDE.md from dynacell-models (6193b9c, bcbfbe6, c4c5ed0, 74fa18a #497, e1d75cf): how to tell a finished-but-hung job from healthy compute by the CPU-time derivative, the stall watchdog (tools/watch_stalled_jobs.py), per-family predict and fit wall limits, and the forensics to capture before cancelling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(dynacell): scope the grouped eval walk to registered model dirs The grouped generator imported ARMS from the Spotlight-v2 leaf generator, which is not part of this PR, to skip the Spotlight arms that share the canonical prediction tree. Drop that import and the Spotlight model names from _CODE_TO_PAPER, _DETERMINISTIC_MODELS and _CELLDIFF_MODELS, keeping the four 2D seed replicates the 2D appendix scores. The shared tree still holds the arms' stores, and parse_zarr_name raises on an unknown model token (e.g. the CellDiff "variant" segaux_iterative). walk_predictions now skips any model dir outside _REGISTERED_MODEL_DIRS before parsing and prints how many stores it skipped. The test that pinned the Spotlight skip becomes one that pins the unregistered-dir skip and its count. evaluation/paths.py keeps its Spotlight names: without them canonical_model_name would silently resolve an arm's run dir onto its baseline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(dynacell): drop the Spotlight leaf entries from the config guards The Spotlight arm leaves are not part of this PR, so: - test_benchmark_config_composition: drop the eight v1 *_spotlight train leaves, their iPSC predict leaves and data-hparam entries, the _UNMIGRATED_MODELS filter that only they needed, and the Spotlight members of _WIDE_GPU_TRAIN_LEAVES and _LONG_WALL_TRAIN_LEAVES (the 2D pix2pix seed1 replicate keeps its wide-GPU opt-out); - test_paths: drop the two tests that import the Spotlight leaf generator or depend on the grouped generator registering the arms. The canonical_model_name / resolve_model tests of the arm names stay, since evaluation/paths.py keeps those names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(dynacell): describe the 2D seed1 leaves without the Spotlight generator The 40 leaves of the four 2D baseline seed replicates were emitted by the Spotlight-v2 leaf generator, which is not part of this PR, and their headers told the reader to edit that generator. Say instead what they are for: the second draw scored by the *__2d_focus2d_seed1 buckets. Comments only; the configs are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(dynacell): drop the Spotlight eval generator from the generator lists evaluation/README.md and the hand-written-bucket guard's comment in test_evaluate_compose.py named generate_spotlight_v2_eval_configs.py among the generators that write grouped buckets; it is not part of this PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): declare torchvision, which evaluation.metrics imports viscy_utils.evaluation.metrics imports torchvision.ops at module level, and losses.MixedLoss imports that module, but viscy-utils never listed torchvision. Installs only worked because another workspace member pulled it in. The new test-utils CI job syncs viscy-utils alone and failed to collect six test modules with ModuleNotFoundError. The lock gains only the dependency edge; no version moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(dynacell): keep main's batch-launcher test and the writer tests out Two corrections to the test-dynacell-configs job: - main's tests/test_submit_benchmark_batch.py resolves leaves under /hpc and fails on a runner (5 PermissionErrors in CI). The dynacell-models version that runs there arrives with the DynaCell PR, which adds it back. - the four viscy-utils test files (prediction writer, prediction metadata, monitor health, LR cooldown) now run in the test-utils job on both Python versions, so the dynacell job no longer repeats them or syncs viscy-utils' test group for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(viscy-utils): share the fit-callbacks lookup between CLI injectors _inject_checkpoint_guardrails and _inject_optimizer_health_guard each repeated the same walk from the config to fit's trainer.callbacks plus the already-injected check; the `else config` branch of that walk could never run behind the `subcommand != "fit"` return. Both now call _fit_callbacks_without. _callback_field collapses its two identical branches, and the monitored checkpoint's init_args are kept from the search loop instead of being looked up twice. No behavior change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(viscy-utils): drop MonitorHealthCheck's unused check_finite flag Nothing sets check_finite (the CLI injects monitor and patience only, and no config or test passes it), so the callback always checks finiteness; it now does so with math.isfinite instead of building a tensor for one float. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(viscy-utils): stop describing fg masks as built from Otsu thresholds generate_fg_masks does not read the stored otsu_threshold; it binarizes a CLAHE-equalized, median-filtered volume. Its summary line and VisCyTrainer.preprocess's compute_fg_masks entry still said "from Otsu thresholds". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build: leave the hf_demo pytest path to the DynaCell PR Nothing on this branch imports from applications/dynacell/examples/ hf_demo: its only test, build_demo_data_test.py, arrives with the DynaCell PR, which adds the pythonpath entry back with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): guard a fit that sets no callbacks Trainer.callbacks defaults to None, and the CLI injectors returned early unless they found a list, so a fit that set no callbacks ran without OptimizerHealthCheck, the guard meant for every fit. The shared lookup now gives such a fit an empty list to append to (None and [] mean the same to Trainer). The checkpoint guardrails still need a monitored ModelCheckpoint and remain a no-op there. Reported by Copilot on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(viscy-utils): say what the AMP skip counter actually counts The counter resets only when GradScaler grows its scale, so successful optimizer steps can sit between the reductions it counts; the error message claimed every step was skipped "in a row". Describe it as reductions without growth in between, which is what it detects: the backward overflowing faster than growth_interval clean steps can restore the scale. Behavior unchanged. Reported by Copilot on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): resolve viscy_utils.callbacks names lazily VisCyCLI now injects MonitorHealthCheck and OptimizerHealthCheck into every fit by class path through viscy_utils.callbacks, whose __init__ eagerly imported OnlineEvalCallback and with it scikit-learn, which only the eval extra installs. A fit on a base install would therefore fail before training. The package now resolves its exports lazily (PEP 562), like viscy_utils itself. The new test blocks sklearn and imports the health callbacks: the eager __init__ fails it with ModuleNotFoundError, the lazy one passes. Reported by Copilot on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(cytoland): install viscy-data from GitHub in the VCP tutorials The tutorials install viscy-utils from the GitHub monorepo and let the siblings resolve from PyPI. The writer they predict with now reads SlidingWindowDataset.z_planes, which the PyPI viscy-data 0.1.0 does not have (checked in its wheel), so predict would fail with AttributeError in on_predict_start. Install viscy-data from GitHub too, in the .py sources and their paired notebooks. Found by /code-review on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): return an empty CLAHE result when the clip range is zero foreground_mask_from_volume guards a constant volume, but a blank volume with a few hot pixels (under 0.2% of voxels) has equal 0.1/99.9 percentiles too, and equalize_clahe then divided by zero and fed NaN to CLAHE (RuntimeWarning: divide by zero, reproduced on a (20, 64, 64) volume). equalize_clahe now returns zeros there, which the mask's existing flat-histogram guard turns into an empty mask, as for a constant volume. Volumes with any contrast are unchanged. Found by /code-review on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): keep a step-cadence leaf's last.ckpt on its step trigger The injected latest-weights ModelCheckpoint takes over save_last from the monitored one but copied only every_n_epochs (defaulting to 1). A leaf that checkpoints every N steps, typical for max_steps runs with long epochs, would get last.ckpt only at epoch boundaries, so a preempted run resumed from much older weights. The injected callback now copies the monitored callback's every_n_train_steps or train_time_interval when set; epoch-cadence leaves are unchanged. No current leaf combines a monitored checkpoint with a step trigger. Found by /code-review on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): skip only PAPER_KEY-known arms in the grouped eval walk The scope filter skipped every model dir outside the generator's registry and only printed a count. That also silenced the loud "unknown model code-name" a new, never-registered model used to raise, so its stores would drop out of every paper bucket without an error. Now the walk skips only dirs that paths.PAPER_KEY knows but this generator does not register (the Spotlight arms on this branch); anything unknown to both still raises in parse_zarr_name, as on dynacell-models. Tests pin both sides. Found by /code-review on #534. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(dynacell): say the instance-AP audit does not share the walk's arm skip audit_prediction_coverage claimed to apply the same skip filters as walk_predictions. It applies the same test-set filter, but on purpose not the walk's skip of PAPER_KEY models the grouped generator leaves unregistered: listing every unregistered model is what it is for. Found by /code-review on #534. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(dynacell): carry the seg-consistency engine and focus-parity tests test_seg_consistency_training.py exercises the seg_consistency hook of DynacellUNet and DynacellGAN with a stand-in segmenter, and test_seg_consistency_focus.py pins the loss's focus-plane rule to dynacell.evaluation.focus. Both test code this PR ships and import nothing from the Spotlight PR, so they belong here rather than with the Spotlight arms. Verbatim from dynacell-models. Found by /code-review on #534. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): take last.ckpt from every monitored checkpoint The guardrail turned save_last off only on the first monitored ModelCheckpoint. A leaf with two monitored checkpoints (say loss and a validation metric) kept a second last.ckpt writer next to the injected latest-weights callback, so last.ckpt would alternate between "latest" and "that monitor's cadence". All monitored callbacks now give it up; the first still names the monitor, directory and cadence. No current leaf has two monitored checkpoints. Found by /code-review on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(dynacell): one model-dir acceptance rule for parse and walk filter _REGISTERED_MODEL_DIRS restated, as a separate set, the rule _split_model_variant uses to accept a dir name, so the two could drift. _match_model_variant is now that rule (None when the dir does not parse); _split_model_variant wraps it and raises the same errors as before, and the walk filter calls it. No behavior change. Found by /code-review on #534. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): share the foreign-model skip between the walk and the audit walk_predictions skipped the Spotlight arms' dirs on the shared tree, but the instance-AP coverage audit still parsed them, so on this branch it reported 468 coverage errors on the live tree instead of dynacell-models' 102, and its docstring claimed it applied the walk's filters. Both now call one predicate, is_foreign_model_dir: a dir whose model paths.PAPER_KEY knows but the grouped generator does not parse. Measured on the live tree, the audit's error set is identical to dynacell-models' (102 entries; the remainder are pre-existing). The new audit test holds one registered, one foreign and one unknown store and expects exactly the unknown one. Found by /code-review on #534 (findings 1, 9, 10). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): keep raising on campaign models the generator cannot parse pix2pix2d_unetvit_wmatch{,_seed1} are in _CODE_TO_PAPER (and in PAPER_KEY) but not in _DETERMINISTIC_MODELS, so their stores do not parse. On dynacell-models the walk raises on them and the audit lists them. The foreign-model predicate would have skipped them as another campaign's arms. A model the generator itself lists in _CODE_TO_PAPER is not foreign: the walk raises and the audit flags it, as on dynacell-models. With this, the audit's error set on the live tree is identical to dynacell-models' (102 entries, compared path by path). Registering wmatch properly is a separate decision (its comment says "evaluate in focused sibling buckets", and adding it to _DETERMINISTIC_MODELS would fold it into the canonical buckets), so it is left as on dynacell-models. Found by /code-review on #534 (findings 2, 7). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(dynacell): stop pointing at the Spotlight leaf generator from this branch evaluation/paths.py and tests/test_paths.py cited tools/generate_spotlight_v2_leaves.py, which this branch does not have, and test_paths claimed a pin to it that only exists with the Spotlight v2 configs. Say where the generator lives instead. The Spotlight model names stay: without them canonical_model_name would resolve an arm onto its baseline. Comments only. Found by /code-review on #534 (finding 8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): parse the warmup-matched pix2pix2d stores in the grouped walk _CODE_TO_PAPER declares pix2pix2d_unetvit_wmatch{,_seed1} a parse-only registration "same rule as fnet2d_voxelmatched", but unlike that model they were never added to _DETERMINISTIC_MODELS. Their 56 stores on the live tree therefore raised "unknown model code-name", which alone stops walk_predictions. Register them like fnet2d_voxelmatched: they now parse, and, as for every parse-only ablation, they are scored in their focused sibling buckets (the committed *__2d_focus2d_pix2pix2d_wmatch leaves), not by regenerating the canonical ones. Fixes part of FOLLOWUPS #1 (live-tree walk crash). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): skip the listed checkpoint re-predicts in the grouped walk The live tree holds second-checkpoint predictions of registered models beside their canonical stores, under train-set dirs paths.py does not know: fnet3d_paper/ipsc_step{8400,11k}, fnet2d_voxelmatched/ipsc_step200k, fcmae_vscyto{2d_scratch_samplematched,3d_scratch}/ipsc_final and fcmae_vscyto3d_scratch/ipsc_repredict (34 stores). key_from_prediction_store raises on them ("unknown train_set token 'ipsc_final'"), which stopped walk_predictions and filled the instance-AP audit. They come from the voxel-matched FNet-2D and sample-matched UNeXt2-2D ablations, are scored only by hand-written experiment buckets, and no committed config references them, so they are not canonical inputs. Each (model dir, train-set dir) pair is now listed in _CHECKPOINT_REPREDICTS with its campaign; the walk skips and counts them and the audit skips them. A train-set dir that is neither canonical nor listed still raises. Fixes part of FOLLOWUPS #1 (live-tree walk crash). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): name the P1 segcons replicate dirs as foreign in the walk nucleus/pix2pix2d_unetvit_segcons_{s1,ctrl_s1,x10} (12 stores, Spotlight P1 follow-ups of 2026-10-19) sit in the canonical tree but are in neither this generator's registry nor paths.PAPER_KEY, so the PAPER_KEY-based foreign test cannot see them and they raised "unknown model code-name". They are now listed by name in _FOREIGN_MODEL_DIRS and skipped like the PAPER_KEY arms; the skip message no longer says "PAPER_KEY". Any other unknown dir still raises. The new audit test holds one store of each kind on the live tree (registered, checkpoint re-predict, named foreign, wmatch, unknown train set) and expects only the unknown one to be flagged. Fixes the rest of FOLLOWUPS #1 (live-tree walk crash). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): list checkpoint re-predicts per organelle _CHECKPOINT_REPREDICTS matched (model dir, train-set dir) in every organelle, so a future mito/fnet3d_paper/ipsc_step8400 store would have been skipped as a known re-predict instead of raising, though only the ER one exists. Entries are now (organelle, model, train set) triples for exactly the stores on the live tree; a listed token in another organelle raises "unknown train_set token" again (new test, fails before this change). The audit test now uses the nucleus step-11k store, since the step-8400 one is ER-only. Found by /code-review on the FOLLOWUPS #1 fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(dynacell): one skip rule for the grouped walk and the audit walk_predictions and audit_prediction_coverage each called is_foreign_model_dir and is_checkpoint_repredict inline, so a third skip category added to one could silently miss the other and split them again. Both now call skip_reason(zarr_path), which names the category; the walk counts per category. No behavior change. Found by /code-review on the FOLLOWUPS #1 fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(dynacell): guard the grouped registries against the walk crash class Two guards on the hand-kept registries behind the live-tree crash: - every _CODE_TO_PAPER model must parse. At a04b3eb this lists pix2pix2d_unetvit_wmatch{,_seed1}, the omission that stopped walk_predictions; - the name-based skip lists stay disjoint from what the campaign registers: no _FOREIGN_MODEL_DIRS entry in _CODE_TO_PAPER or the parse tuples (it would be dropped silently, since the foreign check runs first), and no re-predict token among the canonical train sets. Found by /code-review on the FOLLOWUPS #1 fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(dynacell): reflow the checkpoint re-predict comment The organelle-scoping edit left a broken line wrap in the _CHECKPOINT_REPREDICTS comment. Comment only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): keep the wmatch refit out of canonical and instance-AP leaves Registering pix2pix2d_unetvit_wmatch{,_seed1} for parsing (97c30a1) fixed the live-tree walk, but a default regeneration would then also fold their 56 stores into the canonical {organelle}_{train_set} buckets and the instance-AP leaves, against DON'T-FOLD: the committed *__2d_focus2d_pix2pix2d_wmatch buckets are where they are scored. They are now listed in _SIBLING_ONLY_MODELS: walk_predictions still returns them, so sibling buckets can be built from it, but main() buckets canonical_predictions() (the walk minus that list) and the instance-AP in_scope() drops them. New tests: canonical_predictions leaves them out, in_scope rejects them. MEASURED on the live tree, both generators run the default way (no --out-root) emit 0 wmatch blocks, and the canonical pool (628 stores) is identical, store by store, to what the pre-fix rules parse, so a default regeneration writes exactly what a04b3eb would have written had its walk not crashed. Found by /code-review on the FOLLOWUPS #1 fix (finding 2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(dynacell): retitle the paper and describe DynaCell as the broader framework The camera-ready title is now "Evaluating Dynamic 3D Virtual Staining of Live Cells with DynaCell": DynaCell names a broader framework for image-based profiling of cell-state dynamics, and this paper's evaluation framework is part of it. Update the title in CITATION.cff (software and preferred citation), the README and both HF card BibTeX entries, and the Croissant citeAs string; rewrite the CITATION.cff abstract and the README tagline to describe the evaluation framework as part of DynaCell. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA * docs(dynacell): call the contribution the DynaCell evaluation suite Mirror paper 1c85e84: "This evaluation framework is part of DynaCell, a broader framework" repeated the noun. The paper's contribution is now the DynaCell evaluation suite, and DynaCell stays the broader framework. The CITATION.cff abstract also says "a set of baseline models", so "suite" means one thing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA * docs(dynacell): retitle to The DynaCell Evaluation Suite The paper's title is now "The DynaCell Evaluation Suite for Dynamic 3D Virtual Staining of Live Cells" (Overleaf b73f25f). Updates CITATION.cff, the README and HF card BibTeX, and the Croissant citeAs. test_croissant.py: 17 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA * chore(dynacell): delete finished one-shot tools Each did a one-time job that is done, and nothing else imports them: - build_migration_manifest, run_migration, restandardize_config_paths (+ tests): the canonical-path migration was applied 2026-07-13; a rebuilt manifest on 07-24 had nothing left to move; - retokenize_deconv_predictions (+ test): the 24 a549__deconv / joint__legacy_deconvgt renames were applied 2026-07-22; paths.py still reads both tokens for the dirs kept on purpose; - scan_nan_microssim_caches (+ test): one audit for the cubic 0.9.0a1 NaN MicroMS3IM caches that #506 fixed; - add_instance_ap_to_ablation_leaves: one codemod, already applied; - smoke_joint_leaf: an April CPU smoke the config-composition tests now cover. The provenance builders (cpdino seg masks, HEK stores, A549 dual fuse, random-init checkpoints) stay. README entries and one bake_best_ckpt docstring line follow. Full suite: 5251 passed, 0 failed (the 18 fewer are the two deleted tests/ files; the 14 colocated tool tests were never collected by the root run). Raised in review by edyoshikun. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc --------- Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

alxndrkalinin and others added 2 commits September 30, 2026 11:44

alxndrkalinin requested a balanced review from Copilot September 30, 2026 18:45

Copilot started reviewing on behalf of alxndrkalinin September 30, 2026 18:46

View session

Copilot AI reviewed Sep 30, 2026

View reviewed changes

alxndrkalinin merged commit 8e41498 into dynacell-models Sep 30, 2026
27 checks passed

alxndrkalinin mentioned this pull request Sep 30, 2026

refactor(dynacell/eval): simplify the #511 prediction-provenance code #514

Merged

alxndrkalinin mentioned this pull request Oct 26, 2026

Port shared VisCy package changes from DynaCell v1.1 #533

Merged

Navigation Menu

Repository navigation

fix(dynacell/eval): invalidate pred caches and metrics when the prediction changes - #511

fix(dynacell/eval): invalidate pred caches and metrics when the prediction changes#511
alxndrkalinin merged 2 commits into
dynacell-modelsfrom
fix/pred-cache-invalidate-on-repredict

alxndrkalinin commented Sep 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

alxndrkalinin commented Sep 30, 2026

Summary

Blast radius on production (measured with this branch's helpers)

Behavior changes to note

Known residuals (documented in docstrings/README)

Review

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants