Skip to content

fix(dynacell/eval): drop constant GT slices from the MicroMS3IM calibration pool - #506

Merged
alxndrkalinin merged 3 commits into
dynacell-modelsfrom
fix/microssim-drop-constant-calibration-slices
Sep 29, 2026
Merged

fix(dynacell/eval): drop constant GT slices from the MicroMS3IM calibration pool#506
alxndrkalinin merged 3 commits into
dynacell-modelsfrom
fix/microssim-drop-constant-calibration-slices

Conversation

@alxndrkalinin

@alxndrkalinin alxndrkalinin commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Leaf-level MicroMS3IM calibration now drops constant GT slices from the α fit pool instead of scoring the whole leaf NaN.

A549 TOMM20_mock.zarr holds six all-zero z-slices (fov0006–0008, t=0, z=0–1; 6 of 5760). The seed-42 draw of 12 (FOV, t) volumes includes (fov0008, 0) in every mito mock leaf (all have 12 FOVs × T=10). cubic can't fit α on a zero-range slice, so fit_microssim returned None and 35 eval dirs scored MicroMS3IM = NaN for every FOV. That covers 18 3D dirs, 15 2D dirs, the randinit leaf and the phase-floor leaf. This has been the case since leaf-level calibration landed in June. The live FNet3D reliability table shows "--" for all three A549 mito columns.

  • fit_microssim (evaluation/metrics.py): drop GT slices with max - min == 0 from both pools, then fit on the rest. It returns None only if every slice is constant.
    • A NaN/inf GT pixel is not dropped. It keeps failing the existing normalized-range check, so the leaf is still NaN, as is a degenerate pool normalization.
    • For a leaf without constant slices nothing changes (the arrays are not touched).
    • score_microssim is unchanged: a constant slice still scores 0.0 in its (FOV, t) mean.
  • The mask uses array methods + np. dispatch, so for CuPy input it stays on the GPU (no host round-trip, no xp).
  • tools/scan_nan_microssim_caches.py: remove the "known mito A549 mock" exemption. Those caches are now stale like any other all-NaN cache.
  • applications/dynacell/CLAUDE.md: record cubic's device-agnostic rule (np. / array methods on NumPy and CuPy arrays, xp only to create arrays on a device; cubic.cuda for transfers).

Evidence (real data)

On fnet3d a549-trained mito mock (cubic 0.9.0a2, A40), scored on every t of fov0000 + fov0008:

calibration pool ri_factor mean MicroMS3IM (20 vols)
production (seed 42) fit fails → None NaN
seed 42 minus the 2 constant slices (of 576) 22.19 0.831
seed 43 (draw avoids blank volumes) 21.46 0.834
seed 44 (draw avoids blank volumes) 23.61 0.826

Dropping the slices moves per-volume scores by at most 0.0058 / 0.0115 relative to the two clean draws, which is within the draw-to-draw spread. On the A40, the GPU fit with a dropped slice matches the CPU fit on the kept slices exactly (offset_gt, offset_pred, max_val, ri_factor).

Tests

  • New test_constant_gt_slice_is_dropped_from_calibration: real _calibrate_microssim on 192×192 plates with one zero slice. It asserts a fit is returned whose normalization equals a direct fit on the pool without that slice. It fails on the base code (assert None is not None).
  • test_constant_gt_slice_scores_nan → test_all_constant_gt_slices_score_nan (end-to-end; NaN only when every slice is constant).
  • Local, cpdino-eval-cubic090a2 venv with the worktree on PYTHONPATH:
    • calibration + scan-tool + test_evaluation_metrics.py: 44 passed
    • tools/: 222 passed
    • the CI dynacell list: 1855 passed, 159 skipped, 2 failed
    • tests/test_evaluation_*.py + src/dynacell/evaluation/*_test.py: 337 passed, 4 failed
    • The 6 failures are in files this PR does not touch: 2 × test_submitter_runs_without_optional_evaluation_dependencies (60 s subprocess timeout), test_load_eval_models_pins_morphem_to_the_configured_revision, 3 × provenance_test.py cubic-pin tests. None is caused by this change. Run in isolation on this branch, all 22 of those tests pass. On the unmodified base (e1f5ad8) the submitter timeout also fails once. The 4 evaluation failures only appear when tests/ and src/ run in one process: live_pipeline_module() re-imports dynacell.evaluation.*, and the provenance tests' string-path monkeypatch then lands on a different module object than the functions they imported. CI does not run that combination.

After merge

Recompute the 35 dirs with force_recompute.final_metrics=true, using mock-only sibling buckets for the 6 mito buckets plus the randinit and phase leaves.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF

alxndrkalinin and others added 3 commits September 29, 2026 12:41
@alxndrkalinin @claude
fix(dynacell/eval): drop constant GT slices from the MicroMS3IM calib…
9704edb 
…ration pool

A549 TOMM20_mock.zarr holds six all-zero z-slices (fov0006-0008, t=0,
z=0-1). The seed-42 draw of 12 (FOV, t) volumes includes (fov0008, 0) in
every mito mock leaf, cubic cannot fit alpha on a zero-range slice, and
fit_microssim returned None, so 35 eval dirs scored MicroMS3IM=NaN for
every FOV.

Constant GT slices are now left out of the fit and alpha is fit on the
rest; score_microssim still scores such a slice 0.0. A NaN/inf GT pixel
is not dropped, so it still makes the leaf NaN through the
normalized-range check, as does a degenerate pool normalization. For a
leaf without constant slices nothing changes.

Measured on fnet3d a549-trained mito mock: fitting the same draw minus
the 2 blank slices of 576 gives scores within 0.0115 per volume of two
draws that avoid the blank volumes (mean 0.831 vs 0.834 / 0.826).

The mask uses array methods and np. dispatch, so it stays on the
input's device (matches the CPU fit exactly on an A40).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
@alxndrkalinin @claude
fix(dynacell/tools): stop exempting mito A549 mock from the all-NaN M…
6dba6da 
…icroSSIM scan

The scan reported all-NaN mito A549 mock caches as "known" because their
GT holds all-zero z-slices that made calibration return None. Calibration
now drops those slices, so an all-NaN mito mock cache is as stale as any
other and must be recomputed. The exemption would have hidden them.

Every all-NaN cache is now reported and fails the scan. The docstring also
notes that the fix does not change the provenance stamp, so a mito mock
cache written under 0.9.0a2 before it landed needs --cubic 0.9.0a2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
@alxndrkalinin @claude
docs(dynacell): adopt cubic's np-not-xp rule for device-agnostic eval…
4620bc0 
… code

Eval code runs on NumPy or CuPy arrays depending on use_gpu. cubic's
AGENTS.md already requires calling np. functions and array methods on
either kind of array and reserving get_array_module for creating arrays
on a given device. Record the same rule here, with the corollary that
derived masks stay on the input's device, and name the three older eval
modules that still use xp so they are not copied from.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvkM1gKifWmoDPq8Drd4JF
@alxndrkalinin
alxndrkalinin requested a balanced review from Copilot September 29, 2026 19:47
Copilot AI reviewed Sep 29, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation consistently filters paired slices, preserves non-finite failure behavior, and includes focused regression coverage.

Review effort: Balanced
Findings: None

What changed in this PR

This PR prevents isolated constant ground-truth slices from invalidating leaf-level MicroMS3IM calibration and updates cache scanning, tests, documentation, and CI commentary accordingly.

Changes:

  • Filters constant GT slices and matching predictions before fitting MicroMS3IM.
  • Treats previously exempt A549 mitochondrial mock all-NaN caches as stale.
  • Adds end-to-end coverage and documents NumPy/CuPy device-agnostic array handling.
File Description
applications/​dynacell/​src/​dynacell/​evaluation/​metrics.py Drops constant GT slices before calibration.
applications/​dynacell/​src/​dynacell/​evaluation/​pipeline.py Updates calibration semantics documentation.
applications/​dynacell/​tests/​test_evaluation_microssim_calibration.py Tests mixed and fully constant calibration pools.
applications/​dynacell/​tools/​scan_nan_microssim_caches.py Removes the known-cache exemption.
applications/​dynacell/​tools/​scan_nan_microssim_caches_test.py Updates scanner behavior tests.
applications/​dynacell/​CLAUDE.md Documents device-agnostic array conventions.
.github/​workflows/​test.yml Updates CI test commentary.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@alxndrkalinin
alxndrkalinin merged commit 2d1ed32 into dynacell-models Sep 29, 2026
27 checks passed
@alxndrkalinin alxndrkalinin mentioned this pull request Sep 29, 2026
Merged
alxndrkalinin added a commit that referenced this pull request Nov 3, 2026
@alxndrkalinin @claude
Port shared VisCy package changes from DynaCell v1.1 (#533)
4a4ac9f 
* feat(viscy-utils): add the segmentation-aware and background losses

Port the Spotlight-motivated losses from dynacell-models so they live
in the shared package before any application consumes them:
SegAuxDice (seg_aux.py) with the SAUNA-map and soft-clDice weight maps
(seg_aux_maps.py), the segmenter-consistency loss (seg_consistency.py),
the Track H background low-pass target (background_lowpass.py), and
SpotlightLoss(return_components=True). Defaults keep SpotlightLoss's
existing behavior; no package imports an application.

Source on dynacell-models: 40de793 (#498), 6c28a31 (#521),
0cbeda1 (#524), 050f553 (#530).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(viscy-utils): build foreground masks with CLAHE, median and Otsu

Port the Spotlight fg-mask recipe from dynacell-models.
generate_fg_masks now binarizes a CLAHE-equalized, 5^3-median-filtered
volume at an Otsu threshold picked on that volume after a sigma=5
Gaussian; compute_otsu uses the same full-resolution smoothing, so the
otsu_grid_spacing parameter is gone from generate_normalization_metadata
and VisCyTrainer.preprocess. The median and Gaussian run through cubic
on the GPU when cubic and a device are present, else SciPy; cubic stays
an optional import. torch is imported lazily so that light callers of
write_meta_field do not pay for it.

Behavior change: Spotlight masks and Otsu thresholds written by
`viscy preprocess` differ from main's.

Source on dynacell-models: 40de793 (#498), d087d96 (#510).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(viscy-utils): make prediction writes resumable, identified and write-once

Port the HCSPredictionWriter and prediction_metadata work from
dynacell-models:
- z_reduction="center" and z_planes (slab-only predicts write only the
  planes they predicted, and record them);
- per-FOV completion markers and the checkpoint/settings identity that
  predict resume and the eval caches key on (prediction_metadata.py);
- refuse writes into an existing store when a FOV outside the run has
  unmarked prediction channels, or when the array would grow;
- assemble each volume in memory and write it once on a background
  thread (bytes unchanged);
- raise NotImplementedError when trainer.world_size > 1, instead of
  letting ranks race on one store.

Source on dynacell-models: 2b848a6, 1be4466, 74fa18a (#497),
8e41498 (#511), 996adb4 (#526), 6c83fff (#531), 8e74a7e (#532).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(viscy-utils): guard every fit with checkpoint and health callbacks

Port the training guardrails from dynacell-models. VisCyCLI now
injects, on every `fit`:
- a latest-weights ModelCheckpoint that owns last.ckpt (the monitored
  checkpoint gets save_last=False), so last.ckpt always means the
  latest epoch, inheriting the leaf's write cadence;
- MonitorHealthCheck, which raises when the monitored metric is
  non-finite or bit-identical for 10 validation epochs;
- OptimizerHealthCheck, which raises after 5 consecutive non-finite
  losses or on GradScaler collapse.
Also adds the opt-in LinearLRCooldown callback and raises the CLI
subprocess test timeout to 120 s (it timed out on main).

Behavior change: this applies to cytoland, dynaclr and dynacell fits.

Source on dynacell-models: 20e54db, daa7477, bc0f444, 74fa18a
(#497), 92693ae, 2f9ef31 (#528).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(viscy-utils): keep SSIM finite under mixed precision; lazy exports

Port two viscy-utils changes from dynacell-models:
- ssim_25d / ms_ssim_25d: clamp the variances, bound the covariance by
  Cauchy-Schwarz through a safe sqrt, floor the stability constants
  symmetrically, and run MS-SSIM outside autocast, so 16-mixed training
  cannot NaN the gradient. SSIM/MS-SSIM loss values change slightly for
  MixedLoss users.
- viscy_utils/__init__ resolves its exports lazily (PEP 562), keeping
  torch and lightning off light import paths.

Source on dynacell-models: a556bc9 (#484), 130e034, cee4e41,
272051c, 59343ad, 8b2413f (#509).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(viscy-data): slab-only and read-once predict, uint8 foreground masks

Port the viscy-data side of the dynacell-models predict and Spotlight
work:
- HCSDataModule.predict_z_planes and SlidingWindowDataset z_planes /
  read_z_planes, so a 2D model predicts only its focus slab;
- cache_volume: with num_workers == 0 predict reads each (position, t)
  volume once and, when every transform is pointwise, normalizes it
  once;
- foreground masks are read in their stored dtype (uint8) and cast to
  float on the device in on_after_batch_transfer.

Behavior change: every app's predict at num_workers=0 takes the
read-once path.

Source on dynacell-models: 40de793 (#498), 996adb4 (#526),
6c83fff (#531), 8e74a7e (#532).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(viscy-models): per-axis CellDiff patches and mask-conditioning channels

Port from dynacell-models:
- patch_size may be a per-axis (z, y, x) tuple through the patch
  embedding, transformer unpatchify and ViT bottleneck, which lets
  CELLDiffNet run Z-preserving at Z=1 for the 2D track (an int is still
  accepted);
- CELLDiffNet(cond_channels=1) adds extra conditioning input channels
  (default keeps the old single-channel net).

Source on dynacell-models: 0126fe6, 0617ed7, 40de793 (#498).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(viscy-models): non-uniform ODE time grids for CellDiff sampling

Port the time_schedule option of the CellDiff ODE integrators from
dynacell-models: uniform (default, unchanged), cosine, or a Beta(p, q)
grid packed toward the data end, which is what lets the DynaCell-lite
recipe sample in 14 velocity evaluations. scipy joins the celldiff
extra for betaincinv.

Source on dynacell-models: 19765e7 (#520).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(viscy-models): add the MorphEm feature extractor

Port MorphEmModel from dynacell-models, the fifth deep-feature
extractor of the DynaCell evaluation. transformers is imported lazily
and the trust_remote_code model is pinned to a Hub commit.

Source on dynacell-models: d3f6422 (#479), 4918d2e, 13cb336,
c9c6914.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(viscy-models): let the PatchGAN discriminator preserve the Z axis

Port preserve_z from dynacell-models, so the 3D PatchGAN runs on Z=1
inputs for pix2pix2d (default off, unchanged).

Source on dynacell-models: c2d0481.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(viscy-models): UNet3D downsample_z for FNet-2D; reject non-5D input

Port from dynacell-models:
- UNet3D(downsample_z=True): False keeps Z through the encoder, which
  is how FNet2D runs on single planes (default unchanged);
- UNet3DBase.forward raises on non-5D input, and a comment flags the
  ConvTranspose checkerboard artifact.

Source on dynacell-models: a556bc9 (#484), 83a1d69, 16b1d31.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(viscy-transforms): stop BatchedRandAffined blanking singleton axes

Port 9640537 from dynacell-models: a size-1 spatial axis (Z=1 for 2D
models) gets an identity row in the affine, instead of being resampled
off the volume and zeroed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(viscy-transforms): mark the sampled normalizers pointwise

NormalizeSampled and MinMaxSampled set pointwise = True, which lets the
read-once predict path in viscy-data normalize a whole volume once
instead of once per window.

Source on dynacell-models: 8e74a7e (#532).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(viscy-transforms): make the Gaussian-noise statistics test deterministic

Port 3409560 (#513) from dynacell-models.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build: parallel test runs with pytest-xdist

Port dfcf9a5 (#512) from dynacell-models: pytest-xdist joins the test
group, and a root conftest.py pins OMP/MKL/OpenBLAS to one thread under
each xdist worker. Without the pin every worker's torch sizes its pool
to all cores, which made `-n 11` 10x slower than it should be and timed
out subprocess tests. CLAUDE.md documents the usage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: register pytest markers at the root and make them strict

Port the marker half of c50300f (#489) from dynacell-models.
strict_markers = true turns an unregistered marker into a collection
error, so a typo'd marker can no longer silently decorate nothing.
hpc_integration moves from the cytoland and dynaclr conftests to the
root: an addinivalue_line registration only holds when that conftest
is in the invocation's ancestry. dynaclr drops its own testpaths so
the root ini applies.

requires_data is registered too: main's
applications/dynacell/tools/generate_grouped_eval_configs_test.py uses
it, and strict markers would otherwise drop that module at collection.
(dynacell-models no longer uses the marker; the dynacell PR that
follows removes the registration with it.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build: put the dynacell hf_demo on the pytest pythonpath

Port the root half of 7b7cb43 (#523) from dynacell-models so the HF
demo's tests import its app module. The directory already exists on
main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(dynaclr): join annotations on the full fov_name

Port d1ea492 from dynacell-models. The orchestrated linear-classifier
fixture joined on the first fov_name component, which failed 7 tests
on main once pytest collects dynaclr's colocated *_test.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(cytoland): correct the Spotlight recipe comments

Port the cytoland recipe comment fixes from 40de793 (#498): the
preprocess flags take explicit values (`--compute_otsu true
--compute_fg_masks true`), fg_threshold and nonzero_threshold are inert
while fg_mask_key is set, and the Otsu-centred target is what makes 0
the FG/BG boundary. Comments only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(cytoland): install the VCP tutorials' unpublished deps from GitHub

Port the tutorial fix from d3f6422 (#479): cytoland and viscy-utils
are not on PyPI yet, so the tutorials install them from the GitHub
monorepo; their siblings resolve from PyPI. Adds the jupytext-paired
notebooks (no outputs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: cut CLAUDE.md content the code or global memory already states

Port 93ac234 from dynacell-models: drop the repo-layout tree, setup
and common-command blocks, lint/docstring rules (pyproject.toml is the
source) and the generic coding-philosophy sections. The DynaCell job
runbook from the same file arrives with the dynacell PR, since it
documents tools that only exist there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: run the viscy-utils tests in their own job

viscy-utils had no CI job on main or on dynacell-models: four of its
test files ran inside the dynacell job and the rest (losses, metrics,
meta_utils, optimizer health, CLI, lazy imports) ran nowhere. Mirror
the viscy-data job on ubuntu-latest for Python 3.12 and 3.13, and make
the gate job wait for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(dynacell): run the launcher, engine and prediction-writer suites

Port the test-dynacell-configs job from dynacell-models, minus the
eight test files that only arrive with the dynacell PR: sync
viscy-utils' test group with dynacell, and add test_submit_benchmark_batch,
test_engine, test_pipeline_cache, feature_metrics_test and the
viscy-utils prediction-writer, prediction-metadata, monitor-health and
LR-cooldown tests. The dynacell-models PR target is not carried over.

Source on dynacell-models: 33d2fee, 74fa18a (#497), 2d1ed32 (#506),
4450efd (#518), 2798dee (#519), 2f9ef31 (#528).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build: re-lock against the ported packages and main's dynacell

Start from dynacell-models' uv.lock and re-resolve with `uv lock`
against this tree, whose applications/dynacell/pyproject.toml is still
main's. Against dynacell-models' lock this drops the ten dynacell-only
packages (dinov3, etils, ftfy, jsonpath-rw, mlcroissant, pandas-stubs,
ply, rdflib, submitit, termcolor) and resolves cubic back to 0.7.0;
the dynacell PR restores them. Against main's lock it adds pytest-xdist
and moves boto3/botocore 1.43.0 -> 1.43.20, cellpose 4.1.1 -> 4.2.1.1,
pyarrow 19.0.1 -> 24.0.0, s3transfer and tracksdata; torch and
lightning are unchanged. `uv lock --check` exits 0 (main's own lock
does not).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(viscy-utils): declare torchvision, which evaluation.metrics imports

viscy_utils.evaluation.metrics imports torchvision.ops at module level,
and losses.MixedLoss imports that module, but viscy-utils never listed
torchvision. Installs only worked because another workspace member
pulled it in. The new test-utils CI job syncs viscy-utils alone and
failed to collect six test modules with ModuleNotFoundError. The lock
gains only the dependency edge; no version moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(dynacell): keep main's batch-launcher test and the writer tests out

Two corrections to the test-dynacell-configs job:
- main's tests/test_submit_benchmark_batch.py resolves leaves under
  /hpc and fails on a runner (5 PermissionErrors in CI). The
  dynacell-models version that runs there arrives with the DynaCell
  PR, which adds it back.
- the four viscy-utils test files (prediction writer, prediction
  metadata, monitor health, LR cooldown) now run in the test-utils job
  on both Python versions, so the dynacell job no longer repeats them
  or syncs viscy-utils' test group for them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(viscy-utils): share the fit-callbacks lookup between CLI injectors

_inject_checkpoint_guardrails and _inject_optimizer_health_guard each
repeated the same walk from the config to fit's trainer.callbacks plus
the already-injected check; the `else config` branch of that walk could
never run behind the `subcommand != "fit"` return. Both now call
_fit_callbacks_without. _callback_field collapses its two identical
branches, and the monitored checkpoint's init_args are kept from the
search loop instead of being looked up twice. No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(viscy-utils): drop MonitorHealthCheck's unused check_finite flag

Nothing sets check_finite (the CLI injects monitor and patience only,
and no config or test passes it), so the callback always checks
finiteness; it now does so with math.isfinite instead of building a
tensor for one float.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(viscy-utils): stop describing fg masks as built from Otsu thresholds

generate_fg_masks does not read the stored otsu_threshold; it binarizes
a CLAHE-equalized, median-filtered volume. Its summary line and
VisCyTrainer.preprocess's compute_fg_masks entry still said "from Otsu
thresholds".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build: leave the hf_demo pytest path to the DynaCell PR

Nothing on this branch imports from applications/dynacell/examples/
hf_demo: its only test, build_demo_data_test.py, arrives with the
DynaCell PR, which adds the pythonpath entry back with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(viscy-utils): guard a fit that sets no callbacks

Trainer.callbacks defaults to None, and the CLI injectors returned
early unless they found a list, so a fit that set no callbacks ran
without OptimizerHealthCheck, the guard meant for every fit. The
shared lookup now gives such a fit an empty list to append to (None
and [] mean the same to Trainer). The checkpoint guardrails still
need a monitored ModelCheckpoint and remain a no-op there.

Reported by Copilot on #533.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(viscy-utils): say what the AMP skip counter actually counts

The counter resets only when GradScaler grows its scale, so successful
optimizer steps can sit between the reductions it counts; the error
message claimed every step was skipped "in a row". Describe it as
reductions without growth in between, which is what it detects: the
backward overflowing faster than growth_interval clean steps can
restore the scale. Behavior unchanged.

Reported by Copilot on #533.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(viscy-utils): resolve viscy_utils.callbacks names lazily

VisCyCLI now injects MonitorHealthCheck and OptimizerHealthCheck into
every fit by class path through viscy_utils.callbacks, whose __init__
eagerly imported OnlineEvalCallback and with it scikit-learn, which
only the eval extra installs. A fit on a base install would therefore
fail before training. The package now resolves its exports lazily
(PEP 562), like viscy_utils itself. The new test blocks sklearn and
imports the health callbacks: the eager __init__ fails it with
ModuleNotFoundError, the lazy one passes.

Reported by Copilot on #533.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(cytoland): install viscy-data from GitHub in the VCP tutorials

The tutorials install viscy-utils from the GitHub monorepo and let the
siblings resolve from PyPI. The writer they predict with now reads
SlidingWindowDataset.z_planes, which the PyPI viscy-data 0.1.0 does not
have (checked in its wheel), so predict would fail with AttributeError
in on_predict_start. Install viscy-data from GitHub too, in the .py
sources and their paired notebooks.

Found by /code-review on #533.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(viscy-utils): return an empty CLAHE result when the clip range is zero

foreground_mask_from_volume guards a constant volume, but a blank
volume with a few hot pixels (under 0.2% of voxels) has equal 0.1/99.9
percentiles too, and equalize_clahe then divided by zero and fed NaN to
CLAHE (RuntimeWarning: divide by zero, reproduced on a (20, 64, 64)
volume). equalize_clahe now returns zeros there, which the mask's
existing flat-histogram guard turns into an empty mask, as for a
constant volume. Volumes with any contrast are unchanged.

Found by /code-review on #533.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(viscy-utils): keep a step-cadence leaf's last.ckpt on its step trigger

The injected latest-weights ModelCheckpoint takes over save_last from
the monitored one but copied only every_n_epochs (defaulting to 1). A
leaf that checkpoints every N steps, typical for max_steps runs with
long epochs, would get last.ckpt only at epoch boundaries, so a
preempted run resumed from much older weights. The injected callback
now copies the monitored callback's every_n_train_steps or
train_time_interval when set; epoch-cadence leaves are unchanged. No
current leaf combines a monitored checkpoint with a step trigger.

Found by /code-review on #533.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(viscy-utils): take last.ckpt from every monitored checkpoint

The guardrail turned save_last off only on the first monitored
ModelCheckpoint. A leaf with two monitored checkpoints (say loss and a
validation metric) kept a second last.ckpt writer next to the injected
latest-weights callback, so last.ckpt would alternate between "latest"
and "that monitor's cadence". All monitored callbacks now give it up;
the first still names the monitor, directory and cadence. No current
leaf has two monitored checkpoints.

Found by /code-review on #533.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(viscy-utils): write AnnData zarr without switching pandas string inference off

anndata 0.12.6 (capped below 0.12.9, which needs pandas<3) cannot write
pandas 3's default Arrow-backed strings to zarr. The workaround was
global: write_embedding_dataset set pd.options.future.infer_string =
False in the caller's process, and the test conftest set it for the
whole session. Tests therefore never ran under the configuration users
have, and the other writers (EmbeddingSnapshot's rewrite, annotation
convert) worked only after the embedding writer had flipped the option.

write_anndata_zarr converts obs/var Arrow strings (columns, categories,
index) to object just before writing, reusing the conversion
append_to_anndata_zarr already had; all three writers and the tests
use it, and both global switches are gone. New tests pin a round trip
under the default, that plain write_zarr still fails (so the TODO is
dropped deliberately once anndata 0.13 lands), and that the embedding
writer leaves the option alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc

* fix(dynaclr): drop the session-wide pandas string-inference switch

dynaclr's pytest_configure set pd.options.future.infer_string = False
for the whole pytest session, so in a repo-wide run every package's
tests ran with pandas 3's default off (it hid the viscy-utils guard
added in the previous commit). split_embeddings flipped it in the
caller's process too. Both go; the eleven test writes and the split
writer use viscy_utils' write_anndata_zarr instead. The cellanome
embedding scripts keep their own switch: each owns its process and
has no test to check a change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc

---------

Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
alxndrkalinin added a commit that referenced this pull request Nov 3, 2026
@alxndrkalinin @claude
chore(dynacell): delete finished one-shot tools
d7986fd 
Each did a one-time job that is done, and nothing else imports them:
- build_migration_manifest, run_migration, restandardize_config_paths
  (+ tests): the canonical-path migration was applied 2026-07-13; a
  rebuilt manifest on 07-24 had nothing left to move;
- retokenize_deconv_predictions (+ test): the 24 a549__deconv /
  joint__legacy_deconvgt renames were applied 2026-07-22; paths.py
  still reads both tokens for the dirs kept on purpose;
- scan_nan_microssim_caches (+ test): one audit for the cubic 0.9.0a1
  NaN MicroMS3IM caches that #506 fixed;
- add_instance_ap_to_ablation_leaves: one codemod, already applied;
- smoke_joint_leaf: an April CPU smoke the config-composition tests
  now cover.
The provenance builders (cpdino seg masks, HEK stores, A549 dual fuse,
random-init checkpoints) stay. README entries and one bake_best_ckpt
docstring line follow. Full suite: 5251 passed, 0 failed (the 18
fewer are the two deleted tests/ files; the 14 colocated tool tests
were never collected by the root run).

Raised in review by edyoshikun.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc
alxndrkalinin added a commit that referenced this pull request Nov 4, 2026
@alxndrkalinin @claude
DynaCell v1.1 application (#534)
7286fb2 
* feat(viscy-utils): add the segmentation-aware and background losses

Port the Spotlight-motivated losses from dynacell-models so they live
in the shared package before any application consumes them:
SegAuxDice (seg_aux.py) with the SAUNA-map and soft-clDice weight maps
(seg_aux_maps.py), the segmenter-consistency loss (seg_consistency.py),
the Track H background low-pass target (background_lowpass.py), and
SpotlightLoss(return_components=True). Defaults keep SpotlightLoss's
existing behavior; no package imports an application.

Source on dynacell-models: 40de793 (#498), 6c28a31 (#521),
0cbeda1 (#524), 050f553 (#530).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(viscy-utils): build foreground masks with CLAHE, median and Otsu

Port the Spotlight fg-mask recipe from dynacell-models.
generate_fg_masks now binarizes a CLAHE-equalized, 5^3-median-filtered
volume at an Otsu threshold picked on that volume after a sigma=5
Gaussian; compute_otsu uses the same full-resolution smoothing, so the
otsu_grid_spacing parameter is gone from generate_normalization_metadata
and VisCyTrainer.preprocess. The median and Gaussian run through cubic
on the GPU when cubic and a device are present, else SciPy; cubic stays
an optional import. torch is imported lazily so that light callers of
write_meta_field do not pay for it.

Behavior change: Spotlight masks and Otsu thresholds written by
`viscy preprocess` differ from main's.

Source on dynacell-models: 40de793 (#498), d087d96 (#510).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(viscy-utils): make prediction writes resumable, identified and write-once

Port the HCSPredictionWriter and prediction_metadata work from
dynacell-models:
- z_reduction="center" and z_planes (slab-only predicts write only the
  planes they predicted, and record them);
- per-FOV completion markers and the checkpoint/settings identity that
  predict resume and the eval caches key on (prediction_metadata.py);
- refuse writes into an existing store when a FOV outside the run has
  unmarked prediction channels, or when the array would grow;
- assemble each volume in memory and write it once on a background
  thread (bytes unchanged);
- raise NotImplementedError when trainer.world_size > 1, instead of
  letting ranks race on one store.

Source on dynacell-models: 2b848a6, 1be4466, 74fa18a (#497),
8e41498 (#511), 996adb4 (#526), 6c83fff (#531), 8e74a7e (#532).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(viscy-utils): guard every fit with checkpoint and health callbacks

Port the training guardrails from dynacell-models. VisCyCLI now
injects, on every `fit`:
- a latest-weights ModelCheckpoint that owns last.ckpt (the monitored
  checkpoint gets save_last=False), so last.ckpt always means the
  latest epoch, inheriting the leaf's write cadence;
- MonitorHealthCheck, which raises when the monitored metric is
  non-finite or bit-identical for 10 validation epochs;
- OptimizerHealthCheck, which raises after 5 consecutive non-finite
  losses or on GradScaler collapse.
Also adds the opt-in LinearLRCooldown callback and raises the CLI
subprocess test timeout to 120 s (it timed out on main).

Behavior change: this applies to cytoland, dynaclr and dynacell fits.

Source on dynacell-models: 20e54db, daa7477, bc0f444, 74fa18a
(#497), 92693ae, 2f9ef31 (#528).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(viscy-utils): keep SSIM finite under mixed precision; lazy exports

Port two viscy-utils changes from dynacell-models:
- ssim_25d / ms_ssim_25d: clamp the variances, bound the covariance by
  Cauchy-Schwarz through a safe sqrt, floor the stability constants
  symmetrically, and run MS-SSIM outside autocast, so 16-mixed training
  cannot NaN the gradient. SSIM/MS-SSIM loss values change slightly for
  MixedLoss users.
- viscy_utils/__init__ resolves its exports lazily (PEP 562), keeping
  torch and lightning off light import paths.

Source on dynacell-models: a556bc9 (#484), 130e034, cee4e41,
272051c, 59343ad, 8b2413f (#509).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(viscy-data): slab-only and read-once predict, uint8 foreground masks

Port the viscy-data side of the dynacell-models predict and Spotlight
work:
- HCSDataModule.predict_z_planes and SlidingWindowDataset z_planes /
  read_z_planes, so a 2D model predicts only its focus slab;
- cache_volume: with num_workers == 0 predict reads each (position, t)
  volume once and, when every transform is pointwise, normalizes it
  once;
- foreground masks are read in their stored dtype (uint8) and cast to
  float on the device in on_after_batch_transfer.

Behavior change: every app's predict at num_workers=0 takes the
read-once path.

Source on dynacell-models: 40de793 (#498), 996adb4 (#526),
6c83fff (#531), 8e74a7e (#532).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(viscy-models): per-axis CellDiff patches and mask-conditioning channels

Port from dynacell-models:
- patch_size may be a per-axis (z, y, x) tuple through the patch
  embedding, transformer unpatchify and ViT bottleneck, which lets
  CELLDiffNet run Z-preserving at Z=1 for the 2D track (an int is still
  accepted);
- CELLDiffNet(cond_channels=1) adds extra conditioning input channels
  (default keeps the old single-channel net).

Source on dynacell-models: 0126fe6, 0617ed7, 40de793 (#498).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(viscy-models): non-uniform ODE time grids for CellDiff sampling

Port the time_schedule option of the CellDiff ODE integrators from
dynacell-models: uniform (default, unchanged), cosine, or a Beta(p, q)
grid packed toward the data end, which is what lets the DynaCell-lite
recipe sample in 14 velocity evaluations. scipy joins the celldiff
extra for betaincinv.

Source on dynacell-models: 19765e7 (#520).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(viscy-models): add the MorphEm feature extractor

Port MorphEmModel from dynacell-models, the fifth deep-feature
extractor of the DynaCell evaluation. transformers is imported lazily
and the trust_remote_code model is pinned to a Hub commit.

Source on dynacell-models: d3f6422 (#479), 4918d2e, 13cb336,
c9c6914.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(viscy-models): let the PatchGAN discriminator preserve the Z axis

Port preserve_z from dynacell-models, so the 3D PatchGAN runs on Z=1
inputs for pix2pix2d (default off, unchanged).

Source on dynacell-models: c2d0481.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(viscy-models): UNet3D downsample_z for FNet-2D; reject non-5D input

Port from dynacell-models:
- UNet3D(downsample_z=True): False keeps Z through the encoder, which
  is how FNet2D runs on single planes (default unchanged);
- UNet3DBase.forward raises on non-5D input, and a comment flags the
  ConvTranspose checkerboard artifact.

Source on dynacell-models: a556bc9 (#484), 83a1d69, 16b1d31.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(viscy-transforms): stop BatchedRandAffined blanking singleton axes

Port 9640537 from dynacell-models: a size-1 spatial axis (Z=1 for 2D
models) gets an identity row in the affine, instead of being resampled
off the volume and zeroed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(viscy-transforms): mark the sampled normalizers pointwise

NormalizeSampled and MinMaxSampled set pointwise = True, which lets the
read-once predict path in viscy-data normalize a whole volume once
instead of once per window.

Source on dynacell-models: 8e74a7e (#532).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(viscy-transforms): make the Gaussian-noise statistics test deterministic

Port 3409560 (#513) from dynacell-models.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build: parallel test runs with pytest-xdist

Port dfcf9a5 (#512) from dynacell-models: pytest-xdist joins the test
group, and a root conftest.py pins OMP/MKL/OpenBLAS to one thread under
each xdist worker. Without the pin every worker's torch sizes its pool
to all cores, which made `-n 11` 10x slower than it should be and timed
out subprocess tests. CLAUDE.md documents the usage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: register pytest markers at the root and make them strict

Port the marker half of c50300f (#489) from dynacell-models.
strict_markers = true turns an unregistered marker into a collection
error, so a typo'd marker can no longer silently decorate nothing.
hpc_integration moves from the cytoland and dynaclr conftests to the
root: an addinivalue_line registration only holds when that conftest
is in the invocation's ancestry. dynaclr drops its own testpaths so
the root ini applies.

requires_data is registered too: main's
applications/dynacell/tools/generate_grouped_eval_configs_test.py uses
it, and strict markers would otherwise drop that module at collection.
(dynacell-models no longer uses the marker; the dynacell PR that
follows removes the registration with it.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build: put the dynacell hf_demo on the pytest pythonpath

Port the root half of 7b7cb43 (#523) from dynacell-models so the HF
demo's tests import its app module. The directory already exists on
main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(dynaclr): join annotations on the full fov_name

Port d1ea492 from dynacell-models. The orchestrated linear-classifier
fixture joined on the first fov_name component, which failed 7 tests
on main once pytest collects dynaclr's colocated *_test.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(cytoland): correct the Spotlight recipe comments

Port the cytoland recipe comment fixes from 40de793 (#498): the
preprocess flags take explicit values (`--compute_otsu true
--compute_fg_masks true`), fg_threshold and nonzero_threshold are inert
while fg_mask_key is set, and the Otsu-centred target is what makes 0
the FG/BG boundary. Comments only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(cytoland): install the VCP tutorials' unpublished deps from GitHub

Port the tutorial fix from d3f6422 (#479): cytoland and viscy-utils
are not on PyPI yet, so the tutorials install them from the GitHub
monorepo; their siblings resolve from PyPI. Adds the jupytext-paired
notebooks (no outputs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: cut CLAUDE.md content the code or global memory already states

Port 93ac234 from dynacell-models: drop the repo-layout tree, setup
and common-command blocks, lint/docstring rules (pyproject.toml is the
source) and the generic coding-philosophy sections. The DynaCell job
runbook from the same file arrives with the dynacell PR, since it
documents tools that only exist there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: run the viscy-utils tests in their own job

viscy-utils had no CI job on main or on dynacell-models: four of its
test files ran inside the dynacell job and the rest (losses, metrics,
meta_utils, optimizer health, CLI, lazy imports) ran nowhere. Mirror
the viscy-data job on ubuntu-latest for Python 3.12 and 3.13, and make
the gate job wait for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(dynacell): run the launcher, engine and prediction-writer suites

Port the test-dynacell-configs job from dynacell-models, minus the
eight test files that only arrive with the dynacell PR: sync
viscy-utils' test group with dynacell, and add test_submit_benchmark_batch,
test_engine, test_pipeline_cache, feature_metrics_test and the
viscy-utils prediction-writer, prediction-metadata, monitor-health and
LR-cooldown tests. The dynacell-models PR target is not carried over.

Source on dynacell-models: 33d2fee, 74fa18a (#497), 2d1ed32 (#506),
4450efd (#518), 2798dee (#519), 2f9ef31 (#528).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build: re-lock against the ported packages and main's dynacell

Start from dynacell-models' uv.lock and re-resolve with `uv lock`
against this tree, whose applications/dynacell/pyproject.toml is still
main's. Against dynacell-models' lock this drops the ten dynacell-only
packages (dinov3, etils, ftfy, jsonpath-rw, mlcroissant, pandas-stubs,
ply, rdflib, submitit, termcolor) and resolves cubic back to 0.7.0;
the dynacell PR restores them. Against main's lock it adds pytest-xdist
and moves boto3/botocore 1.43.0 -> 1.43.20, cellpose 4.1.1 -> 4.2.1.1,
pyarrow 19.0.1 -> 24.0.0, s3transfer and tracksdata; torch and
lightning are unchanged. `uv lock --check` exits 0 (main's own lock
does not).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(dynacell): DynaCell v1.1 benchmark configs from dynacell-models

Port every DynaCell config path of dynacell-models verbatim: the model
leaves (baselines, A549/joint/HEK/brightfield, the 2D track with its
seed replicates and warmup-matched refit, the temporal and
patch-vs-augmentation ablations, CELL-Diff-R2 and the DynaCell-lite
leaves), the grouped and instance-AP eval buckets of the paper models,
the shared overlays, train/predict sets and launcher profiles, and the
deletion of 155 stale config files. The leaves are generated; review
the generators and guard tests in the code commits, not each leaf.

Spotlight v2 arm leaves, their four fit overlays and their 231
spotlight_v2_* eval buckets are left out; they follow in their own PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(dynacell): DynaCell v1.1 application code from dynacell-models

Port every DynaCell code path of dynacell-models verbatim (226
non-merge commits since 2026-06-16; PRs #479-#530): the evaluation
pipeline (cpdino/MorphEm/focus-aware 2D, focus2d slab metrics,
foreground-limited metrics, MicroMS3IM, scale-invariant pixel metrics,
CP shared space, provenance and cache invalidation), the paths
registry, preprocessing/registry/croissant/distribution, the launcher
and stall watchdog, the lite benchmark and CELL-Diff fast inference,
the 2D track, the HF demo, docs, and the engine, celldiff_wrapper and
mask_conditioning modules exactly as on dynacell-models.

The three LightningModules keep their Spotlight training hooks
(seg_aux, seg_consistency, mask_mode, target_bg_lowpass). Every hook
defaults off, the bit-identity tests guard that, and only Spotlight
leaves turn them on; two DynaCell-paper 2D checkpoints store
seg_aux_weight in their hparams, so a hook-free engine could not
resume them. mask_conditioning.py comes along because engine.py and
celldiff_wrapper.py import it.

The files that import the Spotlight leaf generator are trimmed in the
commits that follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(dynacell): lock the dynacell eval, croissant and preprocess deps

Take dynacell-models' uv.lock verbatim, which this tree's pyprojects now
match (`uv lock --check` exits 0). Against the shared-packages lock it
moves cubic 0.7.0 -> 0.9.0a4 and adds dinov3 (pinned commit), etils,
ftfy, jsonpath-rw, mlcroissant, pandas-stubs, ply, rdflib, submitit and
termcolor. A fresh `uv lock` would instead float dinov3 to its current
HEAD and bump mlcroissant/pandas-stubs, so the verbatim lock keeps the
versions the paper evaluations ran with.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: drop the requires_data marker registration

Its only user was main's generate_grouped_eval_configs_test.py, which
the DynaCell port replaced: the live-data tests there are now marked
slow. With strict markers, nothing needs the registration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(dynacell): run the release-path, provenance and watchdog suites

Restore the eight dynacell test files the shared-packages PR left out
of the test-dynacell-configs job, now that they exist:
test_evaluation_microssim_calibration, segmentation_test,
test_distribution, test_croissant, test_main_cli_routing,
test_evaluation_fid_provenance, provenance_test and
watch_stalled_jobs_test. The job now matches dynacell-models'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: add the runbook for jobs that hang while holding an allocation

Port the DynaCell job runbook of the root CLAUDE.md from dynacell-models
(6193b9c, bcbfbe6, c4c5ed0, 74fa18a #497, e1d75cf): how to tell a
finished-but-hung job from healthy compute by the CPU-time derivative,
the stall watchdog (tools/watch_stalled_jobs.py), per-family predict
and fit wall limits, and the forensics to capture before cancelling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(dynacell): scope the grouped eval walk to registered model dirs

The grouped generator imported ARMS from the Spotlight-v2 leaf
generator, which is not part of this PR, to skip the Spotlight arms that
share the canonical prediction tree. Drop that import and the Spotlight
model names from _CODE_TO_PAPER, _DETERMINISTIC_MODELS and
_CELLDIFF_MODELS, keeping the four 2D seed replicates the 2D appendix
scores.

The shared tree still holds the arms' stores, and parse_zarr_name
raises on an unknown model token (e.g. the CellDiff "variant"
segaux_iterative). walk_predictions now skips any model dir outside
_REGISTERED_MODEL_DIRS before parsing and prints how many stores it
skipped. The test that pinned the Spotlight skip becomes one that pins
the unregistered-dir skip and its count.

evaluation/paths.py keeps its Spotlight names: without them
canonical_model_name would silently resolve an arm's run dir onto its
baseline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(dynacell): drop the Spotlight leaf entries from the config guards

The Spotlight arm leaves are not part of this PR, so:
- test_benchmark_config_composition: drop the eight v1 *_spotlight
  train leaves, their iPSC predict leaves and data-hparam entries, the
  _UNMIGRATED_MODELS filter that only they needed, and the Spotlight
  members of _WIDE_GPU_TRAIN_LEAVES and _LONG_WALL_TRAIN_LEAVES (the
  2D pix2pix seed1 replicate keeps its wide-GPU opt-out);
- test_paths: drop the two tests that import the Spotlight leaf
  generator or depend on the grouped generator registering the arms.
  The canonical_model_name / resolve_model tests of the arm names stay,
  since evaluation/paths.py keeps those names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(dynacell): describe the 2D seed1 leaves without the Spotlight generator

The 40 leaves of the four 2D baseline seed replicates were emitted by
the Spotlight-v2 leaf generator, which is not part of this PR, and
their headers told the reader to edit that generator. Say instead what
they are for: the second draw scored by the *__2d_focus2d_seed1
buckets. Comments only; the configs are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(dynacell): drop the Spotlight eval generator from the generator lists

evaluation/README.md and the hand-written-bucket guard's comment in
test_evaluate_compose.py named generate_spotlight_v2_eval_configs.py
among the generators that write grouped buckets; it is not part of this
PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(viscy-utils): declare torchvision, which evaluation.metrics imports

viscy_utils.evaluation.metrics imports torchvision.ops at module level,
and losses.MixedLoss imports that module, but viscy-utils never listed
torchvision. Installs only worked because another workspace member
pulled it in. The new test-utils CI job syncs viscy-utils alone and
failed to collect six test modules with ModuleNotFoundError. The lock
gains only the dependency edge; no version moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(dynacell): keep main's batch-launcher test and the writer tests out

Two corrections to the test-dynacell-configs job:
- main's tests/test_submit_benchmark_batch.py resolves leaves under
  /hpc and fails on a runner (5 PermissionErrors in CI). The
  dynacell-models version that runs there arrives with the DynaCell
  PR, which adds it back.
- the four viscy-utils test files (prediction writer, prediction
  metadata, monitor health, LR cooldown) now run in the test-utils job
  on both Python versions, so the dynacell job no longer repeats them
  or syncs viscy-utils' test group for them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(viscy-utils): share the fit-callbacks lookup between CLI injectors

_inject_checkpoint_guardrails and _inject_optimizer_health_guard each
repeated the same walk from the config to fit's trainer.callbacks plus
the already-injected check; the `else config` branch of that walk could
never run behind the `subcommand != "fit"` return. Both now call
_fit_callbacks_without. _callback_field collapses its two identical
branches, and the monitored checkpoint's init_args are kept from the
search loop instead of being looked up twice. No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(viscy-utils): drop MonitorHealthCheck's unused check_finite flag

Nothing sets check_finite (the CLI injects monitor and patience only,
and no config or test passes it), so the callback always checks
finiteness; it now does so with math.isfinite instead of building a
tensor for one float.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(viscy-utils): stop describing fg masks as built from Otsu thresholds

generate_fg_masks does not read the stored otsu_threshold; it binarizes
a CLAHE-equalized, median-filtered volume. Its summary line and
VisCyTrainer.preprocess's compute_fg_masks entry still said "from Otsu
thresholds".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build: leave the hf_demo pytest path to the DynaCell PR

Nothing on this branch imports from applications/dynacell/examples/
hf_demo: its only test, build_demo_data_test.py, arrives with the
DynaCell PR, which adds the pythonpath entry back with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(viscy-utils): guard a fit that sets no callbacks

Trainer.callbacks defaults to None, and the CLI injectors returned
early unless they found a list, so a fit that set no callbacks ran
without OptimizerHealthCheck, the guard meant for every fit. The
shared lookup now gives such a fit an empty list to append to (None
and [] mean the same to Trainer). The checkpoint guardrails still
need a monitored ModelCheckpoint and remain a no-op there.

Reported by Copilot on #533.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(viscy-utils): say what the AMP skip counter actually counts

The counter resets only when GradScaler grows its scale, so successful
optimizer steps can sit between the reductions it counts; the error
message claimed every step was skipped "in a row". Describe it as
reductions without growth in between, which is what it detects: the
backward overflowing faster than growth_interval clean steps can
restore the scale. Behavior unchanged.

Reported by Copilot on #533.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(viscy-utils): resolve viscy_utils.callbacks names lazily

VisCyCLI now injects MonitorHealthCheck and OptimizerHealthCheck into
every fit by class path through viscy_utils.callbacks, whose __init__
eagerly imported OnlineEvalCallback and with it scikit-learn, which
only the eval extra installs. A fit on a base install would therefore
fail before training. The package now resolves its exports lazily
(PEP 562), like viscy_utils itself. The new test blocks sklearn and
imports the health callbacks: the eager __init__ fails it with
ModuleNotFoundError, the lazy one passes.

Reported by Copilot on #533.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(cytoland): install viscy-data from GitHub in the VCP tutorials

The tutorials install viscy-utils from the GitHub monorepo and let the
siblings resolve from PyPI. The writer they predict with now reads
SlidingWindowDataset.z_planes, which the PyPI viscy-data 0.1.0 does not
have (checked in its wheel), so predict would fail with AttributeError
in on_predict_start. Install viscy-data from GitHub too, in the .py
sources and their paired notebooks.

Found by /code-review on #533.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(viscy-utils): return an empty CLAHE result when the clip range is zero

foreground_mask_from_volume guards a constant volume, but a blank
volume with a few hot pixels (under 0.2% of voxels) has equal 0.1/99.9
percentiles too, and equalize_clahe then divided by zero and fed NaN to
CLAHE (RuntimeWarning: divide by zero, reproduced on a (20, 64, 64)
volume). equalize_clahe now returns zeros there, which the mask's
existing flat-histogram guard turns into an empty mask, as for a
constant volume. Volumes with any contrast are unchanged.

Found by /code-review on #533.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(viscy-utils): keep a step-cadence leaf's last.ckpt on its step trigger

The injected latest-weights ModelCheckpoint takes over save_last from
the monitored one but copied only every_n_epochs (defaulting to 1). A
leaf that checkpoints every N steps, typical for max_steps runs with
long epochs, would get last.ckpt only at epoch boundaries, so a
preempted run resumed from much older weights. The injected callback
now copies the monitored callback's every_n_train_steps or
train_time_interval when set; epoch-cadence leaves are unchanged. No
current leaf combines a monitored checkpoint with a step trigger.

Found by /code-review on #533.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(dynacell): skip only PAPER_KEY-known arms in the grouped eval walk

The scope filter skipped every model dir outside the generator's
registry and only printed a count. That also silenced the loud
"unknown model code-name" a new, never-registered model used to raise,
so its stores would drop out of every paper bucket without an error.
Now the walk skips only dirs that paths.PAPER_KEY knows but this
generator does not register (the Spotlight arms on this branch);
anything unknown to both still raises in parse_zarr_name, as on
dynacell-models. Tests pin both sides.

Found by /code-review on #534.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(dynacell): say the instance-AP audit does not share the walk's arm skip

audit_prediction_coverage claimed to apply the same skip filters as
walk_predictions. It applies the same test-set filter, but on purpose
not the walk's skip of PAPER_KEY models the grouped generator leaves
unregistered: listing every unregistered model is what it is for.

Found by /code-review on #534.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(dynacell): carry the seg-consistency engine and focus-parity tests

test_seg_consistency_training.py exercises the seg_consistency hook of
DynacellUNet and DynacellGAN with a stand-in segmenter, and
test_seg_consistency_focus.py pins the loss's focus-plane rule to
dynacell.evaluation.focus. Both test code this PR ships and import
nothing from the Spotlight PR, so they belong here rather than with
the Spotlight arms. Verbatim from dynacell-models.

Found by /code-review on #534.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(viscy-utils): take last.ckpt from every monitored checkpoint

The guardrail turned save_last off only on the first monitored
ModelCheckpoint. A leaf with two monitored checkpoints (say loss and a
validation metric) kept a second last.ckpt writer next to the injected
latest-weights callback, so last.ckpt would alternate between "latest"
and "that monitor's cadence". All monitored callbacks now give it up;
the first still names the monitor, directory and cadence. No current
leaf has two monitored checkpoints.

Found by /code-review on #533.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(dynacell): one model-dir acceptance rule for parse and walk filter

_REGISTERED_MODEL_DIRS restated, as a separate set, the rule
_split_model_variant uses to accept a dir name, so the two could drift.
_match_model_variant is now that rule (None when the dir does not
parse); _split_model_variant wraps it and raises the same errors as
before, and the walk filter calls it. No behavior change.

Found by /code-review on #534.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(dynacell): share the foreign-model skip between the walk and the audit

walk_predictions skipped the Spotlight arms' dirs on the shared tree,
but the instance-AP coverage audit still parsed them, so on this branch
it reported 468 coverage errors on the live tree instead of
dynacell-models' 102, and its docstring claimed it applied the walk's
filters. Both now call one predicate, is_foreign_model_dir: a dir whose
model paths.PAPER_KEY knows but the grouped generator does not parse.
Measured on the live tree, the audit's error set is identical to
dynacell-models' (102 entries; the remainder are pre-existing). The new
audit test holds one registered, one foreign and one unknown store and
expects exactly the unknown one.

Found by /code-review on #534 (findings 1, 9, 10).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(dynacell): keep raising on campaign models the generator cannot parse

pix2pix2d_unetvit_wmatch{,_seed1} are in _CODE_TO_PAPER (and in
PAPER_KEY) but not in _DETERMINISTIC_MODELS, so their stores do not
parse. On dynacell-models the walk raises on them and the audit lists
them. The foreign-model predicate would have skipped them as another
campaign's arms. A model the generator itself lists in _CODE_TO_PAPER
is not foreign: the walk raises and the audit flags it, as on
dynacell-models. With this, the audit's error set on the live tree is
identical to dynacell-models' (102 entries, compared path by path).

Registering wmatch properly is a separate decision (its comment says
"evaluate in focused sibling buckets", and adding it to
_DETERMINISTIC_MODELS would fold it into the canonical buckets), so it
is left as on dynacell-models.

Found by /code-review on #534 (findings 2, 7).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(dynacell): stop pointing at the Spotlight leaf generator from this branch

evaluation/paths.py and tests/test_paths.py cited
tools/generate_spotlight_v2_leaves.py, which this branch does not
have, and test_paths claimed a pin to it that only exists with the
Spotlight v2 configs. Say where the generator lives instead. The
Spotlight model names stay: without them canonical_model_name would
resolve an arm onto its baseline. Comments only.

Found by /code-review on #534 (finding 8).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(dynacell): parse the warmup-matched pix2pix2d stores in the grouped walk

_CODE_TO_PAPER declares pix2pix2d_unetvit_wmatch{,_seed1} a parse-only
registration "same rule as fnet2d_voxelmatched", but unlike that model
they were never added to _DETERMINISTIC_MODELS. Their 56 stores on the
live tree therefore raised "unknown model code-name", which alone stops
walk_predictions. Register them like fnet2d_voxelmatched: they now parse,
and, as for every parse-only ablation, they are scored in their focused
sibling buckets (the committed *__2d_focus2d_pix2pix2d_wmatch leaves),
not by regenerating the canonical ones.

Fixes part of FOLLOWUPS #1 (live-tree walk crash).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(dynacell): skip the listed checkpoint re-predicts in the grouped walk

The live tree holds second-checkpoint predictions of registered models
beside their canonical stores, under train-set dirs paths.py does not
know: fnet3d_paper/ipsc_step{8400,11k}, fnet2d_voxelmatched/ipsc_step200k,
fcmae_vscyto{2d_scratch_samplematched,3d_scratch}/ipsc_final and
fcmae_vscyto3d_scratch/ipsc_repredict (34 stores). key_from_prediction_store
raises on them ("unknown train_set token 'ipsc_final'"), which stopped
walk_predictions and filled the instance-AP audit. They come from the
voxel-matched FNet-2D and sample-matched UNeXt2-2D ablations, are
scored only by hand-written experiment buckets, and no committed config
references them, so they are not canonical inputs. Each
(model dir, train-set dir) pair is now listed in _CHECKPOINT_REPREDICTS
with its campaign; the walk skips and counts them and the audit skips
them. A train-set dir that is neither canonical nor listed still raises.

Fixes part of FOLLOWUPS #1 (live-tree walk crash).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(dynacell): name the P1 segcons replicate dirs as foreign in the walk

nucleus/pix2pix2d_unetvit_segcons_{s1,ctrl_s1,x10} (12 stores, Spotlight
P1 follow-ups of 2026-10-19) sit in the canonical tree but are in
neither this generator's registry nor paths.PAPER_KEY, so the
PAPER_KEY-based foreign test cannot see them and they raised "unknown
model code-name". They are now listed by name in _FOREIGN_MODEL_DIRS
and skipped like the PAPER_KEY arms; the skip message no longer says
"PAPER_KEY". Any other unknown dir still raises. The new audit test
holds one store of each kind on the live tree (registered, checkpoint
re-predict, named foreign, wmatch, unknown train set) and expects only
the unknown one to be flagged.

Fixes the rest of FOLLOWUPS #1 (live-tree walk crash).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(dynacell): list checkpoint re-predicts per organelle

_CHECKPOINT_REPREDICTS matched (model dir, train-set dir) in every
organelle, so a future mito/fnet3d_paper/ipsc_step8400 store would have
been skipped as a known re-predict instead of raising, though only the
ER one exists. Entries are now (organelle, model, train set) triples
for exactly the stores on the live tree; a listed token in another
organelle raises "unknown train_set token" again (new test, fails
before this change). The audit test now uses the nucleus step-11k
store, since the step-8400 one is ER-only.

Found by /code-review on the FOLLOWUPS #1 fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(dynacell): one skip rule for the grouped walk and the audit

walk_predictions and audit_prediction_coverage each called
is_foreign_model_dir and is_checkpoint_repredict inline, so a third skip
category added to one could silently miss the other and split them
again. Both now call skip_reason(zarr_path), which names the category;
the walk counts per category. No behavior change.

Found by /code-review on the FOLLOWUPS #1 fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(dynacell): guard the grouped registries against the walk crash class

Two guards on the hand-kept registries behind the live-tree crash:
- every _CODE_TO_PAPER model must parse. At a04b3eb this lists
  pix2pix2d_unetvit_wmatch{,_seed1}, the omission that stopped
  walk_predictions;
- the name-based skip lists stay disjoint from what the campaign
  registers: no _FOREIGN_MODEL_DIRS entry in _CODE_TO_PAPER or the
  parse tuples (it would be dropped silently, since the foreign check
  runs first), and no re-predict token among the canonical train sets.

Found by /code-review on the FOLLOWUPS #1 fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(dynacell): reflow the checkpoint re-predict comment

The organelle-scoping edit left a broken line wrap in the
_CHECKPOINT_REPREDICTS comment. Comment only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(dynacell): keep the wmatch refit out of canonical and instance-AP leaves

Registering pix2pix2d_unetvit_wmatch{,_seed1} for parsing (97c30a1)
fixed the live-tree walk, but a default regeneration would then also
fold their 56 stores into the canonical {organelle}_{train_set}
buckets and the instance-AP leaves, against DON'T-FOLD: the committed
*__2d_focus2d_pix2pix2d_wmatch buckets are where they are scored.
They are now listed in _SIBLING_ONLY_MODELS: walk_predictions still
returns them, so sibling buckets can be built from it, but main()
buckets canonical_predictions() (the walk minus that list) and the
instance-AP in_scope() drops them. New tests: canonical_predictions
leaves them out, in_scope rejects them.

MEASURED on the live tree, both generators run the default way (no
--out-root) emit 0 wmatch blocks, and the canonical pool (628 stores)
is identical, store by store, to what the pre-fix rules parse, so a
default regeneration writes exactly what a04b3eb would have written
had its walk not crashed.

Found by /code-review on the FOLLOWUPS #1 fix (finding 2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(dynacell): retitle the paper and describe DynaCell as the broader framework

The camera-ready title is now "Evaluating Dynamic 3D Virtual Staining
of Live Cells with DynaCell": DynaCell names a broader framework for
image-based profiling of cell-state dynamics, and this paper's
evaluation framework is part of it. Update the title in CITATION.cff
(software and preferred citation), the README and both HF card BibTeX
entries, and the Croissant citeAs string; rewrite the CITATION.cff
abstract and the README tagline to describe the evaluation framework
as part of DynaCell.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA

* docs(dynacell): call the contribution the DynaCell evaluation suite

Mirror paper 1c85e84: "This evaluation framework is part of DynaCell, a
broader framework" repeated the noun. The paper's contribution is now
the DynaCell evaluation suite, and DynaCell stays the broader framework.
The CITATION.cff abstract also says "a set of baseline models", so
"suite" means one thing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA

* docs(dynacell): retitle to The DynaCell Evaluation Suite

The paper's title is now "The DynaCell Evaluation Suite for Dynamic 3D
Virtual Staining of Live Cells" (Overleaf b73f25f). Updates
CITATION.cff, the README and HF card BibTeX, and the Croissant citeAs.
test_croissant.py: 17 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA

* chore(dynacell): delete finished one-shot tools

Each did a one-time job that is done, and nothing else imports them:
- build_migration_manifest, run_migration, restandardize_config_paths
  (+ tests): the canonical-path migration was applied 2026-07-13; a
  rebuilt manifest on 07-24 had nothing left to move;
- retokenize_deconv_predictions (+ test): the 24 a549__deconv /
  joint__legacy_deconvgt renames were applied 2026-07-22; paths.py
  still reads both tokens for the dirs kept on purpose;
- scan_nan_microssim_caches (+ test): one audit for the cubic 0.9.0a1
  NaN MicroMS3IM caches that #506 fixed;
- add_instance_ap_to_ablation_leaves: one codemod, already applied;
- smoke_joint_leaf: an April CPU smoke the config-composition tests
  now cover.
The provenance builders (cpdino seg masks, HEK stores, A549 dual fuse,
random-init checkpoints) stay. README entries and one bake_best_ckpt
docstring line follow. Full suite: 5251 passed, 0 failed (the 18
fewer are the two deleted tests/ files; the 14 colocated tool tests
were never collected by the root run).

Raised in review by edyoshikun.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc

---------

Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@alxndrkalinin