Repository navigation
Conversation
* added loss_fn combining mae and kl, tests * get multiple loss_functions * working mae-kl loss * removed tests * added tile input normalization in dataset * add normalize flag to ds with mask * calculate loss metrics on axis depending on image_data_format * parse config in train_script and pass dicts to trainer, to be debugged * refactored networks, to be debugged * working version, have to refactor masked_loss * generalize masked_loss, incomplete * Revert "refactored networks, to be debugged" This reverts commit d3351fd731f5442afb8de311c0ee152b8cdb0e28. * revoked changes in loss, added mask weighting in dataset * debugged masked_loss * Jenny feedback
* modify train_test_split to return sample numbers instead of indices to unique samples * update variable to indicate samples and not indices
* refactored layers * coded conv_blocks, to be debugged * graph ok for Unet2D and 3D * graph ok for UnetStackTo2D * graph ok for image*_to_vector_net * graph ok for unet_stack_stack * modified init to add all the networks * misc errors in train_script * fix for lr_find * Jenny feedback * missed error check for num_dense_units
Fixed network config argument to actually pass network_config instead of the full config object
* Update inference_script.py Added --flat_field and --no_flat_field instead of --flat_field_correct
* added tests for conv_blocks, add support for valid padding
* predict on larger image, tested on 2D models only * add place_operation param to predict_on_full_image, fix tile_image to include edge tiles
* initial untested changes to flatfield * almost done debugging preprocess refactor * finished volume fraction and tested * added row col indices * helper function for dataframe init * tested * deprecated lif read * anitha feedback * updated readme * fixed spelling
* added tests for UNet2D and 3D model
* changed preprocess, taking a break from train * sorted dataframe * reorganized dataframe * tested regressing with/out masks and segmentation
* added augmentation for 2D images * started adding tests for UNetStackTo2D, UNetStackToStack
* added tests for Image2DToVectorNet * added tests for Image3DToVectorNet
* debugging mixed depths in preprocessing * fixed tile sizes * updated readme
* updated readme * halfway through tests * moar tests
* added dataset input params * starting tests * more tests * finished testing * finished testing * bug fix * cleaned up some code * anitha feedback * fixed fixes * fixed fixes
* started prediction cli * untested code * errrrorrrrs * bug fix * not working * working whole image inference on stack to 2d * moving preprocess image to utils * moved if predict statement * tried to debug 3D tests * modified interp_upsampling3D to use static shape when image dimensions are provided * cleaned up import and removed the inference check * Reimplemented crop_layer as a pure tf function Use of dynamic shape in tf results in None along all dims other than channels. This leads to conflicts in other layers have static shapes. Modified InterpUpsampling3D to use static and dynamic shapes as needed. * if depth=1, initialize as 2D model * removed print * save config without default flow style * removed filter_size filter_stack assigment * save config before model creation * removed the check for interp fact of 1 * renamed inference scripts
* started adding functionality that will add to existing tiles if tile dir exists * now support the ability to add tiles to existing tile dir, added threading * changed tile assertions * period to comma * tuple to list * added threading in dataset * bug fix
* added keras-contrib to dockerfile * updated keras to 2.2.4 * updated readme * added tmux, simplified opencv
* removed threading * made workers a config option * temp fix of shape bug * changed dataset to have data_format as input * added squeeze option to dataset * need to squeeze after augment * made lr fig name optional * reverted to TF 1.10 * reverted back to keras 2.1.5 to get UNet2D to run
* find gpu ids, select best. Need to put together with existing code * finalized gpu selection, untested * removed remnant args.gpu
Minor changes to a few of the settings to match available settings in existing config file
* changed data format to shape order * bug fix * adding a check or two * moved test
* add new config file * Add 'pos_ids' parameter to preprocess_script.py * update config file
* changes for multi-channel stack to 2d, metrics=None, split by df index * add random seed to training_table for train-val-test-split
* remove outdated tests * remove nose and add pytest * remove unused import * rename torch_unet to unet * format * linting * remove gpu profiling script this can be now acheived with the lightning profiler * lint with ruff and format with black * format tests * sort import with ruff and update CI workflow * remove duplicate event * black v23.3.0 * format * remove 3.11 support due to lapsolver issue cheind/py-lapsolver#18 * remove tf metric script test * run a minimal subset of pytests
ziw-liu
marked this pull request as ready for review
June 20, 2023 23:11
mattersoflight
added this to the
0.1.0: 2.5D UNet for 3D virtual staining using pytorch lightning milestone
This also auto-generates CLI subcommand help message
* break data setup into smaller methods * wip: transforms use channel names directly * subpackage entries for unet * fix import * support multi-channel input and output * improve slicing syntax * rename val loss log name * move init docstring to class level * fix append channel call * multi-channel prediction writer * rename lightning module for virtual staining
* test stage dataloading * ground truth mask dataset * restore cli script entry for debugging * remove incomplete writer callback default * fix test dataset * compute test metrics with torchmetrics * evaluation submodule * load gt masks as int16 * depend on torchmetrics and pycocotools * trim custom metrics and add mAP * compute pearson correlation and mAP * mark POD as broken * test metrics * fix skipping * log coco metrics * replace print with debug logging * fixed POD metric * unmask POD test * add test coverage * test wrong dimension * fix mask loading * add diameter argument for cellpose prediction api * more flexible save locations * bump torchmetrics to 1.0 * fill missing segmentation metrics with -1 * remove constant input check * evaluate target channel on ground truth --------- Co-authored-by: Soorya Pradeep <soorya.pradeep@czbiohub.org>
mattersoflight
approved these changes
Jul 18, 2023
mattersoflight
left a comment
mattersoflight
left a comment
Collaborator
There was a problem hiding this comment.
I suggest a squash merge and refinement starting from there.
alxndrkalinin
added a commit
that referenced
this pull request
Sep 26, 2026
… model families (#498) * feat(dynacell): register the four Spotlight arms in the model registries canonical_model_name resolves by LONGEST PREFIX, so an unregistered spotlight run dir does not error -- it silently resolves onto the baseline it is meant to be compared against: fnet2d_spotlight -> fnet2d pix2pix3d_unetvit_spotlight -> pix2pix3d_unetvit pix2pix2d_unetvit_spotlight -> pix2pix2d_unetvit The arms would then write into the baselines' eval and cache directories, corrupting the comparison, with nothing in the pipeline complaining. Registering the exact keys fixes it, since a longer exact key wins longest-first. All three lists have to move together. PAPER_KEY is authoritative -- paper_key() raises on anything absent and ParsedZarr.paper_name reads it, not the mirror. _CODE_TO_PAPER is that mirror, pinned against PAPER_KEY by test_paper_key_maps_agree_on_overlap, and it also gates instance-AP emission and audit_prediction_coverage. _DETERMINISTIC_MODELS is the parser accept-list, matched exactly by _split_model_variant, which raises "unknown model code-name" when predictions land on disk rather than at config time. The test also pins fnet3d_paper_spotlight -> fnet3d_paper: that is the natural-but-wrong run-dir token (baseline dir + suffix), and writing it would reintroduce the collapse this commit removes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(dynacell): add an additive otsu_threshold writer for training stores Spotlight needs a per-FOV Otsu threshold in the store's zattrs. The obvious way to get one -- `viscy preprocess --compute_otsu` -- is the wrong way for the iPSC training stores, because it always calls generate_normalization_metadata, which recomputes and REPLACES mean/std/median/iqr at three levels (fov_statistics, dataset_statistics, every timepoint_statistics entry) on both the position nodes and the plate node. Every published iPSC model normalizes against those values. That replacement is harmless only if recomputation is bit-exact. Measured over all 500 positions x 2 target channels x 3 levels: cell.zarr 0 mismatches -- the stock CLI is a true no-op cell_focus.zarr 32032 mismatches The focus store was carved from the full-Z cell.zarr and inherited its parent's normalization block verbatim -- byte-identical on 500 of 500 positions -- so its stored statistics describe 44 planes while the store holds 5. At 4/10615/1304, channel Nuclei: stored mean 417.3512 / std 18.0563, recomputed on the slab 419.8103 / 20.8461. Running the stock CLI there would silently re-base the target normalization of every 2D arm that trains on that store, confounding any spotlight-vs-baseline comparison with a normalization change. This computes the same threshold the stock path computes -- identical grid sample, median filter, threshold_otsu call and constant-input guard -- and writes only that one key. write_meta_field merges at the channel level, so passing fov_statistics back in full is what makes it additive. generate_fg_masks reads the key from zattrs and so runs unmodified afterwards. --undo is an exact inverse, pinned by a test, because this mutates an 86 GB store every published iPSC number depends on. The tool also refuses to leave a store half-written: _collate_norm_meta takes its stat-key set from sample 0 of the batch, so a partially-keyed store raises KeyError only on batches whose sample 0 carries the key -- intermittently, hours in, and on baseline fits too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * docs: fix the Spotlight preprocess command and label fg_threshold inert Both Spotlight mode recipes and the cytoland VSCyto3D example document the setup step as `viscy preprocess --compute_otsu --compute_fg_masks`. Run literally that fails -- jsonargparse wants an explicit value: error: argument --compute_otsu: expected one argument The same files set fg_threshold: 0.0 alongside fg_mask_key: fg_mask, and the cytoland header goes further, claiming the 0.0 "is valid ONLY because normalizations use subtrahend: otsu_threshold". That reads as though the threshold is what defines foreground. It is not: SpotlightLoss.forward short-circuits on `if fg_mask is not None`, so with fg_mask_key set the mask comes from the store and the threshold is never consulted. What the Otsu-centred target actually buys is that 0 becomes the FG/BG boundary in the model's OUTPUT space, which the Dice term's tunable sigmoid needs regardless. The field is labelled rather than deleted, so behaviour is unchanged for anyone who unsets fg_mask_key and falls back to it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(viscy-utils): let SpotlightLoss return its two terms forward() returns a single fused scalar and exposes neither the masked MSE nor the Dice term. That is fine for the FNet arms, where the loss IS the objective, but not for a GAN: swapping Spotlight in under lambda_l1 changes the reconstruction term's scale, and lambda_l1 = 10 was tuned for L1's. Whether the discriminator has started to dominate is only visible in the two terms separately, so without this the main calibration risk on a 4-day fit is unmonitorable. return_components=False keeps the existing return exactly, so every caller (including the cytoland engine) is untouched; the tests assert the returned total is torch.equal to the default call's on both the precomputed-mask and runtime-Otsu branches. The terms are handed out by the loss rather than re-derived by the caller on purpose. Re-deriving them in the engine would silently fork from what is actually being optimized, which is the one thing that would make the logged numbers worthless for calibration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * fix(dynacell): re-binarize fg_mask before DynacellUNet's loss reads it HCSDataModule patches fg_mask into gpu_augmentations whenever fg_mask_key is set (hcs.py:188-189), so the mask is warped with the image -- correctly, since otherwise it would no longer align. But _PaddedRandomAffine3D defaults to mode="bilinear" and sliding_window.py casts the mask to float32, so the affine resamples it CONTINUOUSLY. Measured on a realistic blob mask with unetvit3d_fit.yml's parameters: input unique values: 2 output exactly 0 or 1: 0.5864 unique values: 26455 41% of voxels reach the loss carrying fractional weight. SpotlightLoss then computes a soft-weighted mean (fg_per_ch is a sum, not a count) and a soft-vs-soft Dice -- different semantics from a flip-only augmentation stack, which keeps the mask exactly binary. So the same loss means two different things depending on which augmentations an arm happens to use, which breaks any cross-family comparison. Thresholding at the point of use makes every arm feed SpotlightLoss a binary mask, and is a no-op for the flip-only ones. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(dynacell): make the DynacellGAN reconstruction loss swappable DynacellGAN hardcoded F.l1_loss at three sites -- the G step and both validation passes (raw and EMA) -- so the two pix2pix families could not run a foreground-aware objective at all, while the FNet families already could via DynacellUNet's loss_function. recon_loss defaults to nn.L1Loss(), so every published pix2pix run is bit-identical. All three sites move together deliberately. Validation has to compute the same loss training does, or `monitor: loss/validate_ema` selects checkpoints against a different objective than the one being optimized. recon_loss is added to save_hyperparameters(ignore=...). Without it the module is pickled into the checkpoint, and VisCyCLI._parse_ckpt_path lets checkpoint hparams win on non-fit subcommands for any key the predict leaf does not declare -- and a single pickled class makes torch.load(weights_only=True), which _ckpt_state_dict uses, raise for the whole file. The test asserts the absence directly rather than asserting the checkpoint reloads: SpotlightLoss holds only plain floats, so it round-trips fine even with the ignore dropped, and a reload-succeeds test would pass while the bug shipped. Verified by removing the ignore and watching that test fail. loss/g_l1_train keeps its name even when the term is no longer L1 -- renaming it would break three engine tests and the dashboards -- but the two Spotlight components are now logged beside it as loss/g_recon_{masked_mse,dice}_train. That pair is the only monitor on this change's main risk: lambda_l1 = 10 was tuned for L1's magnitude, and Spotlight sits on a different scale, so the swap re-balances the generator objective against the adversarial term. The mask is re-binarized here too, for the reason given in the preceding commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(dynacell): add the Spotlight fit and predict leaves for 8 iPSC arms Four families (fnet3d, fnet2d, pix2pix3d_unetvit, pix2pix2d_unetvit) x nucleus + membrane, iPSC-trained: 4 shared model overlays, 8 train leaves and 34 predict leaves. Each leaf is copied from its baseline and differs only in the loss, the mask plumbing and the target subtrahend, because the comparison is only meaningful if nothing else moved. Verified: all 8 compose data hparams (batch_size, z_window_size, yx_patch_size, num_workers) identical to their baselines. Three things the copies get right that the obvious version does not. The target DIVISOR differs by family and must not be standardised. fnet3d/fnet2d declare mean/std in their own leaf; the pix2pix leaves inherit median/iqr from targets/<organelle>.yml. deep_merge REPLACES lists, so one shared normalizations snippet would silently change two things on the pix2pix arms -- median -> otsu_threshold AND iqr -> std. The divisor is load-bearing: the Dice term's tunable sigmoid clamps everything at or below 0 and puts its entire soft transition in x in (0, ~0.05) in DIVISOR UNITS, so it sets the scale of the only band where that term has gradient. Change the subtrahend, never the divisor. The 3D and 2D Run-D recipes live at different layers. pix2pix2d bakes lambda_l1: 10.0 / lecam_gamma: 0.3 / max_epochs: 40 into its overlay; pix2pix3d leaves them in train_4gpu_modernized.yml. So the 3D spotlight leaf repeats that block verbatim and the 2D one must not -- copying the 3D overlay alone would yield lambda_l1=100, LeCam 0 and 20 epochs. The ckpt run-dir token is the registered key, not baseline+suffix: fnet3d_paper_spotlight resolves to fnet3d_paper. ckpt_path names last.ckpt as a placeholder because `--ckpt best` resolves the checkpoint DIRECTORY from that field and exits when it is absent. 34 predicts, not 32: both membrane 3D baselines carry a fifth HEK condition, mirrored so every arm stays condition-matched. Roster notes: TRAIN_LEAVES takes all 8, but PREDICT_LEAF_FILES takes only the ipsc_confocal condition, since test_predict_leaf_composes asserts "test_cropped" in data_path and the A549/HEK stores are not under it. _WIDE_GPU_TRAIN_LEAVES gains the two pix2pix2d arms, which inherit the baseline's measured <80 GB opt-out. Two further roster tests needed handling: _EXPECTED_DATA_HPARAMS gains an entry per spotlight model (pinning the match with its baseline), and the 2D arms are excluded from _MIGRATED_TRAIN_LEAVES because dataset_ref: null points them at the offline focus store, which has no manifest entry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(dynacell): scope the HEK predict generator per model Every predict__hek_mantis_kras_a549xy.yml in the tree is emitted by this generator, and _MODEL_DIRS / _JOB_STEM / _HARDWARE_PROFILE are keyed together precisely so a new family fails loud rather than silently inheriting a wall limit that does not fit it. Hand-copying a HEK leaf instead would make it invisible here: the next regeneration would neither refresh nor re-emit it, and the _hek_z_window rescale would not be re-applied. Adding the two 3D spotlight families to the roster alone does not work, though. The generator walks the full (organelle x train pool) grid and collects a FileNotFoundError per missing tuple, returning 1. Spotlight fit only iPSC-trained nucleus + membrane while HEK evaluates only membrane and mito, so exactly one of each family's six tuples exists and the tool would go red on the other five. _MODEL_SCOPE narrows those two families and leaves the original four spanning everything. Verified: the generator now emits 26 leaves and exits 0, the two spotlight HEK leaves among them, and regeneration modifies no baseline leaf. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * perf(viscy-utils): route the fg_mask median filter through cubic generate_fg_masks spends essentially all of its wall clock in scipy.ndimage.median_filter, which is CPU-only. Measured on an A40 with the exact (44, 624, 924) float32 volume and (1, 3, 3) footprint cell.zarr runs: scipy.ndimage.median_filter 5.44 s cubic.scipy.ndimage.median_filter + ascupy 0.03 s cubic is how the rest of the codebase reaches the GPU already (see dynacell.evaluation.metrics): its proxies dispatch on the INPUT ARRAY's device, so uploading with ascupy selects the cupyx kernel and handing it plain numpy selects SciPy -- one call site, both backends. That is preferred over calling cupyx directly, which would fork a second GPU convention in the same repo. A median filter selects an existing element rather than computing a new one, so there is no floating-point reassociation and the backends agree bit-for-bit; the tests pin that on both footprints and on both cubic's CPU and GPU paths, and skip rather than fail where cubic or a device is absent. Import is optional because viscy-utils does not depend on cubic -- it ships in dynacell's `eval` and `preprocess` extras, and generate_fg_masks is a preprocess-path function. The GPU upload is gated on torch.cuda.is_available() for the same reason metrics.py gates it: cubic imports cleanly without cupy/cucim and ascupy would raise there. Whichever backend is chosen is logged once, since silently running 180x slower than intended is the failure mode worth catching. End-to-end the mask pass is then bound by the zarr write rather than the filter: ~5.8 s/position becomes ~1.9 s, so cell.zarr goes from ~48 min to ~16 min. The immediate payoff is analysis rather than the campaign -- re-thresholding all 500 focus positions x 2 channels took 27 s instead of roughly 50 min. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * fix(viscy-utils): compute Otsu on the original Spotlight recipe generate_normalization_metadata thresholded a decimated grid: it sampled native[:, c, :, ::8, ::8] and applied a 3x3 in-plane median, so the smoothing footprint spanned 17x17 source pixels on the threshold side but 3x3 on the mask side, and nothing smoothed across Z at all. The original implementation (norm_threshold in the fnet_astr_vpa notebooks, which resolves to cubic's downscale_and_filter( downscale_factor=1, filter_size=5) then get_threshold_otsu) is a 5x5x5 median cube at full resolution, a sigma=5 Gaussian, then Otsu -- both footprints spanning Z, and no decimation. The Gaussian was missing here entirely; it is what makes the histogram separable enough to pick a threshold. Scored against binarized cpdino segmentations on each sample's own nucleus_area focus plane, mean IoU / recall, on two datasets with very different voxel geometry: iPSC A549 decimate ::8 + median (1,3,3), before 0.7086 / 0.7309 0.7406 / 0.7558 this recipe 0.7647 / 0.7862 0.7713 / 0.7916 Otsu erodes on both -- precision holds at 0.94-0.97 while recall sits near 0.76 -- so the masks were shrunken rather than mislocated, and this buys back recall at near-flat precision. The Gaussian is used only to pick the threshold. generate_fg_masks applies that threshold to the median cube instead, because thresholding the blurred image smears boundaries and scores worse (0.7288 vs 0.7647), and thresholding the raw volume -- what SpotlightLoss(fg_threshold=0.0) did -- is worse still (0.6676). sigma is the published value and deliberately not tuned. A larger blur scores much better on both datasets (iPSC 0.8713 at sigma=30 and still climbing, A549 0.8649 peaking at sigma=20) but the optimum is dataset-specific and does not follow from voxel spacing; a spacing-derived sigma was measured and refuted. Tuning it would fit a training-time parameter on the split the models are scored on, which biases the Spotlight arms alone since baselines never read fg_mask. Drops otsu_grid_spacing from generate_normalization_metadata and the viscy preprocess surface: the recipe works at full resolution, so the stride has no remaining caller. The full-resolution 5x5x5 median costs 42.8 s per (44, 624, 924) volume on the CPU against ~0.07 s on an A40, so smooth_gaussian follows smooth_median through cubic. Unlike the median it sums rather than selects, so the backends agree to a tolerance, not bitwise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * refactor(dynacell): delegate the otsu writer to the shared recipe write_otsu_thresholds.py reimplemented the threshold computation so it could write the value the stock preprocess pass would have written. That duplication is exactly what breaks when the recipe changes: the tool would have kept writing decimated-grid thresholds after generate_normalization_metadata moved to the full-resolution median cube plus Gaussian. Call otsu_threshold_from_volume instead, so the two cannot diverge. Drops --otsu-grid-spacing along with the stride it fed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(dynacell): let the otsu writer borrow a parent store's thresholds The new Otsu recipe smooths across Z with a 5x5x5 median and a sigma=5 Gaussian, and its benefit is Z accumulation. cell_focus.zarr holds 5 planes, so there is nothing to accumulate over and the recipe's effect inverts: measured over 60 positions, Membrane foreground falls from 0.2496 to 0.2189 against what is on disk, while on the 44-plane parent the same recipe raises it from 0.3219 to 0.3487. That splits along the worst possible axis. The 2D arms train on cell_focus.zarr and the 3D arms on cell.zarr, so shipping one recipe would hand the two halves of the comparison masks built by differently-behaving operators -- a confound the campaign exists to avoid. The previous decimated recipe had no Z footprint and was therefore symmetric across both stores. --threshold-from computes each threshold on another store's matching position. The focus store is a verbatim voxel copy of a slab of its parent, so the parent is the same specimen on the same intensity scale, and both stores end up sharing one estimate made with adequate Z. On the focus store's own masks: Nuclei Membrane on disk (old recipe) 0.3319 0.2496 (2 FOVs under 2%) new recipe on the 5 planes 0.3317 0.2189 (2 FOVs under 2%) new recipe on the parent 0.3846 0.1861 (0 FOVs under 2%) A missing position raises instead of falling back to the target's own pixels, because a silent per-position fallback would reintroduce exactly the two-recipe asymmetry this removes. A parent lacking the channel exits 2 rather than thresholding on the wrong one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(viscy-utils): binarize fg_mask with CLAHE, decoupled from otsu_threshold generate_fg_masks thresholded raw intensities at the stored otsu_threshold, which misses dim objects almost entirely -- whole nuclei and membrane stretches got speckle instead of a mask, visible in the focus-plane overlays and consistent across both cell lines. Switch to the binarization path the original Spotlight runs actually used (scripts/blastoid/preprocess.py::read_preprocess in the fnet_astr_vpa repo): robust percentile rescale, CLAHE, then otsu_binarize(filter_size=5) -- a 5x5x5 median cube and threshold_otsu with no Gaussian, since CLAHE already separates the histogram. Parameters come from that repo's config/allen_segment/preprocessing.yaml (percentile 0.1, clip_limit 0.015, nbins 256) rather than cubic's defaults. Scored against binarized cpdino segmentations, nucleus channel, on each sample's own nucleus_area focus plane, mean IoU / recall: iPSC A549 before (raw + median + gaussian) 0.7637 / 0.7851 0.7713 / 0.7916 this recipe 0.8262 / 0.8539 0.8898 / 0.9654 Membrane improves too, against the carved membrane GT: 0.3181 -> 0.4121 on iPSC and 0.3782 -> 0.4421 on A549. Four of four arms improve. Measured and rejected along the way: a Gaussian on top of CLAHE helps iPSC (0.8644) but hurts A549 (0.8807) while no-blur improves both; the repo's cleanup step (min_obj_size=100, max_hole_size=25) moves nothing here (+0.0003, +0.0000); double-threshold Otsu buys recall at 0.86 precision and inverts its ranking between cell lines, and a mask that swallows background would pull the masked MSE back toward an unmasked one. The mask no longer derives from otsu_threshold, and cannot: CLAHE thresholds in an equalized [0, 1] space while otsu_threshold is in raw intensity units for NormalizeSampled to subtract. Treating one as the other would be a unit error. otsu_threshold is therefore untouched on disk and target normalization is unaffected -- only the masks change. The test that pinned the old dependency (raise KeyError without otsu_threshold) now pins its absence, plus a new test that the mask tracks the fixture's bright half rather than merely being non-empty. Foreground extraction is a preprocessing choice, not part of the Spotlight loss, so this is free to differ from the paper without changing what the campaign compares. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(dynacell): log the Spotlight loss components on the FNet path DynacellUNet returned only the fused scalar, so on the four FNet Spotlight arms there was no way to tell whether the Dice term was contributing anything -- the question lambda_mse rests on. The GAN path already exposes this as loss/g_recon_*_train; the FNet path logged only loss/train and loss/validate. _compute_loss now returns (loss, components) like DynacellGAN's _compute_recon, and training_step logs loss/masked_mse_train and loss/dice_train. The terms come from SpotlightLoss.forward's existing return_components flag rather than being re-derived in the engine: a second derivation would silently fork from the objective actually being optimized, which is what would make the logged numbers worthless for calibration. Measured on the GAN arms, this decomposition showed the Dice term doing 10-40x less work on nucleus than membrane, consistently in 2D and 3D. The FNet arms could not be checked for the same effect at all. Validation deliberately logs no components, matching the GAN path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(viscy-utils): add SegAuxDice, a self-calibrating soft-Dice loss SpotlightLoss needs an Otsu-centred target and puts its sigmoid knee at the raw-intensity Otsu level, which disagrees with the CLAHE+Otsu fg_mask it is scored against and forces a normalization change onto every arm that uses it. SegAuxDice derives the threshold (the target quantile matching the mask's foreground fraction) and the knee width (c * IQR) per patch from the target itself, so it is unit-free across z-score, median/IQR and MinMax [-1, 1] targets. Added on top of a family's usual loss, a Spotlight arm then differs from its baseline by exactly one term. Squared-denominator Dice with a plain sigmoid keeps the gradient non-zero everywhere, unlike the clamped tunable sigmoid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(dynacell): add a seg-aux Dice term to the UNet, GAN and flow engines Spotlight v2 asks whether a segmentation loss added on top of each family's usual loss helps, so the usual loss must stay untouched: DynacellUNet keeps MSE / MixedLoss, DynacellGAN keeps lambda_l1 * L1 and DynacellFlowMatching keeps velocity MSE, each plus seg_aux_weight * Dice. loss/validate (loss/validate_ema for the GAN) stays base-only and the Dice is logged beside it, so ModelCheckpoint picks the best checkpoint by the same criterion as the baseline arm. CellDiff scores the one-step estimate x1_hat = x_t + (1 - t) * v_hat, gated to t >= 1 - seg_aux_t0, and rejects seg_aux on any path where that estimate is not exact. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(dynacell): add the Spotlight v2 eval root and stage-0 buckets Grouped buckets write next to each prediction under the canonical <organelle>/<model>/<train>/<cond> dirs with final_metrics forced, and the paper tables read those CSVs. Re-scoring baselines there under a new pin would silently rewrite the paper's numbers. The generator writes the same layout under /hpc/projects/virtual_staining/training/dynacell/_spotlight_v2 with every pred_* artifact forced fresh, sharing only the GT caches. It gates each prediction store on chunk count before emitting, and the roster is organised in waves so adding Stage-1 arms never re-scores stage 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(dynacell): add joint and mask-conditioned CellDiff variants Two ways to put segmentation into the generative process rather than only into the loss, for 2D and 3D CellDiff: - mask_mode: joint generates [image, mask] as one 2-channel flow, with a gated soft Dice on the mask channel's one-step estimate. No external model is needed at inference; predict writes the image half. - mask_mode: cond adds the mask as a second conditioning channel (cond_channels exposed on CELLDiffNet). Training sees the GT mask through MaskCorruption (erode/dilate, blob drop, blanking) so the model tolerates imperfect masks; predict reads a thresholded mask from a second store via CondMaskSource. Masks are encoded to [-1, 1] so the frozen MinMax target normalization is unchanged, and loss/validate stays the image velocity loss so checkpoint selection matches the baseline's. With the options off the state dict and training loss are bit-identical to before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(dynacell): register the Spotlight v2 model tokens canonical_model_name resolves by longest prefix and the grouped parser splits unknown suffixes into variants, so without explicit entries fnet2d_segaux collapses onto fnet2d and celldiff_2d_segaux is read as celldiff_2d plus a "segaux" variant and raises. Either way a v2 arm would read or write its baseline's eval and cache dirs. Adds the _segaux, _seed1, _v2, UNeXt2-2D membrane probe and CellDiff cjoint/ccond tokens, and makes resolve_model match CellDiff arm checkpoint dirs before the bare-celldiff fallback. test_paths pins every token against collapsing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(dynacell): generate the Spotlight v2 first-wave leaves 36 fit and 108 predict leaves, all derived from each baseline leaf by generate_spotlight_v2_leaves.py, which refuses to emit a leaf whose resolved config differs from its baseline beyond the allowed keys (fg_mask_key, the seg-aux / mask-mode model args, renamed run paths): - <model>_segaux for 8 models x nucleus/membrane - <model>_seed1 noise-floor replicates (2D models, fnet3d nucleus) - fcmae_vscyto3d_scratch_v2 and pix2pix3d_unetvit_v2 fresh baselines: the April UNeXt2-3D fits stopped at epoch 111/147 and the pix2pix3d baselines kept checkpoints only to epoch 23/25 (write halt), so the S arms would otherwise select from a different epoch range - UNeXt2-2D membrane probes (l1, safecrop, jointsteps = 160k steps, the joint run's measured budget) for the failed 2D membrane baseline - CellDiff cjoint/ccond, nucleus, 2D and 3D UNeXt2-3D v2/segaux fits move to the 7-day wall: measured ~2.0 epochs/h from April checkpoint mtimes puts 200 epochs at ~100 h. seg_aux_weight, mask_dice_weight and the C-cond threshold are placeholders pending calibration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(viscy-utils): set the SegAuxDice knee from fg/bg contrast, not IQR Measured on the iPSC baselines' validation patches: on thin-structure (membrane) patches the IQR is ~0.65x the foreground/background contrast, so c * IQR left 11-25% of foreground voxels with a vanishing sigmoid gradient; c * (median fg - median bg) cuts that to 0.5-7.5% and leaves the gradient-balanced weight within ~20% everywhere else. Patches whose mask is not brighter than the background are now the invalid ones. Empty or full masks make one median infinite, so the knee width is substituted before the zeroing multiply to keep invalid entries finite (inf * 0 is NaN). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(dynacell): set calibrated per-cell seg-aux weights in the v2 leaves seg_aux_weight is the median over 20 seeded validation batches of ||grad base|| / ||grad Dice|| at each baseline checkpoint, so the Dice term starts with the same gradient norm as the family's usual loss. It spans 0.36 (UNeXt2-2D membrane) to 16 (pix2pix2d nucleus), even within a family, so it is set per (organelle, arm) rather than per family. C-joint's mask Dice borrows the same-dim CellDiff weight. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * fix(viscy-data): keep fg_mask uint8 on the host, cast to float on GPU Both 4-GPU UNeXt2-3D seg-aux fits were OOM-killed at their first validation. The uint8 mask was upcast to float32 per sample, so every whole-FOV val batch carried a mask as large as the target: 1.5x the baseline batch, multiplied through worker buffers and PyTorch's power-of-two pinned-memory cache (48 G per rank already in the baseline, whose cgroup peak is 437-455 G of 512 G). A 1-GPU repro of the real leaf measured +44% peak pinned memory pre-fix, +8% after. Masks now keep their stored dtype through the dataloader and on_after_batch_transfer casts them to float before GPU augmentations and the loss. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(dynacell/tools): add a --wave filter to the Spotlight-v2 eval generator The generator gates every roster store before writing anything, so a wave whose predicts are still running blocked buckets for waves that are already complete. --wave restricts gating and writing to the named waves. The roster is still loaded whole, so the one-wave-per-condition check keeps spanning every wave. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(dynacell): score the finished pix2pix2d arms as Spotlight-v2 wave1a Moves nucleus pix2pix2d_unetvit_segaux (4 test legs) and the nucleus and membrane pix2pix2d_unetvit_seed1 replicates (iPSC) out of wave1 into their own wave. That way they are scored now instead of waiting on wave1's 3D and CellDiff stores. Adds the 5 generated buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(viscy-utils): add a self-consistent Dice reference to SegAuxDice label="target" scores the soft-thresholded prediction against the target through the same sigmoid, instead of against the binary mask. The mask still sets tau, the knee width and patch validity. With label="mask" the loss is not zero at pred == target wherever the mask is not a threshold of the target (0.07-0.25 on the iPSC baselines), so it also pulls predictions toward the mask. label="target" is minimised exactly at pred == target, which isolates the segmentation-shaped penalty from that pull (Spotlight v2 Stage 2 ablation #1). The default is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(dynacell): add the segauxself Spotlight-v2 arm for the 2D families The segaux arm with SegAuxDice(label="target"), covering fnet2d, UNeXt2-2D, pix2pix2d and CellDiff-2D, nucleus and membrane: 8 fits and 32 predicts. It reuses the segaux arm's calibrated weight, so a composed segauxself fit differs from its segaux sibling only in the Dice label (pinned by a test). Registers the tokens in PAPER_KEY, the grouped-eval parser (CellDiff arm ahead of celldiff_2d so it is not read as a variant) and the wide-GPU opt-out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(dynacell): add a predict-only pix2pix2d_unetvit_last Spotlight-v2 arm The pix2pix2d baseline's canonical stores were predicted from its epoch-37 best checkpoint, while every GAN arm predicts from last.ckpt, because the recorded EMA val L1 does not track the weights. So arm-vs-baseline mixed two checkpoint rules. This arm predicts the baseline's OWN last.ckpt into its own v2-only store (never the canonical store the paper reads): 8 predict leaves, no fit. Arm gains a fit flag; a predict-only arm reads its baseline's checkpoint dir. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(dynacell): score the second batch of finished v2 predicts as wave1b Moves the arms whose predicts completed on 2026-09-25 out of wave1 into their own wave: FNet-2D segaux (nucleus and membrane) and its seed1 replicates; UNeXt2-2D membrane segaux, seed1 and the l1 and safecrop probes; and pix2pix2d membrane segaux. That is 21 conditions in 8 buckets, each store passing the completeness gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * fix(dynacell/tools): keep Spotlight arm stores out of the canonical eval buckets The Spotlight arms (v1 and v2) are registered so their store dirs parse, but they share the canonical prediction tree with the paper models. So walk_predictions put all 73 arm stores into the *_ipsc_trained buckets: a canonical regen would score them there and write their eval dirs under the canonical root. They are scored only by generate_spotlight_v2_eval_configs.py into _spotlight_v2, so the canonical walk now skips them. A test pins the skip list to the leaf generator's ARMS plus the four v1 arms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * fix(dynacell/eval): resolve a predict-only arm from its model_name pix2pix2d_unetvit_last predicts the baseline's own last.ckpt, so its checkpoint path names the baseline and resolve_model's ckpt-segment rule returned pix2pix2d_unetvit, which would share the baseline's eval and cache paths. Predict-only arms now resolve from model_name. No production caller hits this today; the test pins both directions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * docs(dynacell): correct which checkpoint the pix2pix2d baseline stores used The comments said epoch-37 best, which holds for membrane only; the nucleus baseline predict leaf uses epoch=22-step=11500. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * refactor(viscy-utils): import torch at module top in meta_utils torch is a hard dependency of viscy-utils, so it does not belong inside the optional-cubic try/except ImportError block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * fix(viscy-utils): return an empty fg mask for a constant volume explicitly A blank timepoint used to reach an all-zero mask only because the NaN from CLAHE's zero-range rescale propagated through Otsu, with divide/cast RuntimeWarnings along the way. It now returns before CLAHE. The test runs with warnings as errors, and the unguarded path raises there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J --------- Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
alxndrkalinin
added a commit
that referenced
this pull request
Oct 27, 2026
…ed walk
_CODE_TO_PAPER declares pix2pix2d_unetvit_wmatch{,_seed1} a parse-only
registration "same rule as fnet2d_voxelmatched", but unlike that model
they were never added to _DETERMINISTIC_MODELS. Their 56 stores on the
live tree therefore raised "unknown model code-name", which alone stops
walk_predictions. Register them like fnet2d_voxelmatched: they now parse,
and, as for every parse-only ablation, they are scored in their focused
sibling buckets (the committed *__2d_focus2d_pix2pix2d_wmatch leaves),
not by regenerating the canonical ones.
Fixes part of FOLLOWUPS #1 (live-tree walk crash).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alxndrkalinin
added a commit
that referenced
this pull request
Oct 27, 2026
…walk
The live tree holds second-checkpoint predictions of registered models
beside their canonical stores, under train-set dirs paths.py does not
know: fnet3d_paper/ipsc_step{8400,11k}, fnet2d_voxelmatched/ipsc_step200k,
fcmae_vscyto{2d_scratch_samplematched,3d_scratch}/ipsc_final and
fcmae_vscyto3d_scratch/ipsc_repredict (34 stores). key_from_prediction_store
raises on them ("unknown train_set token 'ipsc_final'"), which stopped
walk_predictions and filled the instance-AP audit. They come from the
voxel-matched FNet-2D and sample-matched UNeXt2-2D ablations, are
scored only by hand-written experiment buckets, and no committed config
references them, so they are not canonical inputs. Each
(model dir, train-set dir) pair is now listed in _CHECKPOINT_REPREDICTS
with its campaign; the walk skips and counts them and the audit skips
them. A train-set dir that is neither canonical nor listed still raises.
Fixes part of FOLLOWUPS #1 (live-tree walk crash).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alxndrkalinin
added a commit
that referenced
this pull request
Oct 27, 2026
nucleus/pix2pix2d_unetvit_segcons_{s1,ctrl_s1,x10} (12 stores, Spotlight
P1 follow-ups of 2026-10-19) sit in the canonical tree but are in
neither this generator's registry nor paths.PAPER_KEY, so the
PAPER_KEY-based foreign test cannot see them and they raised "unknown
model code-name". They are now listed by name in _FOREIGN_MODEL_DIRS
and skipped like the PAPER_KEY arms; the skip message no longer says
"PAPER_KEY". Any other unknown dir still raises. The new audit test
holds one store of each kind on the live tree (registered, checkpoint
re-predict, named foreign, wmatch, unknown train set) and expects only
the unknown one to be flagged.
Fixes the rest of FOLLOWUPS #1 (live-tree walk crash).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alxndrkalinin
added a commit
that referenced
this pull request
Oct 27, 2026
_CHECKPOINT_REPREDICTS matched (model dir, train-set dir) in every organelle, so a future mito/fnet3d_paper/ipsc_step8400 store would have been skipped as a known re-predict instead of raising, though only the ER one exists. Entries are now (organelle, model, train set) triples for exactly the stores on the live tree; a listed token in another organelle raises "unknown train_set token" again (new test, fails before this change). The audit test now uses the nucleus step-11k store, since the step-8400 one is ER-only. Found by /code-review on the FOLLOWUPS #1 fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alxndrkalinin
added a commit
that referenced
this pull request
Oct 27, 2026
walk_predictions and audit_prediction_coverage each called is_foreign_model_dir and is_checkpoint_repredict inline, so a third skip category added to one could silently miss the other and split them again. Both now call skip_reason(zarr_path), which names the category; the walk counts per category. No behavior change. Found by /code-review on the FOLLOWUPS #1 fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alxndrkalinin
added a commit
that referenced
this pull request
Oct 27, 2026
…lass Two guards on the hand-kept registries behind the live-tree crash: - every _CODE_TO_PAPER model must parse. At a04b3eb this lists pix2pix2d_unetvit_wmatch{,_seed1}, the omission that stopped walk_predictions; - the name-based skip lists stay disjoint from what the campaign registers: no _FOREIGN_MODEL_DIRS entry in _CODE_TO_PAPER or the parse tuples (it would be dropped silently, since the foreign check runs first), and no re-predict token among the canonical train sets. Found by /code-review on the FOLLOWUPS #1 fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alxndrkalinin
added a commit
that referenced
this pull request
Oct 27, 2026
Brings in #534's fix for the walk/audit crash on the live prediction tree (FOLLOWUPS #1): wmatch registered for parsing, the listed checkpoint re-predicts and the named P1 segcons replicate dirs skipped with counts, one skip rule for walk and audit, registry guard tests. Resolution in generate_grouped_eval_configs.py: dynacell-models' Spotlight registrations and _SPOTLIGHT_ARM_MODELS stay, with the wmatch entries and _FOREIGN_MODEL_DIRS added beside them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 27, 2026
alxndrkalinin
added a commit
that referenced
this pull request
Oct 27, 2026
… leaves
Registering pix2pix2d_unetvit_wmatch{,_seed1} for parsing (97c30a1)
fixed the live-tree walk, but a default regeneration would then also
fold their 56 stores into the canonical {organelle}_{train_set}
buckets and the instance-AP leaves, against DON'T-FOLD: the committed
*__2d_focus2d_pix2pix2d_wmatch buckets are where they are scored.
They are now listed in _SIBLING_ONLY_MODELS: walk_predictions still
returns them, so sibling buckets can be built from it, but main()
buckets canonical_predictions() (the walk minus that list) and the
instance-AP in_scope() drops them. New tests: canonical_predictions
leaves them out, in_scope rejects them.
MEASURED on the live tree, both generators run the default way (no
--out-root) emit 0 wmatch blocks, and the canonical pool (628 stores)
is identical, store by store, to what the pre-fix rules parse, so a
default regeneration writes exactly what a04b3eb would have written
had its walk not crashed.
Found by /code-review on the FOLLOWUPS #1 fix (finding 2).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alxndrkalinin
added a commit
that referenced
this pull request
Nov 4, 2026
* feat(viscy-utils): add the segmentation-aware and background losses Port the Spotlight-motivated losses from dynacell-models so they live in the shared package before any application consumes them: SegAuxDice (seg_aux.py) with the SAUNA-map and soft-clDice weight maps (seg_aux_maps.py), the segmenter-consistency loss (seg_consistency.py), the Track H background low-pass target (background_lowpass.py), and SpotlightLoss(return_components=True). Defaults keep SpotlightLoss's existing behavior; no package imports an application. Source on dynacell-models: 40de793 (#498), 6c28a31 (#521), 0cbeda1 (#524), 050f553 (#530). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-utils): build foreground masks with CLAHE, median and Otsu Port the Spotlight fg-mask recipe from dynacell-models. generate_fg_masks now binarizes a CLAHE-equalized, 5^3-median-filtered volume at an Otsu threshold picked on that volume after a sigma=5 Gaussian; compute_otsu uses the same full-resolution smoothing, so the otsu_grid_spacing parameter is gone from generate_normalization_metadata and VisCyTrainer.preprocess. The median and Gaussian run through cubic on the GPU when cubic and a device are present, else SciPy; cubic stays an optional import. torch is imported lazily so that light callers of write_meta_field do not pay for it. Behavior change: Spotlight masks and Otsu thresholds written by `viscy preprocess` differ from main's. Source on dynacell-models: 40de793 (#498), d087d96 (#510). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-utils): make prediction writes resumable, identified and write-once Port the HCSPredictionWriter and prediction_metadata work from dynacell-models: - z_reduction="center" and z_planes (slab-only predicts write only the planes they predicted, and record them); - per-FOV completion markers and the checkpoint/settings identity that predict resume and the eval caches key on (prediction_metadata.py); - refuse writes into an existing store when a FOV outside the run has unmarked prediction channels, or when the array would grow; - assemble each volume in memory and write it once on a background thread (bytes unchanged); - raise NotImplementedError when trainer.world_size > 1, instead of letting ranks race on one store. Source on dynacell-models: 2b848a6, 1be4466, 74fa18a (#497), 8e41498 (#511), 996adb4 (#526), 6c83fff (#531), 8e74a7e (#532). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-utils): guard every fit with checkpoint and health callbacks Port the training guardrails from dynacell-models. VisCyCLI now injects, on every `fit`: - a latest-weights ModelCheckpoint that owns last.ckpt (the monitored checkpoint gets save_last=False), so last.ckpt always means the latest epoch, inheriting the leaf's write cadence; - MonitorHealthCheck, which raises when the monitored metric is non-finite or bit-identical for 10 validation epochs; - OptimizerHealthCheck, which raises after 5 consecutive non-finite losses or on GradScaler collapse. Also adds the opt-in LinearLRCooldown callback and raises the CLI subprocess test timeout to 120 s (it timed out on main). Behavior change: this applies to cytoland, dynaclr and dynacell fits. Source on dynacell-models: 20e54db, daa7477, bc0f444, 74fa18a (#497), 92693ae, 2f9ef31 (#528). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): keep SSIM finite under mixed precision; lazy exports Port two viscy-utils changes from dynacell-models: - ssim_25d / ms_ssim_25d: clamp the variances, bound the covariance by Cauchy-Schwarz through a safe sqrt, floor the stability constants symmetrically, and run MS-SSIM outside autocast, so 16-mixed training cannot NaN the gradient. SSIM/MS-SSIM loss values change slightly for MixedLoss users. - viscy_utils/__init__ resolves its exports lazily (PEP 562), keeping torch and lightning off light import paths. Source on dynacell-models: a556bc9 (#484), 130e034, cee4e41, 272051c, 59343ad, 8b2413f (#509). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-data): slab-only and read-once predict, uint8 foreground masks Port the viscy-data side of the dynacell-models predict and Spotlight work: - HCSDataModule.predict_z_planes and SlidingWindowDataset z_planes / read_z_planes, so a 2D model predicts only its focus slab; - cache_volume: with num_workers == 0 predict reads each (position, t) volume once and, when every transform is pointwise, normalizes it once; - foreground masks are read in their stored dtype (uint8) and cast to float on the device in on_after_batch_transfer. Behavior change: every app's predict at num_workers=0 takes the read-once path. Source on dynacell-models: 40de793 (#498), 996adb4 (#526), 6c83fff (#531), 8e74a7e (#532). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-models): per-axis CellDiff patches and mask-conditioning channels Port from dynacell-models: - patch_size may be a per-axis (z, y, x) tuple through the patch embedding, transformer unpatchify and ViT bottleneck, which lets CELLDiffNet run Z-preserving at Z=1 for the 2D track (an int is still accepted); - CELLDiffNet(cond_channels=1) adds extra conditioning input channels (default keeps the old single-channel net). Source on dynacell-models: 0126fe6, 0617ed7, 40de793 (#498). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(viscy-models): non-uniform ODE time grids for CellDiff sampling Port the time_schedule option of the CellDiff ODE integrators from dynacell-models: uniform (default, unchanged), cosine, or a Beta(p, q) grid packed toward the data end, which is what lets the DynaCell-lite recipe sample in 14 velocity evaluations. scipy joins the celldiff extra for betaincinv. Source on dynacell-models: 19765e7 (#520). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-models): add the MorphEm feature extractor Port MorphEmModel from dynacell-models, the fifth deep-feature extractor of the DynaCell evaluation. transformers is imported lazily and the trust_remote_code model is pinned to a Hub commit. Source on dynacell-models: d3f6422 (#479), 4918d2e, 13cb336, c9c6914. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-models): let the PatchGAN discriminator preserve the Z axis Port preserve_z from dynacell-models, so the 3D PatchGAN runs on Z=1 inputs for pix2pix2d (default off, unchanged). Source on dynacell-models: c2d0481. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(viscy-models): UNet3D downsample_z for FNet-2D; reject non-5D input Port from dynacell-models: - UNet3D(downsample_z=True): False keeps Z through the encoder, which is how FNet2D runs on single planes (default unchanged); - UNet3DBase.forward raises on non-5D input, and a comment flags the ConvTranspose checkerboard artifact. Source on dynacell-models: a556bc9 (#484), 83a1d69, 16b1d31. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-transforms): stop BatchedRandAffined blanking singleton axes Port 9640537 from dynacell-models: a size-1 spatial axis (Z=1 for 2D models) gets an identity row in the affine, instead of being resampled off the volume and zeroed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(viscy-transforms): mark the sampled normalizers pointwise NormalizeSampled and MinMaxSampled set pointwise = True, which lets the read-once predict path in viscy-data normalize a whole volume once instead of once per window. Source on dynacell-models: 8e74a7e (#532). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(viscy-transforms): make the Gaussian-noise statistics test deterministic Port 3409560 (#513) from dynacell-models. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build: parallel test runs with pytest-xdist Port dfcf9a5 (#512) from dynacell-models: pytest-xdist joins the test group, and a root conftest.py pins OMP/MKL/OpenBLAS to one thread under each xdist worker. Without the pin every worker's torch sizes its pool to all cores, which made `-n 11` 10x slower than it should be and timed out subprocess tests. CLAUDE.md documents the usage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: register pytest markers at the root and make them strict Port the marker half of c50300f (#489) from dynacell-models. strict_markers = true turns an unregistered marker into a collection error, so a typo'd marker can no longer silently decorate nothing. hpc_integration moves from the cytoland and dynaclr conftests to the root: an addinivalue_line registration only holds when that conftest is in the invocation's ancestry. dynaclr drops its own testpaths so the root ini applies. requires_data is registered too: main's applications/dynacell/tools/generate_grouped_eval_configs_test.py uses it, and strict markers would otherwise drop that module at collection. (dynacell-models no longer uses the marker; the dynacell PR that follows removes the registration with it.) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build: put the dynacell hf_demo on the pytest pythonpath Port the root half of 7b7cb43 (#523) from dynacell-models so the HF demo's tests import its app module. The directory already exists on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(dynaclr): join annotations on the full fov_name Port d1ea492 from dynacell-models. The orchestrated linear-classifier fixture joined on the first fov_name component, which failed 7 tests on main once pytest collects dynaclr's colocated *_test.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(cytoland): correct the Spotlight recipe comments Port the cytoland recipe comment fixes from 40de793 (#498): the preprocess flags take explicit values (`--compute_otsu true --compute_fg_masks true`), fg_threshold and nonzero_threshold are inert while fg_mask_key is set, and the Otsu-centred target is what makes 0 the FG/BG boundary. Comments only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(cytoland): install the VCP tutorials' unpublished deps from GitHub Port the tutorial fix from d3f6422 (#479): cytoland and viscy-utils are not on PyPI yet, so the tutorials install them from the GitHub monorepo; their siblings resolve from PyPI. Adds the jupytext-paired notebooks (no outputs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: cut CLAUDE.md content the code or global memory already states Port 93ac234 from dynacell-models: drop the repo-layout tree, setup and common-command blocks, lint/docstring rules (pyproject.toml is the source) and the generic coding-philosophy sections. The DynaCell job runbook from the same file arrives with the dynacell PR, since it documents tools that only exist there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: run the viscy-utils tests in their own job viscy-utils had no CI job on main or on dynacell-models: four of its test files ran inside the dynacell job and the rest (losses, metrics, meta_utils, optimizer health, CLI, lazy imports) ran nowhere. Mirror the viscy-data job on ubuntu-latest for Python 3.12 and 3.13, and make the gate job wait for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(dynacell): run the launcher, engine and prediction-writer suites Port the test-dynacell-configs job from dynacell-models, minus the eight test files that only arrive with the dynacell PR: sync viscy-utils' test group with dynacell, and add test_submit_benchmark_batch, test_engine, test_pipeline_cache, feature_metrics_test and the viscy-utils prediction-writer, prediction-metadata, monitor-health and LR-cooldown tests. The dynacell-models PR target is not carried over. Source on dynacell-models: 33d2fee, 74fa18a (#497), 2d1ed32 (#506), 4450efd (#518), 2798dee (#519), 2f9ef31 (#528). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build: re-lock against the ported packages and main's dynacell Start from dynacell-models' uv.lock and re-resolve with `uv lock` against this tree, whose applications/dynacell/pyproject.toml is still main's. Against dynacell-models' lock this drops the ten dynacell-only packages (dinov3, etils, ftfy, jsonpath-rw, mlcroissant, pandas-stubs, ply, rdflib, submitit, termcolor) and resolves cubic back to 0.7.0; the dynacell PR restores them. Against main's lock it adds pytest-xdist and moves boto3/botocore 1.43.0 -> 1.43.20, cellpose 4.1.1 -> 4.2.1.1, pyarrow 19.0.1 -> 24.0.0, s3transfer and tracksdata; torch and lightning are unchanged. `uv lock --check` exits 0 (main's own lock does not). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(dynacell): DynaCell v1.1 benchmark configs from dynacell-models Port every DynaCell config path of dynacell-models verbatim: the model leaves (baselines, A549/joint/HEK/brightfield, the 2D track with its seed replicates and warmup-matched refit, the temporal and patch-vs-augmentation ablations, CELL-Diff-R2 and the DynaCell-lite leaves), the grouped and instance-AP eval buckets of the paper models, the shared overlays, train/predict sets and launcher profiles, and the deletion of 155 stale config files. The leaves are generated; review the generators and guard tests in the code commits, not each leaf. Spotlight v2 arm leaves, their four fit overlays and their 231 spotlight_v2_* eval buckets are left out; they follow in their own PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(dynacell): DynaCell v1.1 application code from dynacell-models Port every DynaCell code path of dynacell-models verbatim (226 non-merge commits since 2026-06-16; PRs #479-#530): the evaluation pipeline (cpdino/MorphEm/focus-aware 2D, focus2d slab metrics, foreground-limited metrics, MicroMS3IM, scale-invariant pixel metrics, CP shared space, provenance and cache invalidation), the paths registry, preprocessing/registry/croissant/distribution, the launcher and stall watchdog, the lite benchmark and CELL-Diff fast inference, the 2D track, the HF demo, docs, and the engine, celldiff_wrapper and mask_conditioning modules exactly as on dynacell-models. The three LightningModules keep their Spotlight training hooks (seg_aux, seg_consistency, mask_mode, target_bg_lowpass). Every hook defaults off, the bit-identity tests guard that, and only Spotlight leaves turn them on; two DynaCell-paper 2D checkpoints store seg_aux_weight in their hparams, so a hook-free engine could not resume them. mask_conditioning.py comes along because engine.py and celldiff_wrapper.py import it. The files that import the Spotlight leaf generator are trimmed in the commits that follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build(dynacell): lock the dynacell eval, croissant and preprocess deps Take dynacell-models' uv.lock verbatim, which this tree's pyprojects now match (`uv lock --check` exits 0). Against the shared-packages lock it moves cubic 0.7.0 -> 0.9.0a4 and adds dinov3 (pinned commit), etils, ftfy, jsonpath-rw, mlcroissant, pandas-stubs, ply, rdflib, submitit and termcolor. A fresh `uv lock` would instead float dinov3 to its current HEAD and bump mlcroissant/pandas-stubs, so the verbatim lock keeps the versions the paper evaluations ran with. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: drop the requires_data marker registration Its only user was main's generate_grouped_eval_configs_test.py, which the DynaCell port replaced: the live-data tests there are now marked slow. With strict markers, nothing needs the registration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(dynacell): run the release-path, provenance and watchdog suites Restore the eight dynacell test files the shared-packages PR left out of the test-dynacell-configs job, now that they exist: test_evaluation_microssim_calibration, segmentation_test, test_distribution, test_croissant, test_main_cli_routing, test_evaluation_fid_provenance, provenance_test and watch_stalled_jobs_test. The job now matches dynacell-models'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: add the runbook for jobs that hang while holding an allocation Port the DynaCell job runbook of the root CLAUDE.md from dynacell-models (6193b9c, bcbfbe6, c4c5ed0, 74fa18a #497, e1d75cf): how to tell a finished-but-hung job from healthy compute by the CPU-time derivative, the stall watchdog (tools/watch_stalled_jobs.py), per-family predict and fit wall limits, and the forensics to capture before cancelling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(dynacell): scope the grouped eval walk to registered model dirs The grouped generator imported ARMS from the Spotlight-v2 leaf generator, which is not part of this PR, to skip the Spotlight arms that share the canonical prediction tree. Drop that import and the Spotlight model names from _CODE_TO_PAPER, _DETERMINISTIC_MODELS and _CELLDIFF_MODELS, keeping the four 2D seed replicates the 2D appendix scores. The shared tree still holds the arms' stores, and parse_zarr_name raises on an unknown model token (e.g. the CellDiff "variant" segaux_iterative). walk_predictions now skips any model dir outside _REGISTERED_MODEL_DIRS before parsing and prints how many stores it skipped. The test that pinned the Spotlight skip becomes one that pins the unregistered-dir skip and its count. evaluation/paths.py keeps its Spotlight names: without them canonical_model_name would silently resolve an arm's run dir onto its baseline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(dynacell): drop the Spotlight leaf entries from the config guards The Spotlight arm leaves are not part of this PR, so: - test_benchmark_config_composition: drop the eight v1 *_spotlight train leaves, their iPSC predict leaves and data-hparam entries, the _UNMIGRATED_MODELS filter that only they needed, and the Spotlight members of _WIDE_GPU_TRAIN_LEAVES and _LONG_WALL_TRAIN_LEAVES (the 2D pix2pix seed1 replicate keeps its wide-GPU opt-out); - test_paths: drop the two tests that import the Spotlight leaf generator or depend on the grouped generator registering the arms. The canonical_model_name / resolve_model tests of the arm names stay, since evaluation/paths.py keeps those names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(dynacell): describe the 2D seed1 leaves without the Spotlight generator The 40 leaves of the four 2D baseline seed replicates were emitted by the Spotlight-v2 leaf generator, which is not part of this PR, and their headers told the reader to edit that generator. Say instead what they are for: the second draw scored by the *__2d_focus2d_seed1 buckets. Comments only; the configs are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(dynacell): drop the Spotlight eval generator from the generator lists evaluation/README.md and the hand-written-bucket guard's comment in test_evaluate_compose.py named generate_spotlight_v2_eval_configs.py among the generators that write grouped buckets; it is not part of this PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): declare torchvision, which evaluation.metrics imports viscy_utils.evaluation.metrics imports torchvision.ops at module level, and losses.MixedLoss imports that module, but viscy-utils never listed torchvision. Installs only worked because another workspace member pulled it in. The new test-utils CI job syncs viscy-utils alone and failed to collect six test modules with ModuleNotFoundError. The lock gains only the dependency edge; no version moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(dynacell): keep main's batch-launcher test and the writer tests out Two corrections to the test-dynacell-configs job: - main's tests/test_submit_benchmark_batch.py resolves leaves under /hpc and fails on a runner (5 PermissionErrors in CI). The dynacell-models version that runs there arrives with the DynaCell PR, which adds it back. - the four viscy-utils test files (prediction writer, prediction metadata, monitor health, LR cooldown) now run in the test-utils job on both Python versions, so the dynacell job no longer repeats them or syncs viscy-utils' test group for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(viscy-utils): share the fit-callbacks lookup between CLI injectors _inject_checkpoint_guardrails and _inject_optimizer_health_guard each repeated the same walk from the config to fit's trainer.callbacks plus the already-injected check; the `else config` branch of that walk could never run behind the `subcommand != "fit"` return. Both now call _fit_callbacks_without. _callback_field collapses its two identical branches, and the monitored checkpoint's init_args are kept from the search loop instead of being looked up twice. No behavior change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(viscy-utils): drop MonitorHealthCheck's unused check_finite flag Nothing sets check_finite (the CLI injects monitor and patience only, and no config or test passes it), so the callback always checks finiteness; it now does so with math.isfinite instead of building a tensor for one float. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(viscy-utils): stop describing fg masks as built from Otsu thresholds generate_fg_masks does not read the stored otsu_threshold; it binarizes a CLAHE-equalized, median-filtered volume. Its summary line and VisCyTrainer.preprocess's compute_fg_masks entry still said "from Otsu thresholds". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build: leave the hf_demo pytest path to the DynaCell PR Nothing on this branch imports from applications/dynacell/examples/ hf_demo: its only test, build_demo_data_test.py, arrives with the DynaCell PR, which adds the pythonpath entry back with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): guard a fit that sets no callbacks Trainer.callbacks defaults to None, and the CLI injectors returned early unless they found a list, so a fit that set no callbacks ran without OptimizerHealthCheck, the guard meant for every fit. The shared lookup now gives such a fit an empty list to append to (None and [] mean the same to Trainer). The checkpoint guardrails still need a monitored ModelCheckpoint and remain a no-op there. Reported by Copilot on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(viscy-utils): say what the AMP skip counter actually counts The counter resets only when GradScaler grows its scale, so successful optimizer steps can sit between the reductions it counts; the error message claimed every step was skipped "in a row". Describe it as reductions without growth in between, which is what it detects: the backward overflowing faster than growth_interval clean steps can restore the scale. Behavior unchanged. Reported by Copilot on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): resolve viscy_utils.callbacks names lazily VisCyCLI now injects MonitorHealthCheck and OptimizerHealthCheck into every fit by class path through viscy_utils.callbacks, whose __init__ eagerly imported OnlineEvalCallback and with it scikit-learn, which only the eval extra installs. A fit on a base install would therefore fail before training. The package now resolves its exports lazily (PEP 562), like viscy_utils itself. The new test blocks sklearn and imports the health callbacks: the eager __init__ fails it with ModuleNotFoundError, the lazy one passes. Reported by Copilot on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(cytoland): install viscy-data from GitHub in the VCP tutorials The tutorials install viscy-utils from the GitHub monorepo and let the siblings resolve from PyPI. The writer they predict with now reads SlidingWindowDataset.z_planes, which the PyPI viscy-data 0.1.0 does not have (checked in its wheel), so predict would fail with AttributeError in on_predict_start. Install viscy-data from GitHub too, in the .py sources and their paired notebooks. Found by /code-review on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): return an empty CLAHE result when the clip range is zero foreground_mask_from_volume guards a constant volume, but a blank volume with a few hot pixels (under 0.2% of voxels) has equal 0.1/99.9 percentiles too, and equalize_clahe then divided by zero and fed NaN to CLAHE (RuntimeWarning: divide by zero, reproduced on a (20, 64, 64) volume). equalize_clahe now returns zeros there, which the mask's existing flat-histogram guard turns into an empty mask, as for a constant volume. Volumes with any contrast are unchanged. Found by /code-review on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): keep a step-cadence leaf's last.ckpt on its step trigger The injected latest-weights ModelCheckpoint takes over save_last from the monitored one but copied only every_n_epochs (defaulting to 1). A leaf that checkpoints every N steps, typical for max_steps runs with long epochs, would get last.ckpt only at epoch boundaries, so a preempted run resumed from much older weights. The injected callback now copies the monitored callback's every_n_train_steps or train_time_interval when set; epoch-cadence leaves are unchanged. No current leaf combines a monitored checkpoint with a step trigger. Found by /code-review on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): skip only PAPER_KEY-known arms in the grouped eval walk The scope filter skipped every model dir outside the generator's registry and only printed a count. That also silenced the loud "unknown model code-name" a new, never-registered model used to raise, so its stores would drop out of every paper bucket without an error. Now the walk skips only dirs that paths.PAPER_KEY knows but this generator does not register (the Spotlight arms on this branch); anything unknown to both still raises in parse_zarr_name, as on dynacell-models. Tests pin both sides. Found by /code-review on #534. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(dynacell): say the instance-AP audit does not share the walk's arm skip audit_prediction_coverage claimed to apply the same skip filters as walk_predictions. It applies the same test-set filter, but on purpose not the walk's skip of PAPER_KEY models the grouped generator leaves unregistered: listing every unregistered model is what it is for. Found by /code-review on #534. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(dynacell): carry the seg-consistency engine and focus-parity tests test_seg_consistency_training.py exercises the seg_consistency hook of DynacellUNet and DynacellGAN with a stand-in segmenter, and test_seg_consistency_focus.py pins the loss's focus-plane rule to dynacell.evaluation.focus. Both test code this PR ships and import nothing from the Spotlight PR, so they belong here rather than with the Spotlight arms. Verbatim from dynacell-models. Found by /code-review on #534. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): take last.ckpt from every monitored checkpoint The guardrail turned save_last off only on the first monitored ModelCheckpoint. A leaf with two monitored checkpoints (say loss and a validation metric) kept a second last.ckpt writer next to the injected latest-weights callback, so last.ckpt would alternate between "latest" and "that monitor's cadence". All monitored callbacks now give it up; the first still names the monitor, directory and cadence. No current leaf has two monitored checkpoints. Found by /code-review on #533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(dynacell): one model-dir acceptance rule for parse and walk filter _REGISTERED_MODEL_DIRS restated, as a separate set, the rule _split_model_variant uses to accept a dir name, so the two could drift. _match_model_variant is now that rule (None when the dir does not parse); _split_model_variant wraps it and raises the same errors as before, and the walk filter calls it. No behavior change. Found by /code-review on #534. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): share the foreign-model skip between the walk and the audit walk_predictions skipped the Spotlight arms' dirs on the shared tree, but the instance-AP coverage audit still parsed them, so on this branch it reported 468 coverage errors on the live tree instead of dynacell-models' 102, and its docstring claimed it applied the walk's filters. Both now call one predicate, is_foreign_model_dir: a dir whose model paths.PAPER_KEY knows but the grouped generator does not parse. Measured on the live tree, the audit's error set is identical to dynacell-models' (102 entries; the remainder are pre-existing). The new audit test holds one registered, one foreign and one unknown store and expects exactly the unknown one. Found by /code-review on #534 (findings 1, 9, 10). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): keep raising on campaign models the generator cannot parse pix2pix2d_unetvit_wmatch{,_seed1} are in _CODE_TO_PAPER (and in PAPER_KEY) but not in _DETERMINISTIC_MODELS, so their stores do not parse. On dynacell-models the walk raises on them and the audit lists them. The foreign-model predicate would have skipped them as another campaign's arms. A model the generator itself lists in _CODE_TO_PAPER is not foreign: the walk raises and the audit flags it, as on dynacell-models. With this, the audit's error set on the live tree is identical to dynacell-models' (102 entries, compared path by path). Registering wmatch properly is a separate decision (its comment says "evaluate in focused sibling buckets", and adding it to _DETERMINISTIC_MODELS would fold it into the canonical buckets), so it is left as on dynacell-models. Found by /code-review on #534 (findings 2, 7). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(dynacell): stop pointing at the Spotlight leaf generator from this branch evaluation/paths.py and tests/test_paths.py cited tools/generate_spotlight_v2_leaves.py, which this branch does not have, and test_paths claimed a pin to it that only exists with the Spotlight v2 configs. Say where the generator lives instead. The Spotlight model names stay: without them canonical_model_name would resolve an arm onto its baseline. Comments only. Found by /code-review on #534 (finding 8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): parse the warmup-matched pix2pix2d stores in the grouped walk _CODE_TO_PAPER declares pix2pix2d_unetvit_wmatch{,_seed1} a parse-only registration "same rule as fnet2d_voxelmatched", but unlike that model they were never added to _DETERMINISTIC_MODELS. Their 56 stores on the live tree therefore raised "unknown model code-name", which alone stops walk_predictions. Register them like fnet2d_voxelmatched: they now parse, and, as for every parse-only ablation, they are scored in their focused sibling buckets (the committed *__2d_focus2d_pix2pix2d_wmatch leaves), not by regenerating the canonical ones. Fixes part of FOLLOWUPS #1 (live-tree walk crash). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): skip the listed checkpoint re-predicts in the grouped walk The live tree holds second-checkpoint predictions of registered models beside their canonical stores, under train-set dirs paths.py does not know: fnet3d_paper/ipsc_step{8400,11k}, fnet2d_voxelmatched/ipsc_step200k, fcmae_vscyto{2d_scratch_samplematched,3d_scratch}/ipsc_final and fcmae_vscyto3d_scratch/ipsc_repredict (34 stores). key_from_prediction_store raises on them ("unknown train_set token 'ipsc_final'"), which stopped walk_predictions and filled the instance-AP audit. They come from the voxel-matched FNet-2D and sample-matched UNeXt2-2D ablations, are scored only by hand-written experiment buckets, and no committed config references them, so they are not canonical inputs. Each (model dir, train-set dir) pair is now listed in _CHECKPOINT_REPREDICTS with its campaign; the walk skips and counts them and the audit skips them. A train-set dir that is neither canonical nor listed still raises. Fixes part of FOLLOWUPS #1 (live-tree walk crash). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): name the P1 segcons replicate dirs as foreign in the walk nucleus/pix2pix2d_unetvit_segcons_{s1,ctrl_s1,x10} (12 stores, Spotlight P1 follow-ups of 2026-10-19) sit in the canonical tree but are in neither this generator's registry nor paths.PAPER_KEY, so the PAPER_KEY-based foreign test cannot see them and they raised "unknown model code-name". They are now listed by name in _FOREIGN_MODEL_DIRS and skipped like the PAPER_KEY arms; the skip message no longer says "PAPER_KEY". Any other unknown dir still raises. The new audit test holds one store of each kind on the live tree (registered, checkpoint re-predict, named foreign, wmatch, unknown train set) and expects only the unknown one to be flagged. Fixes the rest of FOLLOWUPS #1 (live-tree walk crash). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): list checkpoint re-predicts per organelle _CHECKPOINT_REPREDICTS matched (model dir, train-set dir) in every organelle, so a future mito/fnet3d_paper/ipsc_step8400 store would have been skipped as a known re-predict instead of raising, though only the ER one exists. Entries are now (organelle, model, train set) triples for exactly the stores on the live tree; a listed token in another organelle raises "unknown train_set token" again (new test, fails before this change). The audit test now uses the nucleus step-11k store, since the step-8400 one is ER-only. Found by /code-review on the FOLLOWUPS #1 fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(dynacell): one skip rule for the grouped walk and the audit walk_predictions and audit_prediction_coverage each called is_foreign_model_dir and is_checkpoint_repredict inline, so a third skip category added to one could silently miss the other and split them again. Both now call skip_reason(zarr_path), which names the category; the walk counts per category. No behavior change. Found by /code-review on the FOLLOWUPS #1 fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(dynacell): guard the grouped registries against the walk crash class Two guards on the hand-kept registries behind the live-tree crash: - every _CODE_TO_PAPER model must parse. At a04b3eb this lists pix2pix2d_unetvit_wmatch{,_seed1}, the omission that stopped walk_predictions; - the name-based skip lists stay disjoint from what the campaign registers: no _FOREIGN_MODEL_DIRS entry in _CODE_TO_PAPER or the parse tuples (it would be dropped silently, since the foreign check runs first), and no re-predict token among the canonical train sets. Found by /code-review on the FOLLOWUPS #1 fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(dynacell): reflow the checkpoint re-predict comment The organelle-scoping edit left a broken line wrap in the _CHECKPOINT_REPREDICTS comment. Comment only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): keep the wmatch refit out of canonical and instance-AP leaves Registering pix2pix2d_unetvit_wmatch{,_seed1} for parsing (97c30a1) fixed the live-tree walk, but a default regeneration would then also fold their 56 stores into the canonical {organelle}_{train_set} buckets and the instance-AP leaves, against DON'T-FOLD: the committed *__2d_focus2d_pix2pix2d_wmatch buckets are where they are scored. They are now listed in _SIBLING_ONLY_MODELS: walk_predictions still returns them, so sibling buckets can be built from it, but main() buckets canonical_predictions() (the walk minus that list) and the instance-AP in_scope() drops them. New tests: canonical_predictions leaves them out, in_scope rejects them. MEASURED on the live tree, both generators run the default way (no --out-root) emit 0 wmatch blocks, and the canonical pool (628 stores) is identical, store by store, to what the pre-fix rules parse, so a default regeneration writes exactly what a04b3eb would have written had its walk not crashed. Found by /code-review on the FOLLOWUPS #1 fix (finding 2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(dynacell): retitle the paper and describe DynaCell as the broader framework The camera-ready title is now "Evaluating Dynamic 3D Virtual Staining of Live Cells with DynaCell": DynaCell names a broader framework for image-based profiling of cell-state dynamics, and this paper's evaluation framework is part of it. Update the title in CITATION.cff (software and preferred citation), the README and both HF card BibTeX entries, and the Croissant citeAs string; rewrite the CITATION.cff abstract and the README tagline to describe the evaluation framework as part of DynaCell. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA * docs(dynacell): call the contribution the DynaCell evaluation suite Mirror paper 1c85e84: "This evaluation framework is part of DynaCell, a broader framework" repeated the noun. The paper's contribution is now the DynaCell evaluation suite, and DynaCell stays the broader framework. The CITATION.cff abstract also says "a set of baseline models", so "suite" means one thing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA * docs(dynacell): retitle to The DynaCell Evaluation Suite The paper's title is now "The DynaCell Evaluation Suite for Dynamic 3D Virtual Staining of Live Cells" (Overleaf b73f25f). Updates CITATION.cff, the README and HF card BibTeX, and the Croissant citeAs. test_croissant.py: 17 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA * chore(dynacell): delete finished one-shot tools Each did a one-time job that is done, and nothing else imports them: - build_migration_manifest, run_migration, restandardize_config_paths (+ tests): the canonical-path migration was applied 2026-07-13; a rebuilt manifest on 07-24 had nothing left to move; - retokenize_deconv_predictions (+ test): the 24 a549__deconv / joint__legacy_deconvgt renames were applied 2026-07-22; paths.py still reads both tokens for the dirs kept on purpose; - scan_nan_microssim_caches (+ test): one audit for the cubic 0.9.0a1 NaN MicroMS3IM caches that #506 fixed; - add_instance_ap_to_ablation_leaves: one codemod, already applied; - smoke_joint_leaf: an April CPU smoke the config-composition tests now cover. The provenance builders (cpdino seg masks, HEK stores, A549 dual fuse, random-init checkpoints) stay. README entries and one bake_best_ckpt docstring line follow. Full suite: 5251 passed, 0 failed (the 18 fewer are the two deleted tests/ files; the 14 colocated tool tests were never collected by the root run). Raised in review by edyoshikun. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc --------- Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mirroring the PyTorch implementation from microDL at mehta-lab/microDL@3d8eb24.
Relicensing
Now licensed under BSD-3-Clause and the new entity name (CZ Biohub SF).
TODO: