Skip to content

build(dynacell): bump cubic to 0.9.0a2, keep 0.9.0a1 metric caches valid - #503

Merged
alxndrkalinin merged 7 commits into
dynacell-modelsfrom
bump/cubic-0.9.0a2
Sep 29, 2026
Merged

alxndrkalinin merged 7 commits into
dynacell-modelsfrom
bump/cubic-0.9.0a2

Conversation

@alxndrkalinin

@alxndrkalinin alxndrkalinin commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Moves the dynacell eval stack from cubic 0.9.0a1 to 0.9.0a2. Existing 0.9.0a1 metric caches stay valid.

  • Pin. Updates the pyproject git-tag pins, uv.lock (the cubic entry only; uv lock --check passes), REQUIRED_CUBIC_VERSION, and the run_eval_direct.slurm default venv, which is now VisCy-venvs/cpdino-eval-cubic090a2.
  • Cache compatibility. Adds CUBIC_VERSIONS_EQUIVALENT_TO = {"0.9.0a2": frozenset({"0.9.0a1"})}. It is keyed by the declared version, so the next bump starts with no equivalents. metrics_provenance_matches accepts caches stamped with the versions listed for REQUIRED_CUBIC_VERSION, but only while that pin is installed. The CP-space binding rules are unchanged, and 0.8.0a2 caches are still refused.
  • Scan tool. Adds tools/scan_nan_microssim_caches.py, a read-only scan. See the caveat below.

Release scope

git diff v0.9.0a1..v0.9.0a2 touches cubic/__init__.py (the version), cubic/metrics/microssim/ri_factor.py and that file's tests. Nothing else changes. The RI-factor fit now reduces its objective in bounded chunks, accumulating in float64 where 0.9.0a1 took a float32 .mean(), and pools the per-slice elements into preallocated buffers. Peak memory drops from 40.5 to 12.9 GB.

Two edge cases now raise ValueError: get_global_ri_factor on 0 slices and get_ri_factor on empty elements. In dynacell, _calibrate_microssim returns early on 0 positions or pairs. The one remaining empty case (D=0) already raised ValueError under 0.9.0a1 (np.concatenate([])), and the caller catches ValueError, so behavior is unchanged.

Equivalence (measured)

cpdino-eval-cubic090a2 was built from the cpdino-eval-cubic090a1 uv pip freeze with --no-deps, changing only the cubic line. uv pip freeze diff: cubic only.

Acceptance tolerance: MicroMS3IM relative delta ≤ 1e-6, far below the tables' 2-decimal rounding.

MicroMS3IM at production calibration settings. Leaf: A549-trained fnet3d_paper ER a549__denv, 12 FOVs. Calibration used the default max_pairs=12 and seed 42, giving a 576×640×960 float32 pool. MicroMS3IM was then scored on every (FOV, t). No torch model was loaded; the run was on an A40.

cubic 0.9.0a1 cubic 0.9.0a2 delta
fitted alpha 17.883591651916504 17.883591651916504 0
MicroMS3IM, 108 (FOV, t), range 0.663–0.899 max abs 0, max rel 0

The fit objective itself does change: on a fixed 640×960 float32 pair it moves about 7e-8 relative between the versions (S(α) 0.97342079878 vs 0.97342073207). The root finder lands on the same alpha anyway, so the comparison can detect a difference in the objective even though the result is identical.

Other columns. A full eval of 1 FOV × 7 timepoints of the same leaf ran with max_pairs=4. It produced bit-identical pixel_metrics (PCC, SSIM, NRMSE, PSNR, SI_*, Spectral_PCC, XY/Z_FSC) and mask_metrics columns. These columns do not call the RI-factor code. CP feature columns were not scored in that run; the release diff contains no CP code.

Caveat: all-NaN MicroMS3IM in old caches

Where the 0.9.0a1 calibration ran out of memory (pre-#502 code), a cache holds MicroMS3IM = NaN for every FOV, and 0.9.0a2 would have produced a value. Reusing such a cache keeps the NaN. tools/scan_nan_microssim_caches.py --root /hpc/projects/virtual_staining/training/dynacell lists them.

Scan after the re-eval jobs 36590711–18 finished (2026-09-29 09:36 PDT, rc=0): 930 sidecars, all stamped 0.9.0a1. 35 are all-NaN, and 0 are unexplained. All 35 are the mito A549 mock data_range case, which predates this bump:

  • 33 canonical mito/<model>/<train>/a549__mock dirs: 11 models × {a549, ipsc, joint} train sets, except celldiff_r2 (a549 and joint) and celldiff_r2_iterative (ipsc only);
  • 2 legacy a549/evaluations_with_embeddings/eval_{phase,vscyto3d_randinit}_mitochondria_mock dirs.

No cache holds an OOM-caused NaN. The gate itself gets no special case.

Tests

Suite: provenance_test.py, tests/test_evaluation_*.py, test_pipeline_cache.py, test_pixel_metrics_parity.py, tools/scan_nan_microssim_caches_test.py.

  • 0.9.0a1 venv on origin/dynacell-models: 264 passed, 1 failed.
  • 0.9.0a2 venv on this branch: 274 passed, 1 failed.

The failure in both is the known order-dependent test_load_eval_models_pins_morphem_to_the_configured_revision.

Not in this PR

🤖 Generated with Claude Code

https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA

alxndrkalinin and others added 2 commits September 28, 2026 23:29
cubic 0.9.0a2 chunks the MicroSSIM RI-factor objective and pools the
per-slice SSIM elements into preallocated buffers, dropping the fit's
peak from 40.5 to 12.9 GB on 576x640x960 float32. Nothing else in the
release touches numeric code (git diff v0.9.0a1..v0.9.0a2 changes only
cubic/metrics/microssim/ri_factor.py plus its tests).

The uv.lock change is the cubic entry and its two requires-dist refs
only; `uv lock --check` passes on it. REQUIRED_CUBIC_VERSION and the
run_eval_direct.slurm default venv (cpdino-eval-cubic090a2, same
freeze as cpdino-eval-cubic090a1 except cubic) move with the pin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA
….9.0a2

metrics_provenance_matches refuses any cache stamped with another cubic,
so the 0.9.0a2 pin alone would force every leaf to recompute. The bump
does not move a value: scoring one FOV x 7 timepoints of the A549-trained
fnet3d_paper ER a549__denv predictions under both eval venvs gives
bit-identical pixel_metrics and mask_metrics columns (MicroMS3IM
included) and the same fitted alpha, 20.037598609924316. CP feature
columns were not scored in that run; the release diff touches no CP
code.

CUBIC_VERSIONS_EQUIVALENT_TO_REQUIRED lists the prior versions the
declared pin reproduces. They match only while the declared pin is
installed, and the CP-space binding check is unchanged. 0.8.0a2 caches
are still refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA
The test inlined the same write/load/overwrite/save sequence the new
helper provides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The dependency and cache-provenance changes are consistent and tested; only minor documentation corrections remain.

Review effort: Balanced
Findings: 2 Low severity

Open (2)
What changed in this PR

Updates Dynacell’s evaluation stack to cubic 0.9.0a2 while retaining compatible 0.9.0a1 metric caches.

Changes:

  • Bumps cubic pins, lockfile entry, and evaluation environment.
  • Adds guarded cache compatibility for cubic 0.9.0a1.
  • Tests and documents the updated provenance contract.
File Description
uv.lock Locks cubic 0.9.0a2.
applications/​dynacell/​tools/​run_eval_direct.slurm Uses the new evaluation environment.
applications/​dynacell/​src/​dynacell/​evaluation/​provenance.py Adds version-equivalent cache handling.
applications/​dynacell/​src/​dynacell/​evaluation/​provenance_test.py Tests compatibility and rejection rules.
applications/​dynacell/​src/​dynacell/​evaluation/​CLAUDE.md Updates evaluation guidance.
applications/​dynacell/​pyproject.toml Pins cubic 0.9.0a2.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread applications/dynacell/pyproject.toml Resolved
alxndrkalinin and others added 4 commits September 28, 2026 23:47
…version

A flat set of equivalent versions would keep accepting 0.9.0a1 caches
after the next pin bump, although no one measured 0.9.0a1 against the
new version. CUBIC_VERSIONS_EQUIVALENT_TO maps each declared version to
the versions measured to reproduce it, and the gate looks up
REQUIRED_CUBIC_VERSION, so a bump starts with no equivalents.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA
Reusing 0.9.0a1 caches under 0.9.0a2 keeps whatever they hold. Where the
0.9.0a1 MicroSSIM calibration ran out of memory, the cache holds
MicroMS3IM=NaN for every FOV, and 0.9.0a2's bounded fit would have scored
it. The scan lists such caches so they can be recomputed. Mito A549 mock,
which is all-NaN for an unrelated data_range reason, is reported as known.

Run on /hpc/projects/virtual_staining/training/dynacell on 2026-09-28:
926 sidecars, 35 all-NaN stamped 0.9.0a1, all mito A549 mock, 0
unexplained. Read-only; the eval gate is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA
The first measurement fitted alpha on 4 calibration pairs. Production
fits 12 (576 slices), and 0.9.0a2 also changes the objective's
accumulation from float32 to chunked float64, so bit-identity at 4 pairs
did not carry over. Re-measured on the A549-trained fnet3d_paper ER
a549__denv leaf with max_pairs=12, seed 42: alpha is 17.883591651916504
under both versions and MicroMS3IM over all 108 (FOV, t) differs by 0.
The objective itself moves ~7e-8 relative, which the root finder absorbs.

The equivalence is now stated as within a tolerance (MicroMS3IM relative
delta <= 1e-6) instead of bit-identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA
"Only bounds memory" undersold the release. 0.9.0a2 also accumulates the
RI-factor objective in float64 instead of float32 and raises ValueError
on an empty pool. The pyproject note that a pin bump "is paired with a
full re-eval" contradicted this bump, which instead reuses caches through
a measured equivalence entry. It now names both routes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015kuht3npKtb2o5E6h19xbA

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The dependency bump, compatibility gate, operational scanner, and associated tests are consistent and complete.

Review effort: Balanced
Findings: None

Resolved since last review (2)

@alxndrkalinin
alxndrkalinin merged commit 1597a34 into dynacell-models Sep 29, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants