[alxndrkalinin]

Summary

Adds opt-in foreground-limited pixel metrics to the dynacell evaluation: FG_PCC, FG_SI_SSIM, FG_SI_NRMSE, FG_SI_PSNR and FG_frac in pixel_metrics.csv, behind a new pixel_metrics.foreground block that is off by default. With it off, the pixel/mask CSVs, NPYs, provenance file and a GPU spectral run are byte-identical to ba71f4cb.

Motivation: Spotlight v1 (arXiv:2507.05383) reports PSNR/MicroMS3IM/FRC on the foreground because the background dominates the voxel count. Its evaluation code zeroes the background of both images and runs the whole-volume metric. That creates sharp edges, and it inflates PSNR by exactly 10·log10(1/FG_frac). This PR measures the foreground region without modifying the images.

Design

• Foreground weight w, from the GT only, so every model is scored on the same region.
  • Gaussian smoothing (0.5 µm for nucleus, membrane and mito; 1.0 µm for ER), then Otsu on the smoothed volume, then a 0.5 µm Gaussian feather, giving w in [0, 1].
  • Zero-padded GT planes are excluded from the threshold.
  • Defaults are in a code table keyed by target_name. Grouped leaves set target_name directly and never load the target config group.
  • Raw Otsu (source: otsu, Spotlight's mask) is available for comparison.
• Pointwise metrics:
  • FG_PCC is a w-weighted Pearson.
  • FG_SI_NRMSE and FG_SI_PSNR use a w-weighted scale-invariant affine fit and weighted MSE, with data_range taken from the GT inside the hard foreground.
• FG_SI_SSIM: normalized (partial) convolution SSIM in the spirit of dycheck (Gao et al. 2022).
  • Local moments come from w-weighted Gaussian windows, normalized by the local weight mass.
  • The SSIM map is averaged with w, using skimage's border crop.
  • There is no image zeroing and no window erosion. cubic's eroded masked mean is empty for structures 1–9 voxels wide.
• Invariant (tested): with w ≡ 1, the FG columns reproduce PCC/SI_SSIM/SI_NRMSE/SI_PSNR to ≤ 7e-16 in float64 on a real A549 volume (GPU and CPU).
• Cache: the foreground recipe is recorded in the metrics provenance. The cache check requires the FG columns when the block is enabled, so a parameter change forces a recompute; a missing recipe also recomputes.
• Other:
  • metrics.py stays import-light, as in perf(dynacell/eval): keep torch off the light evaluation import path #510.
  • In reporting tables, FG_PCC, FG_SI_SSIM and FG_SI_PSNR rank higher-is-better. FG columns are not added to the default metric lists.
• Deferred: MicroMS3IM (scalar-only and fit per leaf) and the spectral metrics, because a mask correlates the two images at high frequency.

Validation

• Synthetic tests:
  • Identical foreground with a noisy background: FG gives 1.0 PCC and 1.0 SSIM, against 0.62 / 0.44 whole-image.
  • Error confined to the foreground: FG drops further than whole-image (SSIM 0.86 vs 0.98).
  • Uncorrelated texture inside identical blobs: Spotlight's zeroed background gives PCC 0.996 and SSIM 0.993. Ours, on the same raw Otsu mask, gives −0.03 and 0.008.
• Real data (50 iPSC lite FOVs and 12 A549 mock FOVs × 3 timepoints; nucleus and membrane; Spotlight-v2 segaux vs baseline pairs):
  • Whole-image and FG verdicts differ in 14 of 24 cells, never with opposite significant signs.
  • FG is a different readout, not a steadier one, so Spotlight's "whole worsens, FG consistent" pattern does not reproduce.
• Cost: about +0.12 s per 48×640×960 volume and +113 MiB GPU memory on an A40.

Caveats

• The membrane foreground is intensity-defined: bright puncta plus the basal sheet on iPSC, with lateral walls only partly included. It is a "signal region", not a membrane mask.
• The 0.5 µm feather also weights object edges, so FG columns reward correct placement as well as interior fidelity; setting feather_sigma_um: 0 scores the interior only.
• No full dynacell evaluate run with FG on has been done yet. It is covered by the cache-only pipeline fixture and offline scoring.
• Pre-existing and unrelated: three monkeypatch tests in provenance_test.py are order-dependent when run in one process after live_pipeline_module() users. The default testpaths never hits this.

Tests

• Full dynacell suite in the main venv (-n 11): 3568 passed, 175 skipped, 0 failed.
• New and touched test files, rerun independently: 58 passed, pytest rc 0.
• Eval venv, serial: 60 passed, plus provenance_test 23 passed when run alone.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

[Copilot]

Copilot review overview

🟡 Changes recommended

Foreground edge cases can include padded planes or terminate evaluation with division-by-zero errors.

Review effort: Balanced
Findings: 1

Open (1)

• Re-clear padded planes after feathering weights · New

What changed in this PR

Adds opt-in foreground-limited pixel metrics to DynaCell evaluation.

Changes:

• Implements weighted foreground metric computation.
• Adds configuration, provenance, and cache invalidation.
• Adds reporting support and comprehensive tests.

File	Description
test_reporting_tables_extended.py	Tests foreground metric ranking.
test_evaluation_pipeline.py	Updates pipeline stubs.
test_evaluation_foreground_pipeline.py	Tests pipeline integration and caching.
test_evaluation_foreground_metrics.py	Tests foreground metric behavior.
reporting/​tables.py	Defines foreground ranking direction.
evaluation/​README.md	Documents the feature.
evaluation/​provenance.py	Records foreground recipes.
evaluation/​provenance_test.py	Tests provenance matching.
evaluation/​pipeline.py	Integrates foreground scoring and cache validation.
evaluation/​metrics.py	Implements foreground weights and metrics.
evaluation/​_configs/​eval.yaml	Adds default-off configuration.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.