Skip to content

refactor(track): --init/--cluster tracking CLI with Nextflow subworkflow - #281

Merged
tayllatheodoro merged 7 commits into
mainfrom
refactor-track
Jul 29, 2026
Merged

tayllatheodoro merged 7 commits into
mainfrom
refactor-track

Conversation

@ieivanov

@ieivanov ieivanov commented Jul 11, 2026 •

Copy link
Copy Markdown
Collaborator

Re-opens the tracking work after #280 was merged then reverted on main (see revert commit on main). Same content as #280, with Taylla Theodoro credited as co-author. To be merged after testing.

Summary

This tracking-only PR is preferred over #260 — splitting tracking out of the combined track + rename-channels PR lets it be tested and merged on its own. Rename-channels stays in #260 (needs further discussion).

  • Track CLI: add --init and --cluster (slurm/local/debug) to the unified biahub track, plus --input-images-path for Nextflow to override config input paths. Factor _init_output_plate following the deskew/flat-field/reconstruct pattern.
  • CLI consistency: type hints match the option callbacks (list[Path]/Path), provenance written to each output position's zattrs (biahub-track), canonical slurm_args (job name, slurm_mem preserving total node memory, parallelism comment) and docstring, focus-based z-resolution, and a cellpose segmentation backend.
  • Monkey-patch: port _patch_ultrack_readonly_buffer() (scikit-image #6378 — Cython _map_array rejects the read-only memoryviews numpy 2.x produces). Remove when scikit-image >= 0.27 is available.
  • Nextflow: add nextflow/modules/tracking.nf (biahub track --init / --cluster debug); update example_track_settings.yml.
  • Settings: correct ProcessingInputChannel.path and TrackingSettings.blank_frames_path annotations to Path | None to match their validators.

Scope

Tracking only — 5 files. Excludes everything rename-channels, which remains in #260.

Test results

  • pytest tests/test_cli/test_track_cli.py — 6 passed (init, debug single-position, local, blank-frames).
  • ruff check / ruff format clean.

Test plan

  • Unit tests (init, debug, local, blank-frames)
  • CLI integration on real data
  • Nextflow subworkflow end-to-end (GPU, multiple positions)

🤖 Generated with Claude Code

ieivanov and others added 2 commits July 10, 2026 19:27
Split the tracking half out of #260 so it can be tested and merged
independently of the rename-channels work.

- Add `--init` and `--cluster` (slurm/local/debug) to the unified
  `biahub track` CLI, plus `--input-images-path` for Nextflow to
  override config input paths. Factor `_init_output_plate` following
  the deskew/flat-field/reconstruct pattern.
- Align the CLI with the canonical command template: type hints match
  the option callbacks, provenance written to output zattrs, canonical
  slurm_args/docstring, focus-based z-resolution + cellpose backend.
- Port `_patch_ultrack_readonly_buffer()` (scikit-image #6378 workaround)
  so the unified CLI carries the read-only-buffer fix ultrack needs.
- Add `nextflow/modules/tracking.nf` calling `biahub track --init` /
  `--cluster debug`, and update example_track_settings.yml.
- Fix ProcessingInputChannel.path / TrackingSettings.blank_frames_path
  annotations to Path | None to match their validators.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Taylla Milena Theodoro <tayllamth@gmail.com>
Wire track_wf into mantis-v2.nf after virtual-stain, following the same
chaining pattern as PR #263. Tracking is a 2-input step: it reads the
reconstruct output for plate structure and the virtual-stain output for
image data, gated on virtual_stain_done.

- add params.track_config + validation
- include track_wf and add track: '4-track' to the directory layout
- call track_wf(all_positions, reconstruct_output, virtual_stain_output,
  track_output, track_config, virtual_stain_done.done)

Co-authored-by: Taylla Milena Theodoro <tayllamth@gmail.com>
@tayllatheodoro
tayllatheodoro force-pushed the refactor-track branch 6 times, most recently from 4b01c54 to a89ba4d Compare July 24, 2026 00:08
Replace the four overlapping z-handling mechanisms (mode, z_range with
magic sentinels, use_focus/z_total/focus_*, and implicit np.mean
projection) with a single `z_slicing` config plus `output_mode`.

- Add ZSlicing (method: all|central|range|focus, default all). The
  method decides which of the other fields are used; the rest have
  defaults and are ignored (focus window_size defaults to 48).
- resolve_z_slice drives read-time slicing for all/central/range; focus
  loads the full stack and resolves its fixed window per-FOV after
  loading (apply_focus_slicing), from the already-loaded focus channel
  (no extra zarr open). The same window is applied to every channel so
  they stay Z-aligned; with dask only the focus channel's stack is read
  in full.
- _init_output_plate uses the fixed window count (min(window_size, Z)),
  so the output plate Z always matches the tracked Z (fixes focus+3D).
- Correct the output write for 3D (T,Z,Y,X) and guard that
  output_mode=2D requires projected (T,Y,X) labels.
- Resolve a null primary input_images path to --input-images-path if
  given, else to the -i input plate; later null entries stay virtual.
- Update example YAML, docstrings (two-source input model), and tests.
Cellpose tracking jobs are long-running and GPU-bound; the preempted
partition (2-day cap, evictable) could kill them mid-run. Default to the
gpu partition instead; override via --sbatch-filepath when needed.
Tracking OOM-killed a 64 GB GPU worker on a (67, 6, 86, 1664, 1193) float32
timelapse: three sites eagerly called .compute() on a whole (T, Z, Y, X) channel,
a 42.6 GB single allocation, even though none of them needed it all at once.

- run_preprocessing_pipeline: feed the pipeline lazily. array_apply already
  slices one timepoint at a time into a zarr output, and a reduction over a dask
  array (np.mean) streams chunk-wise; only the much smaller result is
  materialised. numpy dispatches on dask via __array_function__, and functions
  that are not dask-aware fall back to np.asarray internally -- the same eager
  behaviour this replaces, so it is never worse.
- apply_focus_slicing: keep the focus channel lazy. _median_focus_plane already
  indexes per timepoint, so this reads one (Z, Y, X) volume per step instead of
  the full stack. Restores the pre-refactor footprint, where focus resolved to a
  read-time slice rather than load-everything-then-window.
- cellpose_segmentation: mean-project before materialising, so peak is the
  (T, Y, X) result rather than the (T, Z, Y, X) stack.
- fill_empty_frames_from_csv: it mutates in place, so materialise there -- but
  only when there are frames to fill.

Measured on the FOV that OOM-killed at 64 GB (z_slicing.method: all, all 86
planes): peak RSS 2.52 GB, verified under a 16 GB cap.
@tayllatheodoro

tayllatheodoro commented Jul 27, 2026 •

Copy link
Copy Markdown
Collaborator

Real-data integration test: passing

Tracked 2026_07_14_A549_MAP4_ZIKV (input (67, 6, 86, 1664, 1193) float32, cellpose nuclei, z_slicing.method: all) on SLURM GPU.

The first attempt OOM-killed a 64 GB worker. Root cause was three sites eagerly calling .compute() on a whole (T, Z, Y, X) channel — a 42.6 GB single allocation — none of which needed the full stack at once. Fixed in 0ef2eec:

Site Was
run_preprocessing_pipeline materialised 42.6 GB before np.mean; array_apply already slices per timepoint and dask reductions stream chunk-wise
apply_focus_slicing materialised the focus channel, though _median_focus_plane already indexes per timepoint
cellpose_segmentation computed then mean-projected
fill_empty_frames_from_csv mutates in place, so it does need concrete arrays — now materialises only when there are frames to fill

Worth calling out: apply_focus_slicing was a regression this PR introduced. Pre-refactor, focus resolved to a read-time slice, so load_data only ever read window_size planes; a58bcd3 changed it to load-everything-then-window. So method: focus would have OOM'd identically to method: all — the earlier DENV run survived only because its 40-plane window was applied at read time.

Results

Before After
Outcome OOM-killed COMPLETED
Elapsed — 1:55
Peak RSS >64 GB 10.2 GB

Isolating the load+projection under a 16 GB cap: peak RSS 2.52 GB on the FOV that previously OOM'd at 64 GB.

Output: (67, 1, 1, 1664, 1193) uint32 nuclei_labels, labels present in all 67/67 timepoints, ~24 nuclei/frame, 78 track labels, biahub-track provenance written.

No resource-estimation change was needed — 64 GB now has ~6x headroom.

  • Unit tests (init, debug, local, blank-frames) — 20 passed
  • CLI integration on real data
  • Nextflow subworkflow end-to-end (GPU, multiple positions)

… + virtual-stain

Track was a 2-input step: -i pointed at 2-reconstruct for the plate
structure and --input-images-path at 3-virtual-stain for the pixels, with
both branches running in parallel after virtual-stain.

The assembled plate already carries the phase and virtual-stain channels
under their original names (concatenate preserves channel names and only
dedups collisions), so track can read one store for both inputs. track.py
already supports this: the first input_images entry with a null path
resolves to --input-images-path, so no config change is needed.

Track now waits on assemble_done rather than virtual_stain_done, so it
runs after assemble instead of alongside it. Two consequences, both noted
in the workflow comments: wall clock grows by the assemble time, and any
Z/Y/X crop or time_indices subset in the concatenate config is what
tracking sees.

@tayllatheodoro tayllatheodoro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Full nf pipeline and tracking working, tested with dataset /hpc/projects/intracellular_dashboard/organelle_dynamics/2026_03_26_A549_CAAX_H2B_DENV_ZIKV

@tayllatheodoro
tayllatheodoro merged commit 995fc7b into main Jul 29, 2026
5 checks passed
@ieivanov

Copy link
Copy Markdown
Collaborator Author

Thanks for testing and merging this PR!

ieivanov added a commit that referenced this pull request Jul 31, 2026
Resolve conflict in biahub/track.py: keep main's --init/--cluster
tracking CLI (#281), which moves plate creation and resource estimation
above the conflicted block, and adapt the estimate_resources call to the
3-tuple (time_minutes, num_cpus, gb_ram_per_cpu) return from this branch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants