Repository navigation
feat(nextflow): make pipeline steps optional (skip tracking for neuromast/zebrafish) #306
Description
Problem
nextflow/mantis-v2.nf runs a fixed chain: flat-field → deskew → reconstruct → virtual-stain → assemble → track. Every step is mandatory. The workflow body errors without each config:
if (!params.track_config) error "Provide --track_config"
if (!params.concatenate_config) error "Provide --concatenate_config"and then unconditionally wires the step:
track_trigger = assemble_done.done
track_wf(all_positions, track_input, track_input_images, track_output, params.track_config, track_trigger)Tracking is an A549 step. For neuromast/zebrafish datasets the deliverable is 5-assemble; tracking is not part of what those runs are for, and the tracking parameters (cellpose diameter, min_area/max_area, linking max_distance) are tuned for A549 cells, not neuromasts.
There is no way to express that today. A zebrafish run has to:
- author or copy a
track.ymlit does not want, - pass it to
--track_configpurely so the workflow validates and starts, - spend GPU wall time on a
4-trackfan-out after assemble, - produce a
4-track/<ds>.zarrthat is discarded.
Since #282 / #279 reordered track to run after assemble and read the assembled plate, this is strictly serial: the unwanted step delays nothing else, but it does extend the run past the point where the deliverable already exists.
What I'd like
A way to declare which steps a run performs. Rough shape, not a strong opinion on syntax:
--steps flat_field,deskew,reconstruct,virtual_stain,assemble # neuromast
--steps flat_field,deskew,reconstruct,virtual_stain,assemble,track # A549 (default)with the config requirement following from the selection — --track_config required only if track is in the list. Nextflow's native -entry is the other obvious option, but it needs one named entry workflow per useful prefix and does not compose as well as a list.
Two things worth preserving:
directory_layout()stays the single source of truth. A skipped step should not renumber the ones after it —5-assembleshould stay5-assemblewhether or not track runs, so paths remain comparable across datasets.- Validate the selection up front. Related: #304 asks for all step
inits to run at the start so config errors surface immediately rather than hours in. Step selection should be validated in the same early pass — including rejecting a selection whose first step's input does not exist.
Why it matters beyond zebrafish
The same gap shows up whenever the chain needs to stop early or restart partway:
- re-running only
assembleafter fixing a concatenate config, without re-validating the upstream configs - stopping after
reconstructto eyeball phase before committing GPU hours to virtual staining - #292's verify-then-clean, which wants assemble and track decoupled
Workaround today
Pass a placeholder track.yml and discard 4-track. This is what the reconstruct-with-nextflow skill does for zebrafish (PR #305, templates/configs/zebrafish/track.yml), with the placeholder marked as such and the plan telling the user the output is a by-product. Once this lands, that placeholder and the caveat around it can be deleted.
Activity
tayllatheodoro commentedon Aug 11, 2026
Prior art: #250, and why it does not cover this
#250 ("standalone step entry points") is the closest existing work, but it does not resolve this, for three reasons:
isDraft: true,mergeable: CONFLICTING, last updated 2026-05-19.nextflow/mantis-v2-timelapse.nf;mainnow has onlynextflow/mantis-v2.nf.rename-channelspiece overlaps refactor: rename-channels Nextflow subworkflow (track superseded by #281) #260.It is also a different mechanism. Standalone entry points let you rerun one step in isolation. That does let a zebrafish run avoid tracking by driving steps individually, but it does not make a step optional within the pipeline — which is what is wanted, so that one
nextflow runproduces the right deliverable for the dataset family.Verified on
maintoday:nextflow/mantis-v2.nfhas a singleworkflow {block and no named entry workflows, andbiahub nfexposes onlylist-positions. So there is no step-selection mechanism at all right now.Fresh evidence for the priority
A smoke test on
2026_08_05_dynatrack_2dpf(8 positions, T=2) ran the full chain: flat-field, deskew, compute-transfer-function, apply-inv-tf, virtual-stain-preprocess, virtual-stain and concatenate all COMPLETED on the first attempt, and5-assemblecame out correct — 6 channels, finite data.The run still reported
FAILED, purely because ofrun_track— the step this dataset family does not want. Two separate faults there (#309): a cupy/CPU-pathTypeError, and the task dropping into(Pdb)inside the SLURM job, which holds a GPU allocation to the wall limit and then gets retried because a time-limit kill is in the retryable 130–145 range.So the cost of a mandatory step is not just wasted wall time — it turns a fully successful reconstruction into a failed run, and burns GPU allocations retrying a deterministic failure in work that was going to be discarded.
Being able to say
--steps ...,assemblewould have made that run green and saved 6 × (wall limit) × 8 positions of GPU time.tayllatheodoro commentedon Aug 11, 2026
Cross-ref: the workaround in use today is a marked placeholder
track.ymlplus discarding4-track, shipped in #305 (templates/configs/zebrafish/track.yml,references/caveats.md§4). Both come out when this lands.feat(nextflow): assemble, track and QC are optional (#306)
docs(skill): reconstruct-with-nextflow gains QC, and drops the track …
refactor(nextflow): step directory numbers are positions, not names
feat(qc): QC integrated into mantis-v2, with imaging-qc as a direct d…