Repository navigation
Description
Problem or use case
A Nikon NIS-Elements plate scan writes one .nd2 file per well, named like:
Plate000_WellA01_ChannelBrightfield,GFP-531_Seq0000.nd2
Plate000_WellA02_ChannelBrightfield,GFP-531_Seq0001.nd2
...
Our datasets include single-plane scans (e.g. 282 files on a 384-well plate) and time-lapse z-stacks (48 files, T=43 × Z=49 × C=2 × 2704 × 4263, several TB total).
iohub convert handles one ND2 file at a time: stage positions (P) become FOVs, and with no plate metadata (ND2Dataset sets stage_positions = []) they go into a single row (0/0/0, 0/1/0, ...). With per-well files this means:
- there is no way to get one HCS plate from the folder;
- nothing reads the well from the file name. The
Seqnumber is serpentine acquisition order, so ordering by file name orSeqputs wells in the wrong place; - wells in one run can have different T (acquisition stopped early; see ND2 conversion writes garbage for frames past
sequenceCount(acquisitions stopped early) #481), but the converter takes the array shape from a single reader.
Proposed improvement
- Folder input.
read_images(<dir>)/iohub convert -i <dir>detects a directory of.nd2files and returns a newND2PlateDataset(aMicroManagerFOVMappinginiohub/nd2.py) with one FOV per (well, stage position). - Well from the file name. Parse
Well(?P<row>[A-Z])(?P<col>\d+)(configurable regex, e.g. a CLI option) intohcs_position_labels→A/1/0, and fillstage_positions[i]["Label"]so position names are the well ids. Never useSeqfor placement. Error on two files for the same well. - Uneven T. Plate T = maximum over wells. Shorter wells are zero-padded lazily (dask) and the padded frames are recorded in the position attributes. Mismatched C/Z/Y/X/dtype raises an error.
- Consistent metadata. Check that channel names, voxel size and the
TimeLoopperiodMsagree across files (and pick the interval from the loop whosetype == "TimeLoop", not the first loop that has a period). - (Optional) Sharding in the converter for v0.5, e.g.
(1,1,1,Y,X)chunks in(1,1,Z,Y,X)shards. For a 49 × 2704 × 4263 uint16 stack the default ZYX chunk is ~1.1 GB and gets Z-halved by the 500 MB cap, which is awkward to read either plane-wise or volume-wise. - (Later) For multi-TB runs on a cluster: resumable conversion (record finished slabs so
convert(slabs=...)can skip them after a preempted job) andND2Dataset.frame_locations()so ND2 can use the byte-range source instead of the lock-serialized dask source.
Alternatives
We have a standalone pipeline that does this today: it builds the plate with iohub.ngff.utils.create_empty_plate (so the output works with biahub) and writes one (well, timepoint) at a time. Parts that could be ported:
- file discovery: parse
Well<Row><Col>(zero-padded ids likeA01↔ HCS pathsA/1), reject duplicate wells, infer the plate format (96/384) from the well ids; - metadata: expected frames (
len(loop_indices)) vs written frames (attributes.sequenceCount),TimeLoopinterval, channel names, voxel size; - reading: stream one timepoint frame by frame (
loop_indices+read_frame), never loading a whole file (up to ~100 GB), with the missing-frame handling from ND2 conversion writes garbage for frames pastsequenceCount(acquisitions stopped early) #481; - writing: T = max over wells, zero-fill provenance in the zattrs, per-(well, t) marker files for resume (read with a directory listing, because per-file
exists()misses files other nodes just wrote on NFS); - tests that monkeypatch the nd2 readers to simulate missing frames and uneven T.
Happy to contribute these as a PR on a feat/nd2-plate-per-well branch if this direction sounds right.
Additional context
Related: #481 (frames past sequenceCount), #434 / #436 (single-file ND2 reader).
Activity