Skip to content

Convert a folder of per-well ND2 files (Nikon plate scan) into one HCS plate #482

Open

Description

@verajanssen-cz

Problem or use case

A Nikon NIS-Elements plate scan writes one .nd2 file per well, named like:

Plate000_WellA01_ChannelBrightfield,GFP-531_Seq0000.nd2
Plate000_WellA02_ChannelBrightfield,GFP-531_Seq0001.nd2
...

Our datasets include single-plane scans (e.g. 282 files on a 384-well plate) and time-lapse z-stacks (48 files, T=43 × Z=49 × C=2 × 2704 × 4263, several TB total).

iohub convert handles one ND2 file at a time: stage positions (P) become FOVs, and with no plate metadata (ND2Dataset sets stage_positions = []) they go into a single row (0/0/0, 0/1/0, ...). With per-well files this means:

Proposed improvement

  1. Folder input. read_images(<dir>) / iohub convert -i <dir> detects a directory of .nd2 files and returns a new ND2PlateDataset (a MicroManagerFOVMapping in iohub/nd2.py) with one FOV per (well, stage position).
  2. Well from the file name. Parse Well(?P<row>[A-Z])(?P<col>\d+) (configurable regex, e.g. a CLI option) into hcs_position_labels → A/1/0, and fill stage_positions[i]["Label"] so position names are the well ids. Never use Seq for placement. Error on two files for the same well.
  3. Uneven T. Plate T = maximum over wells. Shorter wells are zero-padded lazily (dask) and the padded frames are recorded in the position attributes. Mismatched C/Z/Y/X/dtype raises an error.
  4. Consistent metadata. Check that channel names, voxel size and the TimeLoop periodMs agree across files (and pick the interval from the loop whose type == "TimeLoop", not the first loop that has a period).
  5. (Optional) Sharding in the converter for v0.5, e.g. (1,1,1,Y,X) chunks in (1,1,Z,Y,X) shards. For a 49 × 2704 × 4263 uint16 stack the default ZYX chunk is ~1.1 GB and gets Z-halved by the 500 MB cap, which is awkward to read either plane-wise or volume-wise.
  6. (Later) For multi-TB runs on a cluster: resumable conversion (record finished slabs so convert(slabs=...) can skip them after a preempted job) and ND2Dataset.frame_locations() so ND2 can use the byte-range source instead of the lock-serialized dask source.

Alternatives

We have a standalone pipeline that does this today: it builds the plate with iohub.ngff.utils.create_empty_plate (so the output works with biahub) and writes one (well, timepoint) at a time. Parts that could be ported:

  • file discovery: parse Well<Row><Col> (zero-padded ids like A01 ↔ HCS paths A/1), reject duplicate wells, infer the plate format (96/384) from the well ids;
  • metadata: expected frames (len(loop_indices)) vs written frames (attributes.sequenceCount), TimeLoop interval, channel names, voxel size;
  • reading: stream one timepoint frame by frame (loop_indices + read_frame), never loading a whole file (up to ~100 GB), with the missing-frame handling from ND2 conversion writes garbage for frames past sequenceCount (acquisitions stopped early) #481;
  • writing: T = max over wells, zero-fill provenance in the zattrs, per-(well, t) marker files for resume (read with a directory listing, because per-file exists() misses files other nodes just wrote on NFS);
  • tests that monkeypatch the nd2 readers to simulate missing frames and uneven T.

Happy to contribute these as a PR on a feat/nd2-plate-per-well branch if this direction sounds right.

Additional context

Related: #481 (frames past sequenceCount), #434 / #436 (single-file ND2 reader).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: cliCommand-line interfacearea: readersFormat readers, converters, data ingest

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions