Repository navigation
Zarr v2 -> Zarr v3 migration across computational imaging team's frameworks #199
Description
We have decided to upgrade our image processing and analysis pipelines from zarr v2 (ome-zarr-0.4) to zarr v3 (ome-zarr-0.5) format.
The key reason for the switch is that the support for sharding and small chunks in zarr v3 datasets will enable real-time inference for smart microscopy, efficient model training through parallel/distributed data loading, and efficient visualization.
This issue is focused on identifying any disruption in processing that computational imaging team and collaborators may encounter due to the change in format, across iohub, biahub, waveorder, viscy, and shrimpy. We are centralizing the discussion in biahub repo.
iohub has been updated to support reading and writing with both versions: When we write the zarrs during the processing steps we just need to set the version flag in iohub.ngff.open_ome_zarr to v0.5.
We need to make a few key decisions:
-
Should the refactored CLI across the repos "silently" migrate from zarr v2 to zarr v3, or propagate the format? For silent migration, the CLI can read zarr v2/v3 data, and write only zarr v3 data. This is currently preferred to migrate to a better format and reduce data duplication. If the conversion can break existing analysis workflows and the fix is not straightforward, we can propagate the format.
-
Which CLI need to be updated in what order so that silent migration doesn't break existing analysis workflows? We need key users's inputs to decide if migration to zarr v3 will break their data analysis pipelines.
@Soorya19Pradeep, @tayllatheodoro, @gav-sturm can you please add a sub-issue to clarify which CLI are you currently dependent on that could break with migration to zarr v3? We can use the sub-issue to resolve the problems.
Current zarr v2 --> v3 transition plan
- Set version-specific default chunk and shard sizes in
create_empty_plateiohub#401 - Replace create_empty_hcs_zarr with iohub's create_empty_plate #215
- Preserve zarr version during reconstruction #240
- Replace create_empty_hcs_zarr with iohub's create_empty_plate waveorder#540
- [BUG] waveorder writes ome-zarr data as XY images rather than XYZ chunks waveorder#550
- Port to iohub 0.3.2 ImageArray API VisCy#407
- Convert to
version=0.5by default iohub#402
Test plan
- mantis v2 reconstruction should proceed without errors. Intermediate reconstructions should be written in zarr v3 with proper chunking, sharding, and compression
- Dragonfly reconstruction should proceed without error, for now writing data in zarr v2 with proper chunking
- Dragonfly datasets can be converted and reconstructed in zarr v3 (WIP in
/hpc/projects/intracellular_dashboard/organelle_box/2026_04_04_A549_weak_organelles_DENV_ZIKV_time_course/test_zarr_v3/, VS next)
Activity
ieivanov commentedon Feb 17, 2026
I think the change to
v0.5is a no-brainer, when it comes to implementation, the devil may be in the details. We won't know until we try, I think we can do these tests in parallel and surface any issues that we can resolve as a group. My preference would be for a "silent" transition - our pipelines would read zarr v2/v3 and write zarr v3 only. We can create fallback cases (e.g. write zarr v2 if input was v2, if needed).I checked and it looks like
biahub,waveorder, andviscyalready depend oniohub>0.3.shrimpydoes not depend oniohub- it usesacquire-zarrto write data. Note thatshrimpywrites zarr v3 by default, so new data we generate will be coming in this format (I can check if there is a way to write zarr v2, but I think that would be a step in the wrong direction). I've confirmed zarr stores written withshrimpycan be read byiohub: czbiohub-sf/shrimPy#173iohubv0.3 is still in alpha phase: https://github.com/czbiohub-sf/iohub/milestone/3. @srivarra what is needed to release a stable v0.3? Which of the issues on this milestone are critical, and which can be pushed to a v3.1 milestone? We've been using v0.3 alpha versions since last September, and I think we've already resolved a lot of problem - I don't have any complains with the current codebase.It's great that @talonchandler already worked on transitioning
waveorderto zarr v3 and surfaced a get gotchas: Biohub/waveorder#519@srivarra @tayllatheodoro can you prototype the transition of
biahubto zarr v3 using thedeskewfunction? The input should be a zarr v3 raw/skewed dataset and the output should be a zarr v3 deskewed dataset. You can get the zarr v3 input by either usingbiahub convertto convert an old dataset, say of the argolight target, or I can share with you a new dataset acquired with the shrimpy v2 acquisition engine. Let us know what issues come up.@srivarra @edyoshikun can you do a similar prototype for
viscy? Can you confirm if the dataloaders can work with zarr v3 dataset? Can you test if virtual staining can output zarr v3 datasets? Again, surface any issues that come up.Previous discussion on the transition to zarr v3 have centered around picking the right chunk / shard size such that we can enable easy data loading for visualization and processing, reduce the total number of files on the hpc, and preserve the ability to do parallel processing across timepoints and channels. Currently, in zarr v2, or chunk size is
(1, 1, Z, Y, X), limited to 500 MB if needed (i.e. we may usechunk_z < Z). I think our consensus for chunks and shards in zarr v3 is:srivarra commentedon Feb 17, 2026
Looking through the issues for this milestone, I think everything can be pushed back to
v0.3.1. The only thing I'd like to resolve is: Biohub/iohub#374 (do we want to add the shard size here for the conversion step?) the issue of chunking across a dimension where the size is a prime number.Added a bit of
iohubrelated discussion over here: Biohub/iohub#375edyoshikun commentedon Feb 17, 2026
Biohub/VisCy#372
@alxndrkalinin @srivarra I'm starting a PR here. The easy first case was just to parametrize our fixtures for building an HCS plate, but I dont think this covers all the utils in the repo. I am 80% confident this covers all our usecases. I did a quick search on places where we call zarr directly and it only happened in the hcsdataloader when we
prepare_data()for cachine, so I am confident we are good to go. We are leveraging iohub as our reader and we are letting it handle that with Tensostore in the ImageArray.Soorya19Pradeep commentedon Feb 18, 2026
Thank you for the PR @mattersoflight .
I have been trying out biahub CLIs on zarr v3 inputs, including biahub register from #198.
I understand that biahub CLIs dont work with zarr v3 yet. I was under the impression that biahub concatenate works with zarr v3 from the main branch. I didn't get that working either.
talonchandler commentedon Feb 18, 2026
I'm in favor of a "silent" transition. No concerns from the
waveorderside.gav-sturm commentedon Feb 18, 2026
All good to move ahead with silent transition from the OPS pipeline. We currently use biahub for channel registration and this exists as a separate env disconnected from all other pipeline steps. Also, we use Iohub Version: 0.2.1 for the zarr-v2 portions of the pipeline (registration, reconstruction etc ) and Iohub Version: 0.3.0a2 for zarr-v3 portions (all downstream tasks like feature extraction, model inference)... we eventually plan for the entire pipeline to generate zarr-v3 (next couple months)
ieivanov commentedon May 28, 2026
FYI, we are in the final stages of making this transition.
As an overview, mantis datasets are collected as zarr v3 and data reconstruction happens in zarr v3. DF and HF are still converted to zarr v2 and data recon happens in zarr v2. We have the option to convert these data into zarr v3 with
iohub convert -v 0.5and I just double checked that the DF reconstruction pipeline runs well in zarr v3 up to virtual staining. I'll update the biahub viscy dependency next (we had to roll that back due to iohub compatibility issues which are now resolved, see #236), and make sure we ca get thru that steps as well.As a final step in our transition from zarr v2 to zarr v3 we'll make
iohub convertdefault to zarr v3 as in Biohub/iohub#402. Initially we'll have an option to go back to zarr v2 withiohub convert -v 0.4if needed - until v2 is removed from iohub altogether.refactor(qc): depend on imaging-qc-pipeline directly, and call it bare
fix(qc): follow imaging-qc's flat plan and single merge (Phase 20)
feat(qc): QC integrated into mantis-v2, with imaging-qc as a direct d…