Support ZEP 8 URL Syntax

@jbms

Summary

ZEP 8 (zarr-developers/zeps#48) proposes a standardized URL syntax for specifying Zarr nodes across multiple implementations. Implementing this in zarr-python would enable more seamless interoperability between different Zarr tools, making it easier to share dataset locations across different backends.

Motivation

A standardized URL syntax ensures consistent dataset access across different implementations.
The proposed syntax allows for nested storage mechanisms, including ZIP files and hierarchical Zarr groups.
Improved integration with libraries like xarray, which could leverage this for seamless dataset loading.

Proposed Implementation

I would encourage us to handle this in two phases.

Phase 1

Implement handling of ZEP8 URLs for stores implemented in Zarr-Python. Practically, this requires parsing URLs into Store, Path, and ZarrFormat components.

Phase 2

Implement an entrypoint mechanism for 3rd party stores (e.g. Icechunk). This would allow external Store developers to provide a function to generate a Store given a URL.

Example Usage in xarray

With ZEP 8 URL syntax, an xarray.Dataset could be opened as follows:

import xarray as xr

dataset = xr.open_dataset(
    "gs://my-bucket/my-archive.zip|zip:inner-dir|zarr3:subgroup",
    engine="zarr"
)

This URL specifies:

A ZipStore (my-archive.zip) stored in Google Cloud Storage (gs://my-bucket/)
A nested hierarchy inside the Zarr store (subgroup)

Another example:

dataset = xr.open_dataset(
    "https://foobar.com/my-archive|icechunk|zarr3",
    engine="zarr"
)

This URL specifies:

An Icechunk Store available over https

Additional Considerations

Backward compatibility with existing fsspec URLs.

cc @jbms

xref: earth-mover/icechunk#576, zarr-developers/zeps#48

this is a great idea!

Nice idea! Could this be used to specify whether to use fsspec, obstore, or some other Store implementations (e.g., a Store that's coupled to a caching store)?

@ianhi

Hi all 👋 — I'd like to take another run at closing this issue, and I want to share the plan before PRs start landing.

Some history

My first attempt (#3369, continued by @ianhi in #3567) stalled largely under its own weight: a single ~4.8k-line PR, a fragile scheme-detection heuristic, storage_options that didn't thread through the pipeline path, and zarr2:/zarr3: segments that were parsed and then silently dropped. Meanwhile the spec itself matured and moved out of the ZEP process: @jbms now maintains it as an implementation-independent standard at https://github.com/jbms/url-pipeline, with a formal ABNF grammar and per-scheme specs. The new implementation will target that spec (including a few small schema changes), and I'll use the spec's name for the feature: URL pipelines.

Why this is much easier now than a year ago

Several improvements have landed in zarr-python over the past year do most of the infrastructural work the old PR had to hand-roll:

make_store() — store creation was refactored into a single central dispatch point (zarr.storage._common.make_store), so the pipeline hook is one small branch in one function instead of special cases scattered through make_store_path.
parse_store_url() — string URLs are now parsed once, up front, with Windows drive-letter handling already solved. The old PR re-solved this twice with regexes.
Registry.lazy_load(use_entrypoint_name=True) — the entry-point registry pattern used by chunk-key encodings is basically the exact same thing a URL-adapter registry needs; the old PR's custom registry subclass can now be done in ~10 lines of code.
First-class memory:// handling (ManagedMemoryStore) and the cleaned-up fsspec path semantics mean the builtin schemes need no adapters at all — non-piped URLs take exactly the same path they take today.

The result: the review-critical core is roughly 350 source lines, and routing is dead simple — a URL goes through the pipeline machinery only if it contains a | (or its scheme has a registered adapter). No scheme denylists, no fsspec special-casing, and zero downstream-specific code in zarr core.

The plan: 5 PRs, opened one at a time

I'll open each PR only after the previous one merges, so review load stays small and each step is independently useful. I've had Claude estimate the approximate size of each of these PRs just to make sure they reviewable in a single sitting.

#	PR	Contents	Approx. size
1	`ZipStore` file-like support	`ZipStore` accepts an open binary file object (enables zip-over-anything later; useful on its own)	~30 lines src
2	URL pipeline core	parser, single-method `URLPipelineAdapter` ABC, `zarr.url_adapters` entry-point group, `make_store`/`make_store_path` hooks	~350 lines src
3	`zip:` adapter	local, remote (lazy range-reads via fsspec), and nested adapters; read-only	~60 lines src
4	`zarr:`/`zarr2:`/`zarr3:` format segments	URL-selected format propagates to open/create; conflicts with an explicit `zarr_format` raise	~70 lines src
5	Docs	user-guide section, extension guide for adapter authors, API reference	—

What works at the end of PR 4:

import xarray as xr
import zarr

zarr.open("s3://bucket/data.zip|zarr3:subgroup")
xr.open_zarr("s3://bucket/repo|icechunk://tag.v1/subgroup")   # via the icechunk plugin

Downstream validation

Phase 2 of this issue (the plugin mechanism) is only proven once we have a real third-party store using it, so alongside PR 2 I'll open a companion PR to Icechunk (adapter registered via zarr.url_adapters). I have working end-to-end implementations of this already — including the xarray examples above running against real repos — which is how the adapter contract in PR 2 was validated before proposing it here. The key contract points that came out of that exercise: adapters receive the preceding pipeline both parsed and as an exact string (so a native backend like Icechunk can consume s3://… with its own I/O stack), and they return a store plus a residual path, which is what keeps icechunk-specific logic out of zarr core this time.

A note on process

I'm using AI tooling for part of the implementation work. To be clear about what that means for review: I personally review, test, and sign off on every change before requesting review. The design itself (adapter contract, registry shape, PR split) was worked out against the failure modes of #3369.

I'd love to see this move forward, I think a common cross-implementation url syntax for nested storage formats will be very valuable to the data science community.

A few comments:

I would very much like to avoid divergence between implementations --- ideally all implementations can exactly match the spec. But if you think there is a problem with the spec, it can certainly still be changed quite easily at this point. Note that tensorstore and neuroglancer have some small differences from the spec but I plan to fix them to match the spec, but was waiting for more feedback first.
I understand the importance of being able to use the rust backends with icechunk but I'm concerned that it may lead to subtle differences in behavior that may be confusing to users and undermine the intended composability of the spec. For example will icechunk's s3 store handle credentials in the same way as zarr python? Is there a way to mitigate that problem? More generally, exposing that capability to plugins may lead to all sorts of strange behavior inconsistent with the composition implied by the syntax.

I would very much like to avoid divergence between implementations --- ideally all implementations can exactly match the spec. But if you think there is a problem with the spec, it can certainly still be changed quite easily at this point. Note that tensorstore and neuroglancer have some small differences from the spec but I plan to fix them to match the spec, but was waiting for more feedback first.

we may need a conformance test suite.

I would very much like to avoid divergence between implementations --- ideally all implementations can exactly match the spec. But if you think there is a problem with the spec, it can certainly still be changed quite easily at this point. Note that tensorstore and neuroglancer have some small differences from the spec but I plan to fix them to match the spec, but was waiting for more feedback first.

we may need a conformance test suite.

Do you have some ideas for how to do that? The spec has test code that ensures the examples fully cover the grammar, but that just covers allowed syntax, includes no negative examples, and doesn't address semantics.

Possibly there could be a test suite of operations against a generic python interface that would be easy to support for any implementation.

I would very much like to avoid divergence between implementations --- ideally all implementations can exactly match the spec. But if you think there is a problem with the spec, it can certainly still be changed quite easily at this point. Note that tensorstore and neuroglancer have some small differences from the spec but I plan to fix them to match the spec, but was waiting for more feedback first.

we may need a conformance test suite.

Do you have some ideas for how to do that? The spec has test code that ensures the examples fully cover the grammar, but that just covers allowed syntax, includes no negative examples, and doesn't address semantics.

Possibly there could be a test suite of operations against a generic python interface that would be easy to support for any implementation.

we could have an oracle create a storage state parametrized over a root scheme and some number of adapter schemes. for example, some known array is written to zarr in zip on s3. The oracle supplies the zep 8 url pointing to this resource to the program under test, and that program attempts to read the known value out. I think testing the memory root scheme will require the oracle and test code to operate in the same process.

Navigation Menu

Repository navigation

Support ZEP 8 URL Syntax #2943

Summary

Motivation

Proposed Implementation

Phase 1

Phase 2

Example Usage in xarray

Additional Considerations

Some history

Why this is much easier now than a year ago

The plan: 5 PRs, opened one at a time

Downstream validation

A note on process

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support ZEP 8 URL Syntax #2943

Description

Summary

Motivation

Proposed Implementation

Phase 1

Phase 2

Example Usage in xarray

Additional Considerations

Activity

d-v-b commented on Apr 1, 2025

maxrjones commented on Aug 4, 2025

jhamman commented on Jul 26, 2026

Some history

Why this is much easier now than a year ago

The plan: 5 PRs, opened one at a time

Downstream validation

A note on process

jbms commented on Jul 26, 2026

d-v-b commented on Jul 26, 2026

jbms commented on Jul 26, 2026

d-v-b commented on Jul 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions