fix(dtype): resolve struct fields with the registry that resolves the struct

[d-v-b]

When we read a JSON value from the data_type field in metadata, we resolve the JSON value to a ZDType class, construct the class, or error. We model this operation as f(JSON value) -> ZDtype, but it's more like f(JSON value, registry of data type classes) -> ZDType, because the operation depends on both the actual JSON value AND the set of ZDType classes available.

This PR makes that explicit. Instead of magically conjuring (read: using the global config) a data type registry inside parsing routines, we pass the registry to the parsing routine explicitly via a Context object. We also pass the relevant zarr format and the current location of the parser in the data type JSON, which is relevant for nested dtypes like struct and ensures that an error can point to where exactly the problematic dtype was.

claude wrote this, and I was absolutely delighted when the need for this came up because I plan to extend this pattern to codecs, which will be a great improvement.

@ilan-gold @maxrjones this doesn't add public API but I would like to in later PRs: I think adding Context to our group / array constructors is a great way to keep our internal logic isolated from the mutable global config. The config would be used to create the default context, but consumers don't see the global config. they just see a piece of data.

🤖 AI text below 🤖

Follows #4385 (merged): its review asked for dtype resolution to take a resolver, to support recursive dtypes. Independent of #4386. It also renames the changes/351.bugfix.md fragment #4385 merged under its fork pull request number to 4385.bugfix.md.

Problem

JSON → data type resolution always ended in the module-global data_type_registry. A DataTypeRegistry other than the default only took effect at the top level. A structured data type resolved its fields with the global registry no matter which registry resolved the struct. get_data_type_from_json, parse_dtype and the metadata parsers had no way to take a registry at all.

Change

Two things are kept apart, as in zarr-metadata (Context passed as context=, with the location threaded separately):

• zarr.registry.Context: which extensions a read may resolve, grouped by kind. It's a value, the same for a whole read, and chosen by the caller. For now it holds data_types. Codecs, chunk key encodings and chunk grids can be added later as fields with defaults, without changing any signature. Context.default() refers to the default registries themselves, so it sees later registrations.
• Resolver (zarr.core.context): a context positioned in a document. It holds the context, the Zarr format, and the location in the document. A composite resolves each contained extension with resolver.at(*keys).resolve_data_type(json). A codec read that meets a data type, like cast_value will once codecs join, hands the same context to a data type read through the same object. The Context / Resolver pair mirrors referencing.Registry / referencing.Resolver in python-jsonschema.
• Locations are JSON locations, reported as JSON Pointers. This stays unambiguous across kinds, e.g. No Zarr data type found that matches 'nope' at /configuration/fields/1/data_type. A Zarr V2 data type is located by its name as written in the document, so metadata parsing can later prefix /dtype.
• ZDType._from_json_resolved(data, *, resolver): the one way a registry creates a data type. Its default ignores the resolver and calls from_json(data, zarr_format=resolver.zarr_format), so every existing data type, including ones that override from_json, is created exactly as before. A data type that contains other data types overrides it to resolve them with the resolver.
• Structured / Struct override _from_json_resolved, and their from_json takes context=. A field whose data type matches nothing raises NestedDataTypeValidationError (in zarr.errors, a DataTypeValidationError). The containing data type decides that a failure is nested, and the registry re-raises it rather than trying other data types. This restores a specific message for a bad struct field, which fix(dtype): accept any byte order for byte-order-irrelevant v2 dtype names #4385 had reduced to a mismatch of the whole struct.
• context= keyword (default: Context.default()) on get_data_type_from_json, get_data_type_from_native_dtype, parse_dtype and parse_data_type.
• Entry-point data types now load (separate commit). zarr.registry collected them into DataTypeRegistry._lazy_load_list, but nothing in src/ called _lazy_load(). get, match_dtype and match_json now load them first, as the other registries do.

Precedents

Newer parsing libraries agree on passing the decoding context down explicitly, rather than resolving nested types through a global registry:

• pydantic v2: every custom type's __get_pydantic_core_schema__(source, handler) receives handler, and container types use it for their inner types.
• Rust serde: every type's Deserialize::deserialize(deserializer) receives the deserializer.
• cattrs: hooks are registered on a Converter instance, and hook factories can receive the converter.
• python-jsonschema: it deprecated the implicit RefResolver in favour of a referencing.Registry passed as registry=, with a Resolver positioned in the document. That's the model for Context / Resolver here.
• zarr-metadata: resolve(data, context), with the context passed to every entity's coerce.

Where they differ is whether types that contain nothing receive the context too. Protocols designed from scratch (pydantic, serde, zarr-metadata) give it to every type, and leaves ignore it. Protocols retrofitted onto an existing API split it off: serde's separate DeserializeSeed trait, cattrs checking how many arguments a factory takes, aeson's FromJSON1. This PR takes the from-scratch shape while staying compatible: the hook is on every type, and its default delegates to the existing from_json.

Why not a keyword on ZDType.from_json

That was the first version. It broke on our own entry-point test data type (tests/package_with_entrypoint), which overrides from_json(cls, data, *, zarr_format). With the registry passing an extra keyword to every class, a single registered data type like that raises TypeError and takes down all JSON resolution. A separate hook on every data type, whose default delegates to from_json, avoids that, and the registry needs no type check to decide which data types get the resolver. test_resolve_data_type_overriding_from_json pins this.

Why the resolver carries a location

The location only serves diagnostics: it changes nothing about which data type is resolved. It lets a failure deep in DataTypeRegistry._match_json, which does not know it is resolving a struct field, say where in the document it happened, e.g. No Zarr data type found that matches 'nope' at /configuration/fields/1/data_type.

Tracking the location is standard; libraries get it in one of two ways:

• Passed down with each call: aeson's Parser carries a JSONPath that composites extend with <?>, zod's parse context carries a path, and zarr-metadata passes loc alongside the context. The location is complete when the error is raised.
• Built up as errors propagate: pydantic-core adds outer location segments to errors as they bubble up (hence ValidationError's loc tuples), and python-jsonschema's descend() prepends each child's path segment before re-raising.

This PR passes it down. Building it up would mean each composite catches, prepends its segment, and re-raises. The exception would then either carry the location as structured data or have its message text rewritten at every level. Neither is how zarr-python errors should work; structured problem locations belong to zarr-metadata's problem reports. Passing it down keeps the message complete at the raise, and a composite's only job is resolver.at(...). It also matches zarr-metadata's loc.

Costs:

• The location is why Resolver is its own type; without it, a resolver would be a context plus the Zarr format.
• A Zarr V2 data type is located by its name as written in the document, not by the {"name": ..., "object_codec_id": ...} form zarr-python reads it in, so metadata parsing can prefix /dtype.
• Data types in consolidated metadata are not yet prefixed with /consolidated_metadata/metadata/<name> (see feat: open arrays and groups with a Context #4388).

Not in this PR

• Plumbing context= through metadata parsing, the array and group classes, and open_array / open_group / open / open_consolidated. That's the next PR, which builds on this one.
• Found while testing: a Zarr V2 structured data type nested in another doesn't round trip. to_json writes [['outer', [['x', '|i1']]]], which the v2 structured shape check then rejects. This predates this PR.

Tests

• One test covering both zarr formats and every route into resolution (registry.match_json, get_data_type_from_json(context=), parse_dtype(context=), Struct.from_json(context=)): a struct whose field data type exists only in a local registry resolves that field from the local registry.
• Error cases:
  • without a context, that field isn't found in the default registry (NestedDataTypeValidationError at /configuration/fields/0/data_type);
  • a local registry lacking a field's data type rejects the struct, even though the default registry has that data type;
  • parse_dtype with a native dtype only searches the given registry;
  • a missing field data type is reported with its JSON Pointer: /1/1 for v2, and two levels deep in v3.
• A data type that overrides from_json works as a struct field when resolved through a registry.
• test_entrypoint_dtype no longer calls _lazy_load() by hand, so it now checks automatic loading. The fixture that clears pending entry points also clears the data type registry's list.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.33%. Comparing base (9abad7f) to head (55a8175).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4387      +/-   ##
==========================================
+ Coverage   94.31%   94.33%   +0.02%     
==========================================
  Files          92       93       +1     
  Lines       13069    13127      +58     
==========================================
+ Hits        12326    12384      +58     
  Misses        743      743              

Files with missing lines	Coverage Δ
src/zarr/core/context.py	100.00% <100.00%> (ø)
src/zarr/core/dtype/npy/structured.py	94.35% <100.00%> (+0.60%)	⬆️
src/zarr/core/dtype/registry.py	95.89% <100.00%> (+0.97%)	⬆️
src/zarr/core/dtype/wrapper.py	97.91% <100.00%> (+0.13%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.