Repository navigation
fix: accept uint8 dtype aliases in Zarr v2 - #4381
Merged
Merged
Conversation
d-v-b
approved these changes
Sep 20, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4381 +/- ##
=======================================
Coverage 94.22% 94.22%
=======================================
Files 92 92
Lines 12942 12942
=======================================
Hits 12195 12195
Misses 747 747
🚀 New features to boost your workflow:
|
This was referenced Sep 23, 2026
Closed
d-v-b
added a commit
that referenced
this pull request
Sep 23, 2026
…names (#4385) * fix(dtype): accept any byte order for byte-order-irrelevant v2 dtype names Zarr V2 data type names are NumPy typestrs, whose first character is the byte order ("<", ">", or "|" for not relevant). For single-byte booleans and integers and for raw bytes the byte order is not relevant, so "<i1", ">b1", "<V4" etc. name the same data type as the "|" spelling. Other implementations write these (netCDF-C, Zarr.jl, jzarr), and zarr-python has rejected them since 3.1. The data type registry now retries a v2 name that fails to match with its canonical "|" spelling. The name as written is tried first, so a data type that declares a non-canonical spelling keeps it. This replaces the UInt8-only aliases from #4381, keeping the normalization invisible to the data type classes, and covers structured fields. Fixed-length bytes ("S") are left out: NCZarr gives ">S1" a meaning of its own, and a dedicated data type for it will land first. Tests: the invalid_json_v2/_v3 fixtures of BaseTestZDType are now consumed, and their v2 entries have the v2 data type shape so they fail for their contents. This found two bugs, fixed here: datetime64/timedelta64 accepted a v2 object codec, and a structured data type with an unknown field data type raised ValueError instead of DataTypeValidationError. Assisted-by: ClaudeCode:claude-opus-5-5 * docs: add changelog fragment Assisted-by: ClaudeCode:claude-opus-5-5 * refactor(dtype): enumerate v2 byte order aliases; add check_dtype_spec_no_object_codec_v2 Review follow-ups: - The byte order aliases are an explicit table of the single-byte data types' non-canonical spellings, plus an explicit list of the fixed-length bytes kinds, in place of a regular expression. - check_dtype_spec_no_object_codec_v2 (exported from zarr.dtype) checks a Zarr V2 data type without an object codec. Every data type that is not stored as a NumPy object uses it, so omitting the object codec check (as datetime64/timedelta64 did) is no longer possible by construction. The custom data type example uses it too. Assisted-by: ClaudeCode:claude-opus-5-5 * refactor(dtype): return the v2 spellings to match instead of an optional alias _v2_spellings returns the spellings of a v2 data type JSON in the order they are matched (as written, then canonical), so match_json has no None to check and the precedence rule lives in one place. Assisted-by: ClaudeCode:claude-opus-5-5 * refactor(dtype): make _v2_canonical_name total Every v2 data type name has a canonical spelling: the "|" spelling for a byte order that is not relevant, and the name as written otherwise. The function returns it instead of None, and _v2_spellings adds the canonical spelling when it differs from the name as written. Assisted-by: ClaudeCode:claude-opus-5-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses #3622.
A single-byte element has no internal byte ordering, so
<u1,>u1and|u1can describe the same uint8 data. Currently only|u1is registeredas a UInt8 v2 name: the stored bytes are valid, but metadata parsing rejects
the two aliases and prevents the array from opening.
This accepts both aliases while keeping
|u1as the serialized name.For reviewers
The implementation extends only the accepted UInt8 v2 names.
<u1and>u1are accepted on input, while serialization remains
|u1. Tests cover dtypeand registry parsing, reading local arrays containing
[0, 128, 255],canonical serialization, unchanged on-disk metadata after read-only access,
and rejection of invalid names and object codecs. Other dtypes and v3 are unchanged.
Local validation run by Codex on Windows with Python 3.12.2:
tests/test_store/test_local.py, reproduced on main in the same environment.mypy --platform linuxpasses for 196 files; this is type analysis, not a Linux runtime test.Codex implemented the code, tests and changelog, ran the local checks, and
translated and polished my explanation, and prepared the validation notes. A separate Codex agent performed a static review.
I have reviewed and understand the changes.
Author attestation
TODO
docs/user-guide/*.md— bug fix; changelog addedchanges/