Skip to content

fix(codecs): read the stored codec name "endian" as the bytes codec - #402

Draft
d-v-b wants to merge 18 commits into
mainfrom
fix/endian-codec-alias
Draft

fix(codecs): read the stored codec name "endian" as the bytes codec#402
d-v-b wants to merge 18 commits into
mainfrom
fix/endian-codec-alias

Conversation

@d-v-b

@d-v-b d-v-b commented Oct 10, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

A stored codec named "endian", the name an early draft of the Zarr format 3 specification gave the bytes codec, is now read as the bytes codec, with a warning saying how to store valid metadata. Previously opening such an array raised ValueError: Expected 'bytes'. Got endian instead., even though "endian" was registered as a BytesCodec alias.

zarr-python never wrote "endian": the v3 code it bootstrapped from already wrote "bytes". zarrita did, though. The agent installed zarrita 0.1.0a15 from PyPI and wrote an array with it, which stored {"name": "endian", "configuration": {"endian": "little"}}. That array reads as [0 1 2 3] on this branch and raises on main. Following the rule that metadata a known writer produced is read the way the spec intends, the name is read rather than rejected.

The reading is a repair in zarr.core.metadata.repair, the module that holds all lenient reads of stored metadata. It renames "endian" to "bytes" in codecs, and in the codecs and index_codecs of sharding codecs, nested or not, before the document is validated. No chunk moves, since the bytes codec is read with its stored configuration. The repair warns with RESAVE_HINT: other implementations do not read the name, and re-saving the metadata stores "bytes".

Decisions for review:

  • The repair warns, not silently reads. zarr-python never read this name before, so the rule in the repair module that a reading giving what zarr previously read stays silent does not apply.
  • The "endian" registrations in zarr/codecs/__init__.py and zarr/core/config.py are left in place. Stored metadata no longer reaches them, so they are dead code. Removing a registered name may fall under the deprecation policy in docs/contributing.md, so that is left to a separate change.

🤖 Generated with Claude Code

ianhi and others added 18 commits October 8, 2026 14:04
@ianhi @d-v-b
Fix/from array overwrite source (zarr-developers#4405)
b05a91f 
* fix(array): refuse from_array overwrites that overlap the source

With overwrite=True, from_array deletes the destination before streaming
data from the source. When the source array lives at, above, or below the
destination path in the same store, that delete removes the source's
chunks and the result is all fill values. Raise a ValueError before
anything is deleted instead.

* test(array): check metadata-only overwrite of the source keeps its metadata

* special case weird eq behavior

* docs: number the changelog entry for zarr-developers#4405

---------

Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@d-v-b @claude
fix(ci): run xarray's backend tests from their new package (zarr-deve…
bb17a80 
…lopers#4491)

* fix(ci): run xarray's backend tests from their new package

pydata/xarray#11633 split xarray/tests/test_backends.py and the other
test_backends*.py modules into the xarray/tests/backends package, so the
downstream job failed to collect any tests. Run the package instead.

Assisted-by: ClaudeCode:claude-fable-5-1

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs: add changelog fragment for zarr-developers#4491

Assisted-by: ClaudeCode:claude-fable-5-1

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
@d-v-b @claude
docs: 3.4.1 release notes (zarr-developers#4489)
246fa20 
Built with towncrier from main at bb17a80. The entry for zarr-developers#4466 links to the
pull request instead of issue zarr-developers#4464, which its fragment was named after; the
longer entries are rewritten in plain technical English; and continuation
blocks are indented four spaces so MkDocs keeps them inside their list items.

Assisted-by: ClaudeCode:claude-fable-5-1

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
@safaksm
fix: zero default scalar for bytes fields in structured dtypes (zarr-…
3d1a839 
…developers#4495)

* fix: zero default scalar for bytes fields in structured dtypes

* docs: name changelog entry after zarr-developers#4495
@d-v-b
test(dtype): cover unicode fields in the structured default scalar te…
19e84dc 
…st (zarr-developers#4496)

zarr-developers#4495 also changed the default of `U` fields in structured dtypes from
"0" to "". Add `U4` to the regression test and name string fields in
the changelog entry.

Assisted-by: ClaudeCode:claude-opus-5-5
@dependabot @d-v-b
chore(deps): bump the python-dependencies group with 8 updates (zarr-…
12d7222 
…developers#4488)

Bumps the python-dependencies group with 8 updates:

| Package | From | To |
| --- | --- | --- |
| [google-crc32c](https://github.com/googleapis/python-crc32c) | `1.8.0` | `1.9.0` |
| [msgspec](https://github.com/msgspec/msgspec) | `0.21.1` | `0.22.0` |
| [coverage](https://github.com/coveragepy/coveragepy) | `7.16.1` | `7.16.2` |
| [hypothesis](https://github.com/HypothesisWorks/hypothesis) | `6.168.1` | `6.168.3` |
| [uv](https://github.com/astral-sh/uv) | `0.12.18` | `0.12.21` |
| [botocore](https://github.com/boto/botocore) | `1.43.101` | `1.43.106` |
| [ruff](https://github.com/astral-sh/ruff) | `0.16.8` | `0.16.9` |
| [astroid](https://github.com/pylint-dev/astroid) | `4.3.1` | `4.3.3` |


Updates `google-crc32c` from 1.8.0 to 1.9.0
- [Release notes](https://github.com/googleapis/python-crc32c/releases)
- [Changelog](https://github.com/googleapis/python-crc32c/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/python-crc32c/commits)

Updates `msgspec` from 0.21.1 to 0.22.0
- [Release notes](https://github.com/msgspec/msgspec/releases)
- [Changelog](https://github.com/msgspec/msgspec/blob/main/docs/changelog.md)
- [Commits](msgspec/msgspec@0.21.1...0.22.0)

Updates `coverage` from 7.16.1 to 7.16.2
- [Release notes](https://github.com/coveragepy/coveragepy/releases)
- [Changelog](https://github.com/coveragepy/coveragepy/blob/main/CHANGES.rst)
- [Commits](coveragepy/coveragepy@7.16.1...7.16.2)

Updates `hypothesis` from 6.168.1 to 6.168.3
- [Release notes](https://github.com/HypothesisWorks/hypothesis/releases)
- [Commits](HypothesisWorks/hypothesis@v6.168.1...v6.168.3)

Updates `uv` from 0.12.18 to 0.12.21
- [Release notes](https://github.com/astral-sh/uv/releases)
- [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md)
- [Commits](astral-sh/uv@0.12.18...0.12.21)

Updates `botocore` from 1.43.101 to 1.43.106
- [Commits](boto/botocore@1.43.101...1.43.106)

Updates `ruff` from 0.16.8 to 0.16.9
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ruff@0.16.8...0.16.9)

Updates `astroid` from 4.3.1 to 4.3.3
- [Release notes](https://github.com/pylint-dev/astroid/releases)
- [Changelog](https://github.com/pylint-dev/astroid/blob/main/ChangeLog)
- [Commits](pylint-dev/astroid@v4.3.1...v4.3.3)

---
updated-dependencies:
- dependency-name: google-crc32c
  dependency-version: 1.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: msgspec
  dependency-version: 0.22.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: coverage
  dependency-version: 7.16.2
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: hypothesis
  dependency-version: 6.168.3
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: uv
  dependency-version: 0.12.21
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: botocore
  dependency-version: 1.43.106
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: ruff
  dependency-version: 0.16.9
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: astroid
  dependency-version: 4.3.3
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@maxrjones @claude @d-v-b
docs: add a deprecation policy to the contributing guide (zarr-develo…
09bb377 
…pers#4051)

* docs: add a deprecation policy to the contributing guide

Closes zarr-developers#2924.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: neutral tone

* Add missing component from NEP23

* docs: align the deprecation policy with the versioning section and the code

Review follow-ups on the deprecation policy:

- Explain who each warning class reaches: DeprecationWarning is hidden
  outside __main__ and test runners, FutureWarning is shown to everyone,
  which a behavior change needs.
- Drop the "release first appeared in" requirement from the message; the
  release type is decided at release time under EffVer.
- Widen the recently-introduced-API exception to match the versioning
  section, and make the experimental promotion re-export explicitly exempt.
- Extend the minimum period to behavior changes, name raised exceptions as
  observable behavior, and note that a removal in a minor release is not
  SemVer (review threads).
- Rename the changelog fragment to the PR number.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* use more sentences

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Davis Vann Bennett <davis.v.bennett@gmail.com>
@dependabot @d-v-b
chore(deps): bump cairosvg from 2.9.0 to 2.9.1 (zarr-developers#4499)
b996f03 
Bumps [cairosvg](https://github.com/Kozea/CairoSVG) from 2.9.0 to 2.9.1.
- [Release notes](https://github.com/Kozea/CairoSVG/releases)
- [Changelog](https://github.com/Kozea/CairoSVG/blob/main/NEWS.rst)
- [Commits](Kozea/CairoSVG@2.9.0...2.9.1)

---
updated-dependencies:
- dependency-name: cairosvg
  dependency-version: 2.9.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@pre-commit-ci @d-v-b
chore: update pre-commit hooks (zarr-developers#4482)
2637250 
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.16.6 → v0.16.10](astral-sh/ruff-pre-commit@v0.16.6...v0.16.10)
- [github.com/DavidAnson/markdownlint-cli2: v0.23.2 → v0.23.3](DavidAnson/markdownlint-cli2@v0.23.2...v0.23.3)
- [github.com/numpy/numpydoc: v1.11.0rc0 → v1.11.0](numpy/numpydoc@v1.11.0rc0...v1.11.0)
- [github.com/zizmorcore/zizmor-pre-commit: v1.30.0 → v1.30.1](zizmorcore/zizmor-pre-commit@v1.30.0...v1.30.1)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@2160039878-cyber @codex
@d-v-b @claude
fix: bound memory used by resize chunk cleanup (zarr-developers#4344)
df4a016 
* fix: bound memory used by resize chunk cleanup

Co-Authored-By: OpenAI Codex <noreply@openai.com>

* fix: delete resize chunks with a worker pool instead of lockstep batches

Batches sized to async.concurrency were awaited one after another, so each
group drained to zero in-flight deletions while waiting on its slowest
request. A fixed pool of workers pulling from the shared lazy coordinate
iterator keeps `concurrency` deletions in flight with the same memory bound.

Also rename the changelog fragment to the PR number.

Assisted-by: ClaudeCode:claude-opus-5-5

* fix: cancel remaining resize deletions after the first failure

With gather, a failed deletion raised out of resize while the other
workers kept deleting in the background, racing any retry. A TaskGroup
cancels them; the first exception is unwrapped from the ExceptionGroup
so resize keeps raising the store's own error.

Assisted-by: ClaudeCode:claude-opus-5-5

* fix(array): move the resize deletion pool into concurrent_foreach and size it to the job

Review follow-ups on the bounded resize cleanup:

- Add `zarr.core.common.concurrent_foreach`, a lazy bounded-concurrency
  sibling of `concurrent_map`: `limit` workers share one iterator, the
  first failure stops further items and cancels in-flight calls, and the
  failure is re-raised as itself. Simultaneous failures are kept as the
  `__cause__` group instead of being dropped, and `BaseExceptionGroup` is
  handled so non-Exception failures are unwrapped too.
- A call that fails before its first await no longer fans out to every
  worker, so a read-only store raises once.
- `_resize` computes the number of chunks to delete in O(ndim), skips the
  pool when it is zero (which subsumes the only-growing guard), and sizes
  the pool to `min(limit, n_delete)`.
- The changelog fragment states the new cancel-on-first-failure semantics.
- The pool-saturation test asserts the bound instead of the exact asyncio
  interleaving; helper semantics are tested in tests/test_common.py.

Assisted-by: ClaudeCode:claude-fable-5-1

---------

Co-authored-by: 2160039878-cyber <285580214+2160039878-cyber@users.noreply.github.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Davis Vann Bennett <davis.v.bennett@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@dependabot @d-v-b
chore(deps): bump multidict from 6.7.1 to 6.9.1 (zarr-developers#4500)
43dfe32 
Bumps [multidict](https://github.com/aio-libs/multidict) from 6.7.1 to 6.9.1.
- [Release notes](https://github.com/aio-libs/multidict/releases)
- [Changelog](https://github.com/aio-libs/multidict/blob/master/CHANGES.rst)
- [Commits](aio-libs/multidict@v6.7.1...v6.9.1)

---
updated-dependencies:
- dependency-name: multidict
  dependency-version: 6.9.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@dependabot @d-v-b
chore(deps): bump multidict (zarr-developers#4498)
6d986a9 
Bumps the uv group with 1 update in the /packages/zarr-http-server directory: [multidict](https://github.com/aio-libs/multidict).


Updates `multidict` from 6.7.1 to 6.9.1
- [Release notes](https://github.com/aio-libs/multidict/releases)
- [Changelog](https://github.com/aio-libs/multidict/blob/master/CHANGES.rst)
- [Commits](aio-libs/multidict@v6.7.1...v6.9.1)

---
updated-dependencies:
- dependency-name: multidict
  dependency-version: 6.9.1
  dependency-type: indirect
  dependency-group: uv
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@dependabot @d-v-b
chore(deps): bump werkzeug from 3.1.8 to 3.1.9 (zarr-developers#4497)
382df0f 
Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.1.8 to 3.1.9.
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](pallets/werkzeug@3.1.8...3.1.9)

---
updated-dependencies:
- dependency-name: werkzeug
  dependency-version: 3.1.9
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@dependabot @d-v-b
chore(deps): bump github-community-projects/issue-metrics (zarr-devel…
ab4f2fe 
…opers#4487)

Bumps the actions group with 1 update in the / directory: [github-community-projects/issue-metrics](https://github.com/github-community-projects/issue-metrics).


Updates `github-community-projects/issue-metrics` from 5.0.2 to 5.0.3
- [Release notes](https://github.com/github-community-projects/issue-metrics/releases)
- [Commits](github-community-projects/issue-metrics@a7dc2fb...6715286)

---
updated-dependencies:
- dependency-name: github-community-projects/issue-metrics
  dependency-version: 5.0.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@barlowa124 @devin-ai-integration @d-v-b
fix: reset the codec-pipeline thread pool after fork (zarr-developers…
6599477 
…#4483)

* fix: reset the codec-pipeline thread pool after fork

A forked child inherits codec_pipeline._pool pointing at threads that
do not exist in the child, plus the parent's idle-semaphore permits.
Submissions up to the permit count are consumed without spawning a
thread, so small FusedCodecPipeline reads hang forever.

Add an after_in_child hook that drops the inherited pool, size, and
lock. The lock is recreated because it may have been held by a thread
that no longer exists in the child.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* docs: add changelog entry for zarr-developers#4483

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: barlowa124 <barlowa124@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@nuglifeleoji Leo Ji
@d-v-b @maxrjones
DOC: fix v3 migration guide store table (zarr-developers#3850)
bd0dc84 
* DOC: remove result="ansi" causing empty code blocks in user guide

Closes zarr-developers#3506.

Made-with: Cursor

* DOC: fix store migration guide: remove broken row, add LRUStoreCache note

Removes truncated table row and adds migration note for LRUStoreCache.
Closes zarr-developers#2699. Closes zarr-developers#2857.

Made-with: Cursor

* docs: point LRUStoreCache migration note at CacheStore

Move `LRUStoreCache` into the renamed-stores table, since it was never
deprecated in 2.x, and point it at the experimental
`zarr.experimental.cache_store.CacheStore`. Replace the nonexistent
`fsspec.implementations.caching` module with `fsspec.implementations.cached`,
and add the changelog fragment.

Assisted-by: ClaudeCode:claude-opus-5-5

---------

Co-authored-by: Leo Ji <nuglifeleoji@gmail.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>
@d-v-b
fix(codecs): read the stored codec name "endian" as the bytes codec
6800e6f 
`"endian"` was registered as a `BytesCodec` alias, but `BytesCodec.from_dict`
accepts only `"bytes"`, so a stored codec of that name could not be read.
zarrita 0.1.0a15, written against an early draft of the v3 spec, stored
it. A repair now reads the name as `"bytes"` at the top level and in
sharding codecs and index codecs, warning how to store valid metadata.

Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b
docs: add changelog fragment for #402
b0eedf1 
Assisted-by: ClaudeCode:claude-opus-5-5

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@d-v-b @ianhi @safaksm @maxrjones @2160039878-cyber @barlowa124 @nuglifeleoji