A sharding codec inside another is never validated, and an inner chunk shape that does not divide corrupts data silently #4437

🤖 AI text below 🤖

A ShardingCodec inside another ShardingCodec's codecs is never validated. Its chunk_shape has to evenly divide the chunk it is handed, which is the outer shard's inner chunk. Nothing checks this, and an array whose inner shard does not divide writes and reads back different data:

import numpy as np
import zarr
from zarr.codecs import BytesCodec, ShardingCodec

store = zarr.storage.MemoryStore()
array = zarr.create(
    store=store,
    shape=(16, 16),
    chunks=(8, 8),
    dtype="float32",
    zarr_format=3,
    # The inner shard is handed (4, 4) chunks, which (3, 4) does not divide.
    codecs=[
        ShardingCodec(
            chunk_shape=(4, 4),
            codecs=[ShardingCodec(chunk_shape=(3, 4), codecs=[BytesCodec()])],
        )
    ],
)
data = np.arange(256, dtype="float32").reshape(16, 16)
array[...] = data
print(np.array_equal(zarr.open_array(store, mode="r")[...], data))  # False

This reproduces with zarr 3.4.0 and on main at 8dc50b8.

The spec. The inner chunk shape "must evenly divide the size of the shard shape" of the shard "to which this sharding codec is applied" (sharding-indexed, L131-L135). An inner sharding codec is applied to the outer shard's inner chunks.

Where.

• ArrayV3Metadata validates each top-level codec, at v3.py L540 and L571.
• ShardingCodec.validate checks the array's chunk grid and does not recurse into its own codecs, so an inner sharding codec's validate is never called.

The root cause is the same as #2050: a codec is validated against the array, not against the chunk it is handed.

Expected. Each codec is validated against the shape and data type it is handed. For a sharding codec, that means recursing into its inner codecs with its inner chunk shape.

This came up while writing the shard checks of zarr-metadata's codec pipeline, which refuses this document (draft: #4443).