Repository navigation
writing to a sharded array is 10x slower #3560
Description
Zarr version
3.1.3
Numcodecs version
n/a
Python Version
3.12
Operating System
Linux
Installation
uv
Description
The attached script shows that writing multiple chunks to an uncompressed sharded array is about 10x slower than writing to a non-sharded array. You can reproduce this bechmark with this command:
uv run benchmark_zarr_sharding.py --num-iterations 2 --shape 1000 100000 --chunks 1 100000 --shard-chunks 100 1000000 ^C
Some profiling with cProfile reveals that expenses is dominated by repeated calls to Buffer._add__ from _ShardWriter. Buffer.__add__ concatenates the arrays manually, which will results in many repeated allocations and copies.
np.concatenate((np.asanyarray(self._data), np.asanyarray(other_array)))
It would probably be much faster to coalesce these concats into a single operation. A similar pattern is used for gpu buffers, so it seems a new api is required to support this use case.
Activity
nbren12 commentedon Oct 30, 2025
cc @akshaysubr
[-]writing to sharded array is 10x slower[/-][+]writing to a sharded array is 10x slower[/+]nbren12 commentedon Oct 30, 2025
It might be worth looking into how pyarrow handles this concern: https://arrow.apache.org/docs/3.0/python/generated/pyarrow.cuda.BufferWriter.html. I think that project is quite focused on these kinds of performance issues and avoiding unnecessary copies.
optimize shard writing
d-v-b commentedon Oct 30, 2025
@nbren12 I don't see your benchmark script, could you share it? And maybe we add something that hits the same notes in #3562
nbren12 commentedon Oct 30, 2025
Here is the script. benchmark_zarr_sharding.py
d-v-b commentedon Oct 30, 2025
ok looks like the main thing to add to #3562 would be a partial read / write workload
d-v-b commentedon Oct 31, 2025
(deleted a comment that should have been put in #3561)
optimize shard writing (#3561)