Repository navigation
memory cost of deleting "outside" chunks on resize is unbounded #3650
Description
Zarr version
3.1.5
Description
zarr-python/src/zarr/core/array.py
Lines 1856 to 1859 in 1c05c1a
zarr-python/src/zarr/core/chunk_grids.py
Lines 195 to 198 in 1c05c1a
For very large arrays (even if unpopulated), the set in resize (which has delete_outside_chunks=True by default) will take down a machine :/ .
- I propose
delete_outside_chunks=Falseby default. This is a "maintenance op" and has unbounded cost with high latency stores. It should not be called in a "normal" workflow. delete_outside_chunkscan be a LOT smarter. We know new shape and old shape; so we can construct "outside" chunk coords directly from the diff of the shapes. In particular, simply extending the size of a dimension requires no deletion (!!!).
Steps to reproduce
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
# ]
# ///
#
# This script automatically imports the development branch of zarr to check for issues
import zarr
import numpy as np
store = zarr.storage.MemoryStore()
# Create the array with specified shape and chunks
arr = zarr.create(
shape=(143, 7, 668160, 1336320),
chunks=(1, 1, 928, 928),
dtype=np.float32,
store=store,
overwrite=True
)
arr.resize((144, *arr.shape[1:]))Additional output
No response
Activity
[-]memory cost of deleting old chunks on resize is unbounded[/-][+]memory cost of deleting "outside" chunks on resize is unbounded[/+]d-v-b commentedon Jan 13, 2026
+1 to deferring cleanup activities to a separate routine. I wonder if a function like
prune_chunks(array_metadata, store, path)would fit the bill here. This would remove all chunks that are a) consistent with the chunk grid and b) outside the shape of the array.Skip chunk coordinate enumeration in resize when array is only growing (
jakenotjay commentedon Feb 11, 2026
Exactly what Deepak described took down one of our processes yesterday when updating our icechunk stores to have a single new timestep (it's a very large array), there are no chunks to remove, so this code block should ideally not even be touched.
Instead we saw huge memory usage >20GB and an infinite hang for 2 hours.
I've opened a PR, which temporarily mitigates this by checking if the array has only "expanded" and skips the check.
Skip chunk coordinate enumeration in resize when array is only growing (
Skip chunk coordinate enumeration in resize when array is only growing (