Repository navigation
Negative chunks break in newer version #11288
Description
What happened?
Exporting a dataset to zarr with chunks set to -1 raises an error in newer versions.
ValueError: Expected all values to be non-negative. Got (1, -1, -1) instead.What did you expect to happen?
Exporting a dataset to zarr with chunks set to -1 works without error like in older versions
Minimal Complete Verifiable Example
import xarray as xr
zarr_negative_chunk_path = "zarrs/sample_negative_chunk.zarr"
raster_negative_chunk = xr.Dataset(
{"foo": (("key", "y", "x"), np.zeros((2, 1, 1), dtype="i2"))},
coords={"key": [1, 1], "y": [1], "x": [1]},
)
raster_negative_chunk["foo"].encoding = {
"dtype": "i2",
"chunks": (1, -1, -1),
}
raster_negative_chunk.to_zarr(zarr_negative_chunk_path)Steps to reproduce
- Prepare one venv with
zarr==2.18.3andxarray==2025.1.2 - Prepare one venv with
zarr==3.1.1andxarray==2025.11.0 - Run the above code with each venv. The more recent versions will break with the error shown above while the older versions just work.
MVCE confirmation
- Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- Complete example — the example is self-contained, including all data and the text of any traceback.
- Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
- New issue — a search of GitHub Issues suggests this is not a duplicate.
- Recent environment — the issue occurs with the latest version of xarray and its dependencies.
Relevant log output
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[14], line 12
4 raster_negative_chunk = xr.Dataset(
5 {"foo": (("key", "y", "x"), np.zeros((2, 1, 1), dtype="i2"))},
6 coords={"key": [1, 1], "y": [1], "x": [1]},
7 )
8 raster_negative_chunk["foo"].encoding = {
9 "dtype": "i2",
10 "chunks": (1, -1, -1),
11 }
---> 12 raster_negative_chunk.to_zarr(zarr_negative_chunk_path)
File ~/.venvs/xxx/lib/python3.13/site-packages/xarray/core/dataset.py:2377, in Dataset.to_zarr(self, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks, align_chunks, storage_options, zarr_version, zarr_format, write_empty_chunks, chunkmanager_store_kwargs)
2194 """Write dataset contents to a zarr group.
2195
2196 Zarr chunks are determined in the following way:
(...)
2373 The I/O user guide, with more details and examples.
2374 """
2375 from xarray.backends.writers import to_zarr
-> 2377 return to_zarr( # type: ignore[call-overload,misc]
2378 self,
2379 store=store,
2380 chunk_store=chunk_store,
2381 storage_options=storage_options,
2382 mode=mode,
2383 synchronizer=synchronizer,
2384 group=group,
2385 encoding=encoding,
2386 compute=compute,
2387 consolidated=consolidated,
2388 append_dim=append_dim,
2389 region=region,
2390 safe_chunks=safe_chunks,
2391 align_chunks=align_chunks,
2392 zarr_version=zarr_version,
2393 zarr_format=zarr_format,
2394 write_empty_chunks=write_empty_chunks,
2395 chunkmanager_store_kwargs=chunkmanager_store_kwargs,
2396 )
File ~/.venvs/xxx/lib/python3.13/site-packages/xarray/backends/writers.py:797, in to_zarr(dataset, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks, align_chunks, storage_options, zarr_version, zarr_format, write_empty_chunks, chunkmanager_store_kwargs)
795 # TODO: figure out how to properly handle unlimited_dims
796 try:
--> 797 dump_to_store(dataset, zstore, writer, encoding=encoding)
798 writes = writer.sync(
799 compute=compute, chunkmanager_store_kwargs=chunkmanager_store_kwargs
800 )
801 finally:
File ~/.venvs/xxx/lib/python3.13/site-packages/xarray/backends/writers.py:491, in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
488 if encoder:
489 variables, attrs = encoder(variables, attrs)
--> 491 store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
File ~/.venvs/xxx/lib/python3.13/site-packages/xarray/backends/zarr.py:1056, in ZarrStore.store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
1053 else:
1054 variables_to_set = variables_encoded
-> 1056 self.set_variables(
1057 variables_to_set, check_encoding_set, writer, unlimited_dims=unlimited_dims
1058 )
1059 if self._consolidate_on_close:
1060 kwargs = {}
File ~/.venvs/xxx/lib/python3.13/site-packages/xarray/backends/zarr.py:1276, in ZarrStore.set_variables(self, variables, check_encoding_set, writer, unlimited_dims)
1272 encoded_attrs[DIMENSION_KEY] = dims
1274 encoding["overwrite"] = self._mode == "w"
-> 1276 zarr_array = self._create_new_array(
1277 name=name,
1278 dtype=dtype,
1279 shape=shape,
1280 fill_value=fill_value,
1281 encoding=encoding,
1282 attrs=encoded_attrs,
1283 )
1285 writer.add(v.data, zarr_array, region)
File ~/.venvs/xxx/lib/python3.13/site-packages/xarray/backends/zarr.py:1133, in ZarrStore._create_new_array(self, name, shape, dtype, fill_value, encoding, attrs)
1130 if c in encoding:
1131 encoding["config"][c] = encoding.pop(c)
-> 1133 zarr_array = self.zarr_group.create(
1134 name,
1135 shape=shape,
1136 dtype=dtype,
1137 fill_value=fill_value,
1138 **encoding,
1139 )
1140 zarr_array = _put_attrs(zarr_array, attrs)
1141 return zarr_array
File ~/.venvs/xxx/lib/python3.13/site-packages/zarr/core/group.py:2428, in Group.create(self, *args, **kwargs)
2426 def create(self, *args: Any, **kwargs: Any) -> Array:
2427 # Backwards compatibility for 2.x
-> 2428 return self.create_array(*args, **kwargs)
File ~/.venvs/xxx/lib/python3.13/site-packages/zarr/core/group.py:2556, in Group.create_array(self, name, shape, dtype, data, chunks, shards, filters, compressors, compressor, serializer, fill_value, order, attributes, chunk_key_encoding, dimension_names, storage_options, overwrite, config, write_data)
2453 """Create an array within this group.
2454
2455 This method lightly wraps :func:`zarr.core.array.create_array`.
(...)
2550 AsyncArray
2551 """
2552 compressors = _parse_deprecated_compressor(
2553 compressor, compressors, zarr_format=self.metadata.zarr_format
2554 )
2555 return Array(
-> 2556 self._sync(
2557 self._async_group.create_array(
2558 name=name,
2559 shape=shape,
2560 dtype=dtype,
2561 data=data,
2562 chunks=chunks,
2563 shards=shards,
2564 fill_value=fill_value,
2565 attributes=attributes,
2566 chunk_key_encoding=chunk_key_encoding,
2567 compressors=compressors,
2568 serializer=serializer,
2569 dimension_names=dimension_names,
2570 order=order,
2571 filters=filters,
2572 overwrite=overwrite,
2573 storage_options=storage_options,
2574 config=config,
2575 write_data=write_data,
2576 )
2577 )
2578 )
File ~/.venvs/xxx/lib/python3.13/site-packages/zarr/core/sync.py:208, in SyncMixin._sync(self, coroutine)
205 def _sync(self, coroutine: Coroutine[Any, Any, T]) -> T:
206 # TODO: refactor this to to take *args and **kwargs and pass those to the method
207 # this should allow us to better type the sync wrapper
--> 208 return sync(
209 coroutine,
210 timeout=config.get("async.timeout"),
211 )
File ~/.venvs/xxx/lib/python3.13/site-packages/zarr/core/sync.py:163, in sync(coro, loop, timeout)
160 return_result = next(iter(finished)).result()
162 if isinstance(return_result, BaseException):
--> 163 raise return_result
164 else:
165 return return_result
File ~/.venvs/xxx/lib/python3.13/site-packages/zarr/core/sync.py:119, in _runner(coro)
114 """
115 Await a coroutine and return the result of running it. If awaiting the coroutine raises an
116 exception, the exception will be returned.
117 """
118 try:
--> 119 return await coro
120 except Exception as ex:
121 return ex
File ~/.venvs/xxx/lib/python3.13/site-packages/zarr/core/group.py:1122, in AsyncGroup.create_array(self, name, shape, dtype, data, chunks, shards, filters, compressors, compressor, serializer, fill_value, order, attributes, chunk_key_encoding, dimension_names, storage_options, overwrite, config, write_data)
1022 """Create an array within this group.
1023
1024 This method lightly wraps :func:`zarr.core.array.create_array`.
(...)
1117
1118 """
1119 compressors = _parse_deprecated_compressor(
1120 compressor, compressors, zarr_format=self.metadata.zarr_format
1121 )
-> 1122 return await create_array(
1123 store=self.store_path,
1124 name=name,
1125 shape=shape,
1126 dtype=dtype,
1127 data=data,
1128 chunks=chunks,
1129 shards=shards,
1130 filters=filters,
1131 compressors=compressors,
1132 serializer=serializer,
1133 fill_value=fill_value,
1134 order=order,
1135 zarr_format=self.metadata.zarr_format,
1136 attributes=attributes,
1137 chunk_key_encoding=chunk_key_encoding,
1138 dimension_names=dimension_names,
1139 storage_options=storage_options,
1140 overwrite=overwrite,
1141 config=config,
1142 write_data=write_data,
1143 )
File ~/.venvs/xxx/lib/python3.13/site-packages/zarr/core/array.py:4512, in create_array(store, name, shape, dtype, data, chunks, shards, filters, compressors, serializer, fill_value, order, zarr_format, attributes, chunk_key_encoding, dimension_names, storage_options, overwrite, config, write_data)
4507 mode: Literal["a"] = "a"
4509 store_path = await make_store_path(
4510 store, path=name, mode=mode, storage_options=storage_options
4511 )
-> 4512 return await init_array(
4513 store_path=store_path,
4514 shape=shape_parsed,
4515 dtype=dtype_parsed,
4516 chunks=chunks,
4517 shards=shards,
4518 filters=filters,
4519 compressors=compressors,
4520 serializer=serializer,
4521 fill_value=fill_value,
4522 order=order,
4523 zarr_format=zarr_format,
4524 attributes=attributes,
4525 chunk_key_encoding=chunk_key_encoding,
4526 dimension_names=dimension_names,
4527 overwrite=overwrite,
4528 config=config,
4529 )
File ~/.venvs/xxx/lib/python3.13/site-packages/zarr/core/array.py:4331, in init_array(store_path, shape, dtype, chunks, shards, filters, compressors, serializer, fill_value, order, zarr_format, attributes, chunk_key_encoding, dimension_names, overwrite, config)
4328 if order is not None:
4329 _warn_order_kwarg()
-> 4331 meta = AsyncArray._create_metadata_v3(
4332 shape=shape_parsed,
4333 dtype=zdtype,
4334 fill_value=fill_value,
4335 chunk_shape=chunks_out,
4336 chunk_key_encoding=chunk_key_encoding_parsed,
4337 codecs=codecs_out,
4338 dimension_names=dimension_names,
4339 attributes=attributes,
4340 )
4342 arr = AsyncArray(metadata=meta, store_path=store_path, config=config)
4343 await arr._save_metadata(meta, ensure_parents=True)
File ~/.venvs/xxx/lib/python3.13/site-packages/zarr/core/array.py:748, in AsyncArray._create_metadata_v3(shape, dtype, chunk_shape, fill_value, chunk_key_encoding, codecs, dimension_names, attributes)
745 else:
746 fill_value_parsed = fill_value
--> 748 chunk_grid_parsed = RegularChunkGrid(chunk_shape=chunk_shape)
749 return ArrayV3Metadata(
750 shape=shape,
751 data_type=dtype,
(...)
757 attributes=attributes or {},
758 )
File ~/.venvs/xxx/lib/python3.13/site-packages/zarr/core/chunk_grids.py:179, in RegularChunkGrid.__init__(self, chunk_shape)
178 def __init__(self, *, chunk_shape: ChunkCoordsLike) -> None:
--> 179 chunk_shape_parsed = parse_shapelike(chunk_shape)
181 object.__setattr__(self, "chunk_shape", chunk_shape_parsed)
File ~/.venvs/xxx/lib/python3.13/site-packages/zarr/core/common.py:172, in parse_shapelike(data)
170 if not all(v > -1 for v in data_tuple):
171 msg = f"Expected all values to be non-negative. Got {data} instead."
--> 172 raise ValueError(msg)
173 return data_tuple
ValueError: Expected all values to be non-negative. Got (1, -1, -1) instead.Anything else we need to know?
-1 as a chunk was used to indicate no chunking should be done. Removing the -1 solves the error but I would like to understand what has changed for this error to occur now.
I have read the documentation and the zarr migration guide and found no reference to this specifically.
Environment
Working in
INSTALLED VERSIONS
commit: None
python: 3.10.6 (main, Aug 26 2022, 18:10:56) [GCC 9.4.0]
python-bits: 64
OS: Linux
OS-release: 6.6.87.2-microsoft-standard-WSL2
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: C.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.14.6
libnetcdf: 4.9.3
xarray: 2025.1.2
pandas: 2.2.3
numpy: 2.2.3
scipy: 1.15.2
netCDF4: 1.7.2
pydap: None
h5netcdf: 1.8.1
h5py: 3.12.1
zarr: 2.18.3
cftime: 1.6.5
nc_time_axis: None
iris: None
bottleneck: None
dask: 2024.12.1
distributed: None
matplotlib: 3.10.1
cartopy: 0.25.0
seaborn: None
numbagg: None
fsspec: 2025.10.0
cupy: None
pint: 0.24.4
sparse: None
flox: None
numpy_groupies: None
setuptools: 81.0.0
pip: None
conda: None
pytest: 8.3.2
mypy: 1.9.0
IPython: 8.39.0
sphinx: None
Breaking in
INSTALLED VERSIONS
commit: None
python: 3.13.5 (main, Jul 23 2025, 00:37:22) [Clang 20.1.4 ]
python-bits: 64
OS: Linux
OS-release: 6.6.87.2-microsoft-standard-WSL2
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: C.UTF-8
LOCALE: ('C', 'UTF-8')
libhdf5: 1.14.2
libnetcdf: 4.9.4-development
xarray: 2025.11.0
pandas: 2.2.3
numpy: 2.2.3
scipy: 1.15.2
netCDF4: 1.7.2
pydap: None
h5netcdf: 1.8.1
h5py: 3.12.1
zarr: 3.1.1
cftime: 1.6.5
nc_time_axis: None
iris: None
bottleneck: None
dask: 2024.12.1
distributed: None
matplotlib: 3.10.1
cartopy: 0.25.0
seaborn: None
numbagg: None
fsspec: 2025.10.0
cupy: None
pint: 0.24.4
sparse: None
flox: None
numpy_groupies: None
setuptools: 81.0.0
pip: None
conda: None
pytest: 8.3.2
mypy: 1.9.0
IPython: 8.39.0
sphinx: None
Activity
jsignell commentedon Apr 10, 2026
This is really a change in zarr behavior and would be better fixed in zarr-python. See conversation in #11291 for more information.
hhalaby commentedon Apr 10, 2026
@jsignell thanks for the quick feedback !
d-v-b commentedon May 7, 2026
this will be fixed in the next release of zarr-python