Permission denied when multiple Processes write to the same file #3522
Description
Zarr version
3.1.3
Numcodecs version
0.16.3
Python Version
3.13.4
Operating System
Windows 11 24H2
Installation
using pip into virtual environment
Description
I am encountering an issue when writing to a Zarr array using Python multiprocessing. (Or more specifically: Missuse Dask's Multiprocessing, as i write into a zarr array inside a map_block function)
The problem occurs when the array chunks contain multiple elements. Writing works correctly
when each chunk contains only a single element, but fails when the chunk size is larger.
Example:
Here group is a zarr group and zarr_array is another zarr array.
My code that writes to b_arr in a multprocessed manner works, when this creates the array:
b_arr=group.create_array(
name=name,
shape=zarr_array.cdata_shape,
chunks=(1, 1),
dtype=np.uint64,
fill_value=0,
)
and here not:
b_arr=group.create_array(
name=name,
shape=zarr_array.cdata_shape,
chunks=zarr_array.cdata_shape,
dtype=np.uint64,
fill_value=0,
)
I believe that zarr should work when mutliple processes concurrenty write into an array, so i consider it a bug.
Steps to reproduce
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
# ]
# ///
#
# This script automatically imports the development branch of zarr to check for issues
import zarr
# your reproducer code
# zarr.print_debug_info()Additional output
No response
Activity
[-]PermissionError: [Errno 13] Permission denied when multiple Processes write to the same file[/-][+]Permission denied when multiple Processes write to the same file[/+]FlynnOConnell commentedon Oct 22, 2025
I am getting this error as well
melonora commentedon Nov 3, 2025
I can confirm as well
MaksHess commentedon Dec 12, 2025
Same here
vesuvius: harden ink-detection disk cache against concurrent access
Tomatokeftes commentedon Sep 15, 2026
We're hitting this a lot on Windows 11 (zarr 3.1.3 and 3.3.0) — for us it aborts real writes, not just tests.
Two things that might help unstick it:
It doesn't seem to be about multiple processes. It reproduces single-threaded, in one process, with zarr not imported at all —
tmp.replace(dest)in a loop onto a destination that already exists fails ~4% of the time for me.And it doesn't look like antivirus, which is the usual explanation. Replacing a fresh name never failed here (0 in 4000); only reusing the same name does (~5%). Same write pattern, same scanner, only the name differs — which points more at NTFS file tunnelling. There's a Microsoft thread where it reproduces with Defender explicitly disabled and is fixed on Server 2025: https://learn.microsoft.com/en-us/answers/questions/5559596/movefileex-can-produce-a-access-is-denied-error-on
v2 had a retry for exactly this (#698) and it didn't come across when atomic writes landed in #3412 — @d-v-b suggested porting it on #597. PR here: #4358. Retries only winerror 5/32, leaves the
exclusivepath'sFileExistsErroralone, no platform check needed.fix(store): retry the LocalStore rename when the destination is busy (#…
refactor(store): polish the LocalStore rename retry from #4358 (#4359)