Atomic writes in LocalStore

@rabernat

Zarr v2 took care in DirectoryStore to use atomic writes by renaming files into place, which ensures that writes to disk are either complete or not written at all:

zarr-python/zarr/storage.py

Lines 1146 to 1161 in f1978dd

    
           # write to temporary file 
        
           # note we're not using tempfile.NamedTemporaryFile to avoid restrictive file permissions 
        
           temp_name = file_name + "." + uuid.uuid4().hex + ".partial" 
        
           temp_path = os.path.join(dir_path, temp_name) 
        
           try: 
        
               self._tofile(value, temp_path) 
        
               # move temporary file into place; 
        
               # make several attempts at writing the temporary file to get past 
        
               # potential antivirus file locking issues 
        
               retry_call(os.replace, (temp_path, file_path), exceptions=(PermissionError,)) 
        
           finally: 
        
               # clean up if temp file still exists for whatever reason 
        
               if os.path.exists(temp_path):  # pragma: no cover 
        
                   os.remove(temp_path)

I don't see any similar logic in Zarr v3, which just seems to using Python's file interface (via open):

zarr-python/src/zarr/storage/_local.py

Lines 52 to 55 in c9509ee

    
           with path.open("r+b") as f: 
        
               f.seek(start) 
        
               # write takes any object supporting the buffer protocol 
        
               f.write(value.as_buffer_like())

This would be useful to still support in Zarr v3.

Per @rabernat, this was almost certainly not intentional.

Originally posted by @shoyer in #3410

Use atomic writes in LocalStore

Use atomic writes for new files in LocalStore (#3412)

Navigation Menu

Repository navigation

Atomic writes in LocalStore #3411

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	# write to temporary file
	# note we're not using tempfile.NamedTemporaryFile to avoid restrictive file permissions
	temp_name = file_name + "." + uuid.uuid4().hex + ".partial"
	temp_path = os.path.join(dir_path, temp_name)
	try:
	self._tofile(value, temp_path)

	# move temporary file into place;
	# make several attempts at writing the temporary file to get past
	# potential antivirus file locking issues
	retry_call(os.replace, (temp_path, file_path), exceptions=(PermissionError,))

	finally:
	# clean up if temp file still exists for whatever reason
	if os.path.exists(temp_path): # pragma: no cover
	os.remove(temp_path)

	with path.open("r+b") as f:
	f.seek(start)
	# write takes any object supporting the buffer protocol
	f.write(value.as_buffer_like())

Atomic writes in LocalStore #3411

Description

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions