Skip to content

ObjectStore.list_dir corrupts directory names due to lstrip vs removeprefix #3753

Closed

Description

@TomNicholas

_transform_list_dir in zarr/storage/_obstore.py uses str.lstrip(prefix) on line 263 to strip path prefixes from common_prefixes. But lstrip strips individual characters from the argument, not a prefix string. This corrupts directory names when the prefix shares characters with child directory names.

Line 264 already correctly uses removeprefix for objects — the fix is to do the same for prefixes on line 263.

Reproducer:

# /// script                                                                                                                                                   
# requires-python = ">=3.12"
# dependencies = ["zarr>=3", "obstore", "xarray", "numpy"]
# ///
import asyncio, tempfile
import numpy as np, xarray as xr                                                                                                                               
from obstore.store import LocalStore
from zarr.storage import ObjectStore                                                                                                                           
                
tmpdir = tempfile.mkdtemp()                                                                                                                                    
parent_dir = f"{tmpdir}/bucket_root"
filepath = f"{parent_dir}/subdir/data.zarr"                                                                                                                    
                
xr.Dataset({"temp": (("x", "y"), np.arange(12, dtype="float32").reshape(3, 4))}).to_zarr(                                                                      
    filepath, consolidated=False, zarr_format=2
)                                                                                                                                                              
                
store = LocalStore(prefix=parent_dir)
zarr_store = ObjectStore(store=store)

async def main():
    results = [item async for item in zarr_store.list_dir("subdir/data.zarr")]
    print(results)  # ['emp', '.zattrs', '.zgroup'] — 'temp' corrupted to 'emp'                                                                                
                                                                                                                                                                
asyncio.run(main())

Fix:

- prefixes = [obj.lstrip(prefix).lstrip("/") for obj in list_result["common_prefixes"]]                                                                        
+ prefixes = [obj.removeprefix(prefix).lstrip("/") for obj in list_result["common_prefixes"]]

Activity

  1. TomNicholas commented on Mar 9, 2026

    @TomNicholas
    MemberAuthor

    This is very similar to #2778

  2. dcherian commented on Mar 9, 2026

    @dcherian
    Contributor

    we should add a pre-commit linter that prevents use of lstrip and rstrip

  3. TomNicholas commented on Mar 9, 2026

    @TomNicholas
    MemberAuthor

    oops this is a duplicate of #3639, though the fix for #3639 is not released yet.

    @maxrjones this not being released breaks VirtualiZarr's ZarrParser for any zarr store not in the root of a bucket... (see zarr-developers/VirtualiZarr#912)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions