[v3] revisit runtime config

This issue tracks a evaluation of the v3 runtime config

Context

The v3 branch runtime config currently looks like this:

Lines 34 to 38 in 76c3450

    
           @dataclass(frozen=True) 
        
           class RuntimeConfiguration: 
        
               order: Literal["C", "F"] = "C" 
        
               concurrency: Optional[int] = None 
        
               asyncio_loop: Optional[AbstractEventLoop] = None

This is then attached to Array/Group classes

zarr-python/src/zarr/v3/array.py

Lines 51 to 55 in 76c3450

    
           @dataclass(frozen=True) 
        
           class AsyncArray: 
        
               metadata: ArrayMetadata 
        
               store_path: StorePath 
        
               runtime_configuration: RuntimeConfiguration

A few things are missing here:

User experience
- as a user, I may want to set config settings and forget about them (e.g. order, concurrency)
Portability
- I don't know for sure but I really doubt that putting the AsyncIO loop on the Array class is going to work when it comes to serialization

Improvements

So looking for some ideas for how to manage this better. Two ideas:

Xarray style set-options: https://docs.xarray.dev/en/stable/generated/xarray.set_options.html
- Pros: allows for validation and is typed
- Cons: a bit bespoke, doesn't support environment variables or a config file option
Dask style config - https://donfig.readthedocs.io/en/latest/
- Pros: very flexible framework, support for environment variables and config files, nested namespaces, etc.
- Cons: extra dependency (though we could vendor it), no typing or validation

what do we expect to go in the runtime config?

Order
Concurrency
logging settings
what else?

@maxrjones

I spoke with @maxrjones today about this. Our thought for now was to try using donfig and see how it goes. We can continue to evaluate the dependency vs vendoring and typing/validation as needed.

cc @djhoese

Additional config options:

Specify alternate implementations for CodecPipeline (e.g. for a rust-based codec pipeline)
Specify alternate implementations for codecs (e.g. for GPU-based batch-aware codecs)
Batch size in the HybridCodecPipeline

@normanrz

thanks @normanrz! Joe mentioned you asked about this today. I'm working on getting a minimal PR opened now and should have that submitted within the next couple hours.

@normanrz

@normanrz - #1855 is now in the v3 branch. Should clear the way to add additional config options as needed.

@maxrjones

Thanks @maxrjones for getting this moving!

I think this is a great way of dealing with configurations. Thanks!

Navigation Menu

Repository navigation

[v3] revisit runtime config #1772

Context

A few things are missing here:

Improvements

what do we expect to go in the runtime config?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	@dataclass(frozen=True)
	class RuntimeConfiguration:
	order: Literal["C", "F"] = "C"
	concurrency: Optional[int] = None
	asyncio_loop: Optional[AbstractEventLoop] = None

	@dataclass(frozen=True)
	class AsyncArray:
	metadata: ArrayMetadata
	store_path: StorePath
	runtime_configuration: RuntimeConfiguration

[v3] revisit runtime config #1772

Description

Context

A few things are missing here:

Improvements

what do we expect to go in the runtime config?

Activity

jhamman commented on Apr 19, 2024

normanrz commented on May 8, 2024

maxrjones commented on May 8, 2024

jhamman commented on May 10, 2024

jhamman commented on May 10, 2024

normanrz commented on May 14, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions