You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Object store cache key drops the ABFS container, so two containers in one storage account share a store instance #4993
prepare_object_store_with_configs in native/core/src/parquet/parquet_support.rs caches object store instances under (url_key, config_hash), where url_key is built from:
let url_key = format!("{}://{}",
scheme,&url[url::Position::BeforeHost..url::Position::AfterPort],);
Position::BeforeHost starts after the URL userinfo. For ABFS, the container is the userinfo, not the host:
So abfss://c1@acct.dfs.core.windows.net/... and abfss://c2@acct.dfs.core.windows.net/... both produce the key abfss://acct.dfs.core.windows.net. config_hash is computed over the per-session Hadoop config map, which does not vary by container, so it does not disambiguate them either.
Meanwhile MicrosoftAzureBuilder::parse_url bakes the container into the store instance from the URL userinfo, and the Path handed to that store is container-relative (the bucket/container is stripped by Path::from_url_path(url.path()) in PhysicalPlanner::get_partitioned_files).
The consequence: within one executor process, reading abfss://c2@acct/data/f.parquet after abfss://c1@acct/data/f.parquet should hit the cached c1 store and read c1's data.
Steps to reproduce
Not yet reproduced. This was found by code reading during review of an unrelated change. Confirming it needs an Azure storage account with two containers holding same-named paths, read from a single executor in one application.
Expected behavior
Each ABFS container resolves to its own object store instance, and a read from one container never returns another container's data.
Additional context
S3, GCS, and HDFS are unaffected: their bucket/host lives in the URL host component, which url_key does keep.
The same key derivation is now also used to key a shared Parquet metadata cache, where the impact is narrower (entries are additionally validated against file size and modification time). That is documented as a known limitation there; this issue is about the store cache itself, where no such validation exists.
The object store cache is process-wide and unbounded by design, so a wrong entry persists for the executor's lifetime.
Activity
peterxcli commentedon Jul 27, 2026
let me take a stab at this.
docs: document known correctness issues in the compatibility guide (#…
docs: drop compatibility notes for bugs that are now fixed
docs: drop compatibility notes for bugs that are now fixed (#5154)
docs: document known correctness issues in the compatibility guide (a…
docs: drop compatibility notes for bugs that are now fixed (apache#5154)