You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
partitioned_by_file_group not serialized in FileScanExecConf protobuf breaks datafusion-distributed #22363
The partitioned_by_file_group field in FileScanConfig is not included in the FileScanExecConf protobuf message. After deserialization, the field defaults to false, enabling work stealing? even when the original plan had it disabled.
This does seem to manifest as duplicated rows in results when using datafusion-distributed latest main.
To Reproduce
let scan_config = FileScanConfigBuilder::new(url, file_source).with_partitioned_by_file_group(true).build();assert!(scan_config.partitioned_by_file_group);// truelet bytes = physical_plan_to_bytes(DataSourceExec::from_data_source(scan_config))?;let result = physical_plan_from_bytes(bytes)?;let config = result.downcast_ref::<DataSourceExec>().data_source().downcast_ref::<FileScanConfig>();assert!(config.partitioned_by_file_group);// FAILS - now false
Expected behavior
partitioned_by_file_group should survive protobuf roundtrip
Additional context
Field was introduced in #21351 and #21342 but not added to the protobuf schema. With 3 workers, aggregations return 3x expected values (e.g., count = 198 instead of 66).
Activity
Fix missing field `partitioned_by_file_group` in serialization (apach…