You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ExtractPythonUDFsSuite: "Python UDF should not break column pruning/filter pushdown -- Parquet V1"
Root Cause
native_datafusion scan doesn't populate the input_file_name() / input_file_block_start() / input_file_block_length() metadata that Spark's FileSourceScanExec provides. This is a functional gap.
Possible Fix
In CometScanRule.nativeDataFusionScan(), detect when the query references InputFileName, InputFileBlockStart, or InputFileBlockLength expressions and fall back to native_iceberg_compat.
Related
Discovered in CI for #3307 (enable native_datafusion in auto scan mode).
Activity
Add IgnoreCometNativeDataFusion annotations for failing native_datafu…
fix: native_datafusion scan falls back when input_file_name() is used
fix: unignore input_file_name Spark SQL tests for native_datafusion
fix: unignore input_file_name Spark SQL tests for native_datafusion (a…
chore: re-enable Spark SQL tests for fixed issues in 3.5.8 diff