You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Iceberg writes drop Hadoop GCS configuration for HadoopFileIO + gs:// #5637
The native Iceberg write eligibility path supports gs:// locations and NativeConfig.extractObjectStoreOptions extracts the corresponding fs.gs.* settings from the Hadoop configuration.
However, CometIcebergNativeWrite then passes those extracted settings through:
For a gs:// location, the extracted fs.gs.* settings are therefore not preserved when building the native GCS FileIO.
This is particularly problematic for tables using HadoopFileIO, because GCS credentials, endpoint configuration, and other filesystem settings may exist only in the Hadoop Configuration rather than FileIO.properties().
The JVM Iceberg writer and the native writer can consequently operate with different effective storage configuration.
Depending on the environment, the native writer could:
fail even though the JVM writer succeeds;
use a different GCS endpoint; or
resolve a different ambient/default identity than the one configured for the Hadoop filesystem.
Native writes should fail closed rather than silently changing storage identity or endpoint configuration.
Steps to reproduce
No response
Expected behavior
The native writer must either:
faithfully translate the relevant Hadoop GCS configuration into the native GCS backend configuration; or
fail closed and fall back to the JVM Iceberg writer for HadoopFileIO + gs://.
Until a complete and tested configuration bridge exists, the safer behavior is to decline native acceleration for this combination.
A table using an actual GCSFileIO whose properties already provide supported native gcs.* settings can remain eligible.
A simple fs.gs.* → gcs.* key rename is unlikely to be sufficient for all settings. For example, Hadoop GCS configuration may reference a service-account key file, whereas the native backend can require a different credential representation.
The important invariant for the write path is that enabling Comet must not cause the native writer to resolve a different storage identity or endpoint from the JVM writer.
Activity
fix: Fall back to Spark for native Iceberg writes to gs:// through Ha…
fix: Fall back to Spark for native Iceberg writes to gs:// through Ha…
fix: Fall back to Spark for native Iceberg writes to gs:// through Ha…
fix: Fall back to Spark for native Iceberg writes to gs:// through Ha…
fix: fall back to Spark for native Iceberg writes to gs:// through Ha…