You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support gzip compression in native Parquet writes #4929
Comet's native Parquet writer supports none, snappy, lz4, and zstd. A write configured with spark.sql.parquet.compression.codec=gzip (or .option("compression", "gzip")) falls back to Spark's writer, since CometDataWritingCommand.getSupportLevel reports Unsupported("Unsupported compression codec: gzip").
gzip is a valid Spark Parquet codec and is widely used, so this is an avoidable loss of native write coverage.
While looking at this, two related gaps in the same code path:
parseCompressionCodec does not honor Spark's codec precedence. Spark's ParquetOptions resolves compression > parquet.compression (ParquetOutputFormat.COMPRESSION) > spark.sql.parquet.compression.codec, but Comet checks only the first and third. A write with .option("parquet.compression", "gzip") and the default SQLConf is therefore natively written as snappy, silently substituting the user's requested codec.
Spark accepts uncompressed as a spelling of none, but Comet's supported-codec set only lists none, so uncompressed falls back unnecessarily.
Describe the potential solution
Add a Gzip value to the protobuf CompressionCodec enum and map it in the native Parquet writer to Compression::GZIP(GzipLevel::default()) (level 6, matching parquet-mr's zlib default). Fix the codec precedence and accept the uncompressed alias.
Compression levels remain non-configurable (gzip and zstd use fixed defaults), so parquet-mr's parquet.compression.codec.{gzip,zstd}.level configs are still ignored. That is out of scope here.
Activity