Skip to content

Native Parquet writer ignores Spark's Parquet writer properties (block size, page size, dictionary, writer version, statistics) #5304

Open

Description

@andygrove

What is the problem the feature request solves?

ParquetWriterExec builds its WriterProperties with compression and nothing else (native/core/src/execution/operators/parquet_writer.rs):

let props = WriterProperties::builder()
    .set_compression(compression)
    .build();

Every other Parquet write knob Spark exposes is therefore silently ignored, and files come out with arrow-rs defaults rather than the ones the user configured:

  • parquet.block.size (row group size) — tracked as an unchecked item on [EPIC] Improve Comet Native writer #2967
  • parquet.page.size / parquet.page.row.count.limit
  • parquet.enable.dictionary and dictionary page size
  • spark.sql.parquet.writer.version (PARQUET_1_0 / PARQUET_2_0)
  • statistics level, and bloom filter settings
  • spark.sql.files.maxRecordsPerFile (needs file rolling, so partly separate)

Row group size is the one with real consequences: the default row group size decides read parallelism and memory for every downstream consumer, and a user who set parquet.block.size gets no error, just a differently shaped file.

This also matters beyond the V1 write path — #4658 splits Iceberg writes into writer + committer operators with the stated goal of writing data files natively next, and Iceberg data files are read by engines other than Spark.

Describe the potential solution

Thread the resolved options through the ParquetWriter protobuf message and set them on WriterProperties. Spark's ParquetOptions / ParquetWriteSupport are the reference for precedence and defaults.

Related: #2814 (compression settings), #3425 (INT96), #3427 (Spark version in footer metadata).

Activity

  1. andygrove commented on Oct 16, 2026

    @andygrove
    MemberAuthor

    Measured while auditing nested types in the native writers: for nested rows, the row-count cap makes the row group size grow with the fan-out. On main at c230bf4 (Spark 4.1.3), 100,000 rows of array<struct<s: string, d: double>> with 100 elements each came out of the native writer as one row group of 239 MB (268 MB uncompressed). Spark wrote the same rows as two row groups, of 127 MB and 111 MB. The writer holds the whole in-progress row group in a reservation it can't spill (reservation.try_resize(writer.memory_size())), so a row group the pool won't grant fails the task. At 1,000 elements per row this one would be about 2.4 GB. arrow-rs 59 has WriterProperties::set_max_row_group_bytes, which parquet.block.size could map to.

    One more property the list above doesn't name: parquet-rs truncates column-chunk min/max statistics to 64 bytes (DEFAULT_STATISTICS_TRUNCATE_LENGTH), where parquet-mr writes them in full by default (parquet.statistics.truncate.length). Row-group pruning on long strings is weaker as a result, though still correct.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions