Skip to content

[python] Fix Parquet VARIANT reads with multi-chunk nested data#8848

Open
XiaoHongbo-Hope wants to merge 3 commits into
apache:masterfrom
XiaoHongbo-Hope:fix/variant-parquet-bounded-batches
Open

[python] Fix Parquet VARIANT reads with multi-chunk nested data#8848
XiaoHongbo-Hope wants to merge 3 commits into
apache:masterfrom
XiaoHongbo-Hope:fix/variant-parquet-bounded-batches

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Purpose

PyArrow can raise Nested data conversions not implemented for chunked array outputs when projected VARIANT data is decoded into multiple chunks, including within one row group or inside ROW, ARRAY, and MAP columns.

Use bounded ParquetFile.iter_batches() reads while preserving projection, predicates, filter-only columns, and row-group pruning. Exact post-read projection avoids dotted top-level name collisions, and nested shredded VARIANT values are assembled recursively.

Tests

  • Single-row-group multi-chunk reproducers for top-level and nested VARIANT.
  • Dotted top-level and nested-path projection regressions.
  • Plain and shredded VARIANT in ROW, ARRAY, and MAP, including NULL and overflow values.
  • PyArrow 19: 169 passed, 3 subtests passed.
  • PyArrow 6 targeted compatibility: 13 passed, 3 skipped.

PyArrow dataset scans can fail when a nested VARIANT child produces multiple chunks. Read projected Parquet VARIANT columns with ParquetFile.iter_batches while preserving row-group pruning, predicates, and projection.
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as draft July 25, 2026 14:39
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Read VARIANT Parquet in bounded batches [python] Fix Parquet VARIANT reads with multi-chunk nested data Jul 25, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review July 25, 2026 22:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants