Skip to content

feat(parquet): reconstruct unknown fields on reads - #3073

Draft
manuzhang wants to merge 3 commits into
apache:mainfrom
manuzhang:add-unknown-datatype-parquet-read
Draft

manuzhang wants to merge 3 commits into
apache:mainfrom
manuzhang:add-unknown-datatype-parquet-read

Conversation

@manuzhang

Copy link
Copy Markdown
Member

Which issue does this PR close?

No issue is currently linked.

What changes are included in this PR?

This is the Parquet-read follow-up to #2773 and #3072 and is intentionally opened as a draft.

  • Excludes non-physical Unknown fields from Parquet projection masks and reconstructs them as Arrow null arrays.
  • Preserves row counts for empty and Unknown-only projections.
  • Makes embedded field IDs and name mappings authoritative, with guarded positional fallback for files without either.
  • Reconstructs missing nested fields and materializes primitive and container initial defaults.
  • Includes the requested top-level struct/list/map default fix and regression rather than discarding non-primitive defaults.

Until #2773 and #3072 merge, this draft temporarily includes their commits in its cumulative diff. After both predecessors merge, the branch will be rebased onto main so this PR contains only the Parquet-read commit.

Are these changes tested?

  • Added read-path regressions for Unknown-only projections, filters, schema evolution, missing/deleted IDs, nested fallback, and primitive/nested/top-level defaults.
  • cargo test -p iceberg --lib (1,610 passed)
  • make check-fmt
  • make check-clippy
  • git diff --check

AI Disclosure

This PR was prepared with assistance from Codex.

@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Sep 26, 2026
manuzhang and others added 3 commits September 26, 2026 23:05
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
@manuzhang
manuzhang force-pushed the add-unknown-datatype-parquet-read branch from 049573e to 3287f23 Compare September 26, 2026 15:17
@github-actions github-actions Bot removed the stale label Sep 27, 2026
@mfellner mfellner mentioned this pull request Oct 11, 2026
22 of 64 tasks

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant