Skip to content

fix: use i64 for index file size#586

Merged
JingsongLi merged 1 commit into
apache:mainfrom
XiaoHongbo-Hope:fix_index_file_size_i64
Jul 22, 2026
Merged

fix: use i64 for index file size#586
JingsongLi merged 1 commit into
apache:mainfrom
XiaoHongbo-Hope:fix_index_file_size_i64

Conversation

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #585

Rust declared IndexFileMeta.file_size as i32 even though the index-manifest Avro schema and Java Paimon both use a 64-bit long. Rust builders therefore rejected individual index files larger than 2 GiB, while the fast decoder silently narrowed Java-written values.

This change preserves the full signed 64-bit file size across Rust read and write paths.

Brief change log

  • Widen IndexFileMeta.file_size and all index-file construction paths from i32 to i64.
  • Remove the narrowing cast from the fast index-manifest decoder and clean up redundant downstream conversions.
  • Add a regression that round-trips i32::MAX + 1 through Avro OCF and the fast decoder.
  • Keep checked conversions from u64 and usize so values above i64::MAX still fail explicitly.

Tests

  • cargo fmt --all -- --check
  • cargo test --locked -p paimon --lib — 1800 passed, 1 ignored
  • cargo test --locked -p paimon-datafusion --test system_tables test_table_indexes_system_table -- --exact
  • cargo clippy --locked -p paimon -p paimon-c -p paimon-datafusion --all-targets -- -D warnings

API and Format

This changes the public Rust type of IndexFileMeta.file_size from i32 to i64, so downstream source code with explicit i32 construction must be updated.

The storage format does not change: _FILE_SIZE remains an Avro long. Existing manifests remain compatible, and Java can read values written by this change. Older Rust readers still cannot correctly read values above i32::MAX and should be upgraded.

_ROW_COUNT remains unchanged in this focused PR because widening it affects a broader set of row-count and vector-index APIs.

Documentation

No documentation update is required.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 8c90381 into apache:main Jul 22, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

IndexFileMeta truncates file sizes above 2 GiB

2 participants