[sainimal1ba-hue]

Summary

Parquet statistics pruning currently misses filters such as (col("x") + 1) >= 76 and col("s").str.starts_with("s08"): PyArrow's row-group pruning only evaluates column-vs-literal comparisons against statistics, so anything else keeps every row group. This rewrites those leaves into the exact column ranges they span, so pruning, negation and the fully-matched test all see comparisons PyArrow can evaluate.

Design

For each leaf PyArrow cannot prune with:

• str.starts_with(col, p) becomes col >= p and col < next(p). Boundaries: the empty prefix matches every non-null string, a prefix ending in U+D7FF jumps the surrogate block to U+E000, and after U+10FFFF no upper end exists, so the leaf stays and only the lower bound is appended.
• (col + c) >=/> t becomes col >= t - c on unsigned columns (the kernel promotes to int64, no sum wraps), and col >= t - c and col <= MAX - c on signed columns (the kernel computes at the column's own width, so sums above MAX - c wrap back below the threshold and never match). Literals whose kernel path differs by operand order keep the leaf and only get the necessary lower bound appended.

Because each bound is equivalent to its leaf, nulls included, the leaf can be replaced outright. The rewritten filter is semantically equal to the original, so NOT recurses into it and the fully-matched test (~filter against statistics) runs on the rewritten filter: exact survivor counts drive limit push-down. Filters with no modeled leaf come back unchanged, so nothing else changes for them.

Before / after

Before: Wrapped arithmetic and prefix filters kept every row group. ~((x + 1) >= 901) pruned nothing, and on the issue's reproduction filter((x + 1) >= 1).limit(10) opened 112 files where the plain x >= 0 form opened 28.

After: FooterReader prunes with the exact ranges, groups fully inside the filter are marked fully matched, and ~ prunes too.

flowchart LR
    subgraph Before
        A["IO: Parquet"] --> B["FooterReader"] --> C["Unevaluable filter"] --> D["All row groups"]
    end

    subgraph After
        E["IO: Parquet"] --> F["FooterReader"] --> G["Exact statistics bounds"] --> H["Candidate row groups"]
    end

    D --> I["Original row filter"]
    H --> J["Original row filter"]

    classDef changed fill:#fff9c4,stroke:#f9a825,color:#000
    class G,H changed

Loading

Subsystem: Ray Data IO (Parquet footer reader).

Testing

• python -m pytest -q python/ray/data/tests/datasource_v2/test_footer_reader.py: 256 passed. New regression tests: pruning under ~((id + 1) >= 901), fully-matched groups for arithmetic filters, the signed wrap region, and unsigned promotion.
• ruff check and ruff format --check on both changed files: clean.
• A brute-force equivalence sweep (7 column types x 5 literal forms x 2 comparison operators x 7 boolean contexts, 490 combinations): the rewritten filter selects exactly the same rows as the original wherever the exact bound is claimed, and only a superset on the appended path.

Per the review, the remaining forms (<, <=, == on col + c, col - c, a literal on the left, col * c, float arithmetic, timestamp plus a duration) will follow in a separate PR, each with an exact bound from the start.

Fixes #67565.

AI assistance was used to develop this change. I tested and I have reviewed all the changes.

[gemini-code-assist]

Code Review

This pull request introduces a _derived_statistics_predicate helper function to derive simplified predicates (such as positive integer addition and string prefix matches) that Parquet statistics can evaluate, enabling more effective row-group pruning during Parquet reads. Comprehensive unit tests are also added to verify the correctness and safety of these derived predicates. The reviewer suggested improving the robustness of the derived predicate pruning by supporting commutative addition (e.g., 1 + col("x") in addition to col("x") + 1).

[sainimal1ba-hue]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request introduces _derived_statistics_predicate to optimize Parquet row-group pruning by deriving simpler predicates from arithmetic operations and string prefix matches, along with comprehensive unit tests. The feedback suggests extending this optimization to compound filters by recursively traversing logical AND and OR operators, and adding corresponding test cases to verify this behavior.

[sainimal1ba-hue]

@gemini-code-assist review

[cursor]

Stale Bugbot comment from a previous run.

[gemini-code-assist]

Code Review

This pull request introduces a _derived_statistics_predicate helper to extract simplified conditions from filter expressions (such as addition offsets and string prefixes) for Parquet row-group pruning, and integrates it into the FooterReader. The review feedback recommends dynamically determining the column's bit_width for signed integers instead of hardcoding 64-bit limits, which can cause failures or silent disabling of pruning for smaller integer types like int32. It also suggests adding corresponding test cases to verify that unsafe arithmetic is correctly skipped for these smaller types.

[sainimal1ba-hue]

@gemini-code-assist review

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit be1cf4e. Configure here.}

[gemini-code-assist]

Code Review

This pull request introduces _derived_statistics_predicate to derive simplified predicates from complex filter expressions that Parquet statistics can evaluate, allowing for more aggressive row-group pruning during footer reading. It also adds comprehensive tests to verify this behavior under various conditions, including arithmetic operations, string prefixes, and null values. The review feedback suggests extending this optimization by preserving simple unary operations (like IS_NULL and IS_NOT_NULL) and list operations (like IN and NOT_IN) within compound predicates to further improve pruning power, which also requires importing UnaryExpr.

[sainimal1ba-hue]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request introduces a mechanism to derive a predicate that Parquet statistics can evaluate (_derived_statistics_predicate) to prune row groups during reading, along with comprehensive tests. The feedback suggests optimizing the derived predicate for Operation.IN by avoiding the conservative null-inclusive bound (| expr.left.is_null()), which is only necessary for Operation.NOT_IN.

[sainimal1ba-hue]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request introduces row-group pruning based on derived statistics predicates in the Parquet footer reader. It adds the _derived_statistics_predicate helper to extract evaluable conditions from expressions and applies them during the footer reading process to drop irrelevant row groups. Extensive unit tests are also added to cover various edge cases. The reviewer suggested catching all Exceptions instead of a specific list of exceptions when deriving the predicate to prevent any unexpected runtime errors from crashing the read operation.

[sainimal1ba-hue]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request introduces a _derived_statistics_predicate helper to extract safe pruning predicates from filter expressions, allowing row-group pruning based on Parquet statistics. It also adds comprehensive unit tests covering various data types and edge cases. The reviewer suggests extending this optimization to support unsigned integer columns by dynamically calculating the minimum and maximum bounds based on the integer's signedness.

[sainimal1ba-hue]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request introduces a _derived_statistics_predicate helper function to extract safe, evaluable Parquet statistics predicates from filter expressions, integrating it into the FooterReader to improve row-group pruning efficiency. It also adds extensive unit tests covering various predicate types, boundary conditions, and fallback behaviors. Feedback on the changes suggests expanding the membership check to support other collection types like tuple and set to prevent unnecessary null-inclusive fallbacks, and supporting NumPy integer types in arithmetic bound derivations by safely casting them to Python int to avoid silent overflow issues.

[sainimal1ba-hue]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request introduces a new function _derived_statistics_predicate in the Parquet footer reader to derive safe pruning predicates from filter expressions, enabling row-group pruning for operations like integer addition and string prefix matching. It also updates FooterReader to utilize these derived predicates and adds extensive unit tests. The review feedback suggests improving the robustness of the derived predicate logic for integer addition by normalizing or mirroring comparison operands and operators to support mirrored comparisons.

[AarryaSaraf]

Hi @sainimal1ba-hue, thank you for your contribution!
This makes row-group pruning worse for some filters master already prunes. The derived filter replaces the original one for pruning, and a filter on a float column that the deriver can't handle now skips pruning altogether. For eg. a file with 5 row groups of 200 rows:

path = "/tmp/t.parquet"
x = list(range(1000))
pq.write_table(pa.table({"x": x, "f": [float(v) for v in x]}), path, row_group_size=200)

def kept(expr):
    reader = FooterReader(pafs.LocalFileSystem(), filter_expr=expr)
    chunked = reader._read_and_chunk(FileInfo(path, os.path.getsize(path)))
    return [c.unit_ids[0] for c in chunked.row_groups]

print(kept(((col("x") + 1) >= 1) & ~(col("x") < 800)))
print(kept(col("f").is_in([900.0])))
print(kept(~(col("f") < 800.0)))

master:  [4]              [4]              [4]
this PR: [0, 1, 2, 3, 4]  [0, 1, 2, 3, 4]  [0, 1, 2, 3, 4]

The NaN rows the float check protects are dropped anyway by pyarrow's own scanner in the read task (~(col("f") < 100.0) over row groups [1.0, 2.0] and [50.0, NaN] returns [] on master and on this PR), so that's a separate pyarrow issue. Could we keep every filter that master prunes pruning at least as well?

[sainimal1ba-hue]

Hi @sainimal1ba-hue, thank you for your contribution! This makes row-group pruning worse for some filters master already prunes. The derived filter replaces the original one for pruning, and a filter on a float column that the deriver can't handle now skips pruning altogether. For eg. a file with 5 row groups of 200 rows:

path = "/tmp/t.parquet"
x = list(range(1000))
pq.write_table(pa.table({"x": x, "f": [float(v) for v in x]}), path, row_group_size=200)

def kept(expr):
reader = FooterReader(pafs.LocalFileSystem(), filter_expr=expr)
chunked = reader._read_and_chunk(FileInfo(path, os.path.getsize(path)))
return [c.unit_ids[0] for c in chunked.row_groups]

print(kept(((col("x") + 1) >= 1) & (col("x") < 800)))
print(kept(col("f").is_in([900.0])))
print(kept((col("f") < 800.0)))

master:  [4]              [4]              [4]
this PR: [0, 1, 2, 3, 4]  [0, 1, 2, 3, 4]  [0, 1, 2, 3, 4]

The NaN rows the float check protects are dropped anyway by pyarrow's own scanner in the read task (~(col("f") < 100.0) over row groups [1.0, 2.0] and [50.0, NaN] returns [] on master and on this PR), so that's a separate pyarrow issue. Could we keep every filter that master prunes pruning at least as well?

added a regression test. all 247 tests in the file pass.

dropped the float guard like u told. scanner drops those NaN rows anyway.

swapped the NaN preservation test for one that checks the derived bound never contradicts a NaN row.

kept master pruning. your three cases all give [4]. derived bound runs next to the original filter and only narrows more, so (x + 1) >= 76 and starts_with still prune. if the original filter cant handle a column the derived bound prunes alone. no derived bound, same as master.

[AarryaSaraf]

Since _read_and_chunk now prunes with the original filter first, could the deriver just rewrite the two leaf forms in place (leaf -> leaf & bound) and leave the rest of the filter to pyarrow? That would drop most of _derived_statistics_predicate, and an OR whose other branch the deriver doesn't model would prune too.

[sainimal1ba-hue]

Since _read_and_chunk now prunes with the original filter first, could the deriver just rewrite the two leaf forms in place (leaf -> leaf & bound) and leave the rest of the filter to pyarrow? That would drop most of _derived_statistics_predicate, and an OR whose other branch the deriver doesn't model would prune too.

Done, pushed. The deriver is now _add_statistics_bounds. It walks the
filter and only touches the two leaf forms it models: the arithmetic
comparison and starts_with. Each gets rewritten in place as leaf & bound,
and everything else passes through unchanged, so pyarrow keeps handling
plain comparisons, null checks and membership on its own. That dropped
the EQ/NE, IS_NULL and IN/NOT_IN handling entirely.

An OR with an unmodeled branch prunes now too. The modeled leaf stays in
the filter, so pyarrow still evaluates the other branch against stats
and kills the group when the bound rules it out and the other branch
doesn't match. The old deriver needed both OR branches modeled before it
could prune at all.

The filter also gets split once now. A filter with no modeled leaf comes
back unchanged, so there's no second pass over the same statistics, and
if the added bounds can't be evaluated against a file's stats it falls
back to the plain filter.

Added two regression tests: one counting split passes for a plain
comparison, one checking an OR with an unmodeled branch still prunes.

[AarryaSaraf]

The two bounds added here describe the leaf exactly (starts_with(p) is the same as p <= s < next(p), and x + c >= t the same as x >= t - c unless x + c wraps), so we should also be able to prune under ~ and use them for limit pushdown. Today ~((x + 1) >= 901) keeps every row group, and on the issue's 32 files filter((x + 1) >= 1).limit(10) takes 112 file opens vs 28 for filter(x >= 0).limit(10).

Let's split this into 2 PRs:

This PR

• Keep _add_statistics_bounds with the two forms it has: (col + int) >= / > int and str.starts_with.
• Make both bounds exact, so filters under ~ prune and fully matched row groups drive limit pushdown.
• Update the description to the current design.

Follow-up PR

• More forms, each with an exact bound from the start: <, <=, == on col + c; col - c; a literal on the left (901 <= col + 1); col * c; float arithmetic; timestamp plus a duration (col("ts") + offset >= t0, the issue's use case).