Repository navigation
fix: propagate nested Column drop_invalid_rows without mutation - #2551
Merged
cosmicbboy merged 2 commits intoOct 8, 2026
Merged
cosmicbboy merged 2 commits into
cosmicbboy merged 2 commits into
Conversation
A Column(drop_invalid_rows=True) nested inside a DataFrameSchema drops its own failing rows and returns a filtered frame instead of raising, but the container backends discarded that return value, so invalid rows survived in the validated output (issue unionai-oss#2529). This replaces the pandas in-place approach from PR unionai-oss#2521 with a non-mutating, uniform implementation across the backends that support column-level drop_invalid_rows: - pandas: revert the in-place check_obj.drop(..., inplace=True) mutation and instead thread the filtered frame through the container's local check_obj binding. - polars / narwhals (covers pyarrow): lazy frames are immutable, so the filtered frame returned by a drop_invalid_rows component's validate() is threaded explicitly through the container's local lazy frame and recovered into the returned output. Skipped under head/tail/sample subsampling because a dropped subsample cannot be mapped back onto the full frame (matching the earlier pandas behavior). The abstract run_schema_component_checks(check_obj, schema, components, lazy) -> list[CoreCheckResult] signature is unchanged. Each drop component is validated once for error collection and once to recover its filtered frame; schema-level errors are ignored during the latter because they are already handled by run_schema_component_checks. Backends without column-level drop_invalid_rows (pyspark, ibis, xarray, tensordict) are out of scope: the pandas column API exposes the flag in the ibis API but the ibis backend does not implement a drop path. Adds regression tests for pandas, polars, narwhals and pyarrow asserting the nested column keeps dropping its failing rows (propagating to the full output row across all columns) even when the DataFrameSchema itself does not set drop_invalid_rows, plus a pandas guard that the caller's frame is not mutated. ## Backward compatibility No existing tests were modified except removing the in-place mutation from PR unionai-oss#2521's pandas implementation and adding new regression tests. User-visible outputs are preserved for pandas (the unionai-oss#2521 regression test still passes); the change only removes reliance on mutating the caller's frame, which is an internal implementation detail and not part of the public contract.
The _ArrayKwargs TypedDict declared a 'width' key that is never set (the code only ever writes kwargs['shape']). This made mypy type **kwargs as possibly containing 'width', which newer polars stubs reject with 'Extra argument "width" from **args for Array' when the deprecated arg was removed. Removing the unused key resolves the mypy error without any behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A
Column(drop_invalid_rows=True)nested inside aDataFrameSchemadrops its own failing rows and returns a filtered frame instead of raising, but the container backends discarded that return value, so invalid rows survived in the validated output (issue #2529).This PR fixes that with a non-mutating, uniform implementation and replaces the pandas in-place approach from PR #2521.
What changed
backends/pandas/container.py): reverts the in-placecheck_obj.drop(..., inplace=True)mutation from fix(pandas): propagate a column's own drop_invalid_rows through DataFrameSchema #2521; the filtered frame returned by adrop_invalid_rowscomponent is now threaded through the container's localcheck_objbinding instead — the caller's frame is never mutated.head/tail/samplesubsampling because a dropped subsample cannot be mapped back onto the full frame (matching the earlier pandas behavior).The abstract method signature
is unchanged — no widening of the return type, no fragile special-casing in the
validate()loop.Each drop component is validated once for error collection (in
run_schema_component_checks) and once to recover its filtered frame; schema-level errors from the second pass are ignored because they are already handled byrun_schema_component_checks.Backend scope
Column-level
drop_invalid_rowsis implemented in the pandas, polars, pyarrow (via narwhals) backends — those are covered and now fixed.pyspark,xarray,tensordictdo not expose column-leveldrop_invalid_rows. The ibis API exposes the flag but the ibis backend componentvalidate()has no drop path (it only raises), so implementing nested drop there is a separate feature, not a propagation fix — left out of scope.Tests
Regression tests for pandas, polars, narwhals, pyarrow assert a nested
drop_invalid_rowscolumn keeps dropping its failing rows (propagating to the full output row across all columns) even when theDataFrameSchemaitself does not setdrop_invalid_rows, plus a pandas guard that the caller's frame is not mutated.Backward compatibility
No existing tests were modified except removing #2521's in-place mutation (replaced with the non-mutating propagation) and adding new regression tests. User-visible outputs are preserved: the #2521 pandas regression test still passes unmodified. The only change is removing reliance on mutating the caller's frame, which is an internal implementation detail, not part of the public contract.