Repository navigation
fix: Recognize Parquet list<element: T> types in pa_to_feast_value_type - #6876
Open
joaquinhuigomez wants to merge 1 commit into
Open
joaquinhuigomez wants to merge 1 commit into
joaquinhuigomez wants to merge 1 commit into
Conversation
joaquinhuigomez
requested review from
ejscribner,
robhowley and
shuchu
and removed request for
a team
September 27, 2026 22:39
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #6876 +/- ##
==========================================
+ Coverage 47.66% 47.70% +0.04%
==========================================
Files 422 422
Lines 52396 52400 +4
Branches 7606 7606
==========================================
+ Hits 24973 24999 +26
+ Misses 25632 25612 -20
+ Partials 1791 1789 -2
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
joaquinhuigomez
force-pushed
the
fix/pa-list-type-element-naming
branch
from
September 28, 2026 21:33
a06f363 to
be57370
Compare
pa_to_feast_value_type only parsed the "list<item: T>" spelling. Parquet
files written by pandas/pyarrow read back as "list<element: T>", so schema
inference (feast apply) and write_to_offline_store raised
KeyError: 'list<element: string>' for Array(String), Array(Bytes) and
Array(UnixTimestamp) columns in a FileSource. The hard-coded
"list<element: ...>" fallback keys only covered some element types, and two
of them ("str", "bytes") are names pyarrow never prints. large_list<...>,
"not null" elements and element-named nested lists failed the same way.
Parse every list spelling with one pattern, recurse into the element type
as before, and drop the fallback keys. _parse_pa_type_str, which rebuilds
nested list types for write_to_offline_store, uses the same helper.
Signed-off-by: Joaquin Hui Gomez <132194176+joaquinhuigomez@users.noreply.github.com>
joaquinhuigomez
force-pushed
the
fix/pa-list-type-element-naming
branch
from
October 20, 2026 22:04
be57370 to
0ba5849
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
pa_to_feast_value_typeonly parsed thelist<item: T>spelling of a PyArrow list type. Parquet files written by pandas/pyarrow read back aslist<element: T>. So for aFileSourcecolumn typedArray(String),Array(Bytes)orArray(UnixTimestamp), schema inference infeast applyfails, and so dowrite_to_offline_storeandpush(..., to=PushMode.OFFLINE)(both go throughget_pyarrow_schema_from_batch_source). They raise:Repro:
Integer, float and bool lists only worked because #5118 hard-coded some
list<element: ...>keys. Its string and bytes keys (list<element: str>,list<element: bytes>) use names pyarrow never prints, and there was no timestamp key.large_list<...>,list<item: T not null>and element-named nested lists raise the sameKeyError.Changes:
(large_)list<(item|element): T( not null)>, now parses the list prefix. The recursion intoTand the rule that nested lists map toVALUE_LISTare unchanged. The hard-codedlist<element: ...>keys are removed._parse_pa_type_strinoffline_utils.py, which rebuilds nested list types forwrite_to_offline_store, uses the same helper.list<element: list<element: int64>>now becomeslist<list<int64>>instead of falling back tostring.Every type string that mapped before still maps to the same
ValueType, except the two keys pyarrow never produces.Which issue(s) this PR fixes:
No open issue; repro above.
Checks
git commit -s)Testing Strategy
Misc
Unit:
TestPyArrowListTypeStringsintests/unit/test_type_map.pycovers parametrized spellings forpa_to_feast_value_typeand_parse_pa_type_str, plus atmp_pathtest that writes Parquet withdf.to_parquet, reads the schema throughFileSource, and checks both the inferredValueTypeandget_pyarrow_schema_from_batch_source. 13 of the 18 new cases fail on master; the other 5 pin spellings that already worked.Manual: a local repo with a
FileSourceover adf.to_parquetfile, with string, bytes, timestamp, int, float, bool and nested list columns. On master,feast applywith inference,write_to_offline_store(file and DuckDB offline stores) andpush(to=PushMode.OFFLINE)(DuckDB) all raise theKeyError. With this change they succeed, and materialize plusget_online_featuresreturn the lists.