Repository navigation
Bug with csv type inference #3174
Description
Describe the bug
❯ CREATE EXTERNAL TABLE test
STORED AS CSV
WITH HEADER ROW
LOCATION '../testing/data/csv/aggregate_test_100.csv';
0 rows in set. Query took 0.102 seconds.
❯ select * from test;
ArrowError(ParseError("Error while parsing value 14857091259186476033 for column 9 at line 3"))
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.
Activity
bezbac commentedon Aug 20, 2022
Hello 👋🏻.
I haven't contributed here before but checked out this issue because of its "Good First Issue" label.
Maybe you could help me figure out what the expected behavior is.
The cell's value inside the CSV file (
14857091259186476033) overflows the inferred typeInt64.So assuming the above query is supposed to work given the
aggregate_test_100.csvfile as input, I would suspect the column should be inferred asUInt64.To fix this, changes would likely be needed within the
arrowcrate, as the CSVinfer_schemafunctions utilizes thearrow::civ::reader::infer_reader_schemafunction.Do you have any direction on how to proceed?
alamb commentedon Aug 21, 2022
👋 @bezbac
I think looking into improving the inference in the arrow crate would be a good step
Another potential alternative to using UInt64 might be Decimal128 (though I don't want to mislead you too much here)
kmitchener commentedon Aug 21, 2022
There's also a TODO directly related to this that could be cleaned up as part of this PR -> https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/tests/sql/mod.rs#L624
alamb commentedon Sep 18, 2024
This still happens as of datafusion 42
CookiePieWw commentedon Sep 27, 2024
Hi, I took a look based on @bezbac's findings, and found that
arrow-csvuses regex expr to match strings and then infer ther types. The regex forInt64accept all possible numbers instead of numbers in ranges ofInt64, which caused that the inferred type actually cannot accept the strings.(Seems github cannot attach codes through links that does not belong to this repo, so I copied here)
One of the solutions is to change the regex expr for
Int64to match the range of it, but it seems to be a very complicate one. I told gpt to give me one but the regex expr's more than 300 chars, so I wonder if there're alternatives :)alamb commentedon Sep 28, 2024
I wonder if you could use a length check on the characters too (it wouldn't be perfect, but it might catch more)
As in you could bound the number of characters matched by the regexp to the number of characters in Int64::MAX (20 or so??)
CookiePieWw commentedon Sep 28, 2024
Thanks for your suggestion! But if we want to precisely match it, we have to iter through the prefix of Int64::MAX and then we got:
and so on...
Perhaps we can use a function for match strings here instead of direct regex here, which seems more flexible for me. If you agree on that, I'll try to submit a pr to
arrow-rsfor further discussions :)alamb commentedon Sep 28, 2024
That seems like a good idea to me -- with sufficient care it should also be significantly faster than a regexp based matching approach.
alamb commentedon Oct 2, 2024
I verified that this behavior now works well after upgrading to arrow
53.1.0due to apache/arrow-rs#6481 (thanks @CookiePieWw ❤️ )