Repository navigation
Incorrect behavior of arithmetic operations between time values #12190
Description
Describe the bug
Consider the following query:
DataFusion CLI v41.0.0
> select ('2024-08-27T08:21:27Z'::timestamp + interval '1 day' - interval '1 second' - interval '1 day');
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Utf8("2024-08-27T08:21:27Z") + IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 1, nanoseconds: 0 }") - IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 1000000000 }") - IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 1, nanoseconds: 0 }") |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 2024-08-29T08:21:26 |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.005 seconds.
While the expected result should be 2024-08-27 08:21:26.000000 .
When I add parentheses to the query, the results are what I expect:
> select ('2024-08-27T08:21:27Z'::timestamp + interval '1 day' - interval '1 second') - interval '1 day';
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Utf8("2024-08-27T08:21:27Z") + IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 1, nanoseconds: 0 }") - IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 1000000000 }") - IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 1, nanoseconds: 0 }") |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 2024-08-27T08:21:26 |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.002 seconds.
The issue seems to happening when you have multiple (more than 2?) operands in arithmetic. e.g.:
> select ('2024-08-27T08:21:27Z'::timestamp + interval '1 day' - interval '1 day' - interval '2 day');
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Utf8("2024-08-27T08:21:27Z") + IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 1, nanoseconds: 0 }") - IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 1, nanoseconds: 0 }") - IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 2, nanoseconds: 0 }") |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 2024-08-29T08:21:27 -- Should be 2024-08-25 08:21:27.000000 |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.001 seconds.
When you have two operands, the results seem to match what you'd expect:
> select ('2024-08-27T08:21:27Z'::timestamp + interval '1 day' - interval '2 day');
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Utf8("2024-08-27T08:21:27Z") + IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 1, nanoseconds: 0 }") - IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 2, nanoseconds: 0 }") |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 2024-08-26T08:21:27 |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.002 seconds.
To Reproduce
You can use the same queries mentioned above.
Expected behavior
Mentioned above as well
Additional context
No response
Activity
tshauck commentedon Aug 29, 2024
I don't know the root cause, but I'd just add it seems to to where there's subtraction. E.g.
works, but
doesn't. I wasn't able to find a purely add case that doesn't produce the correct answer (though doesn't necessarily rule it out).
Abdullahsab3 commentedon Aug 30, 2024
Just an educated guess, but could it be that the arithmetic operators are not passed correctly from the parsed AST, or that the AST is not constructed correctly? Not entirely sure, but it seems that the first arithmetic operator is always the one that is applied on the entire expression. For example:
Minus is applied on all of them. I think this is the result of
select interval '2 day' - interval '1 day' - interval '1 day';.Another example:
which seems to be of similar behavior. The minus is applied on all of them, even transforming the latest minus into a plus. I think that's the result of
select interval '2 day' - interval '1 day' - interval '1 day' - interval '5 day' + interval '3 day';Another weird example with a plus:
The results seem to correspond with
select interval '5 day' + interval '1 day' - interval '2 day' + interval '4 day';Abdullahsab3 commentedon Aug 30, 2024
Just found something that could be interesting. I think it's a bug in the AST and the
precedenceorder of operators.I tried explicitly defining the precedence/order with parentheses for the same queries above (trying to mimick how the precedence between the operators is preserved):
Precedence from right to left (which is not correct)
This corresponds with the wrong results that I get from Datafusion:
Precdence from left to right (which is what I think is what should be happening):
The results of these queries match the expected results:
tshauck commentedon Aug 30, 2024
I think the examples have
select (interval '2 day' - interval '1 day') + interval '1 day';backwards?Otherwise, there's a potentially related example: apache/datafusion-sqlparser-rs#1345
findepi commentedon Sep 5, 2024
Indeed, this is parsing problem.
I modified
datafusion/sql/tests/cases/plan_to_sql.rsto include this:DFParserjust wrapssqlparser::parser::Parser, just we will need to fix this in https://github.com/sqlparser-rs/sqlparser-rsAbdullahsab3 commentedon Sep 6, 2024
I think apache/datafusion-sqlparser-rs#1398 fixed this issue.
findepi commentedon Sep 6, 2024
thank you @samuelcolvin and @alamb
alamb commentedon Sep 9, 2024
Nice! I expect the next release of sqlparser to be released sometime this week and thus the change will be available in DataFusion as well: apache/datafusion-sqlparser-rs#1384
alamb commentedon Sep 11, 2024
sqlparser-rs 0.51.0 is now available on crates.io: https://crates.io/crates/sqlparser/0.51.0 🎉
alamb commentedon Sep 12, 2024
I believe this will be fixed by #12222 from @samuelcolvin