Repository navigation
Test DataFusion 44.0.0 with Sail #13855
Description
Activity
Take
Smooth Sailing testing commit
5d563d9from DataFusionmainbranch.The following test reports run tests on the PR branch and compares them against the tests from the Sail
mainbranch:Will re-test once there is a release candidate.
Reacted by Andrew LambSmooth Sailing testing commit
5d563d9from DataFusionmainbranch.The following test reports run tests on the PR branch and compares them against the tests from the Sail
mainbranch:* [DataFusion 44.0.0 Pre-Release Testing lakehq/sail#335 (comment)](https://github.com/lakehq/sail/pull/335#issuecomment-2556353768) * [DataFusion 44.0.0 Pre-Release Testing lakehq/sail#335 (comment)](https://github.com/lakehq/sail/pull/335#issuecomment-2556416669)After a brief review of the errors I suspect there may be something up with lists and structs but I would have to see the actual logic being tested to know for certain. Many of the errors are expected - functions not existing, etc.
After a brief review of the errors I suspect there may be something up with lists and structs but I would have to see the actual logic being tested to know for certain. Many of the errors are expected - functions not existing, etc.
I feel like we have had several bugs / issues with structs / lists the last few releases so it would be amazing if we could figure out what is going on before the release. @shehabgamin is this something you can help with?
After a brief review of the errors I suspect there may be something up with lists and structs but I would have to see the actual logic being tested to know for certain. Many of the errors are expected - functions not existing, etc.
I feel like we have had several bugs / issues with structs / lists the last few releases so it would be amazing if we could figure out what is going on before the release. @shehabgamin is this something you can help with?
@Omega359 @alamb For sure, I will gladly take a look! If there are issues with lists and/or structs, it likely wouldn't be due to changes in the 44.0.0 pre-release that I tested, though. The Sail main branch and the PR branch that is testing the pre-release have the exact same errors (with the exception of slight rewording of some error logs).
This leads me to believe that any issues with lists and/or structs would likely be preexisting. I will take a close look at all of the errors, but, with that in mind, should we scope the issues into a separate GitHub issue if we do find errors? Also @Omega359, let me know if any errors in particular caught your eye.
This leads me to believe that any issues with lists and/or structs would likely be preexisting. I will take a close look at all of the errors, but, with that in mind, should we scope the issues into a separate GitHub issue if we do find errors? Also @Omega359, let me know if any errors in particular caught your eye.
That would make sense. Glad to hear the errors aren't new
Smooth Sailing testing commit
073a3b1from DataFusion Release Candidate branch (https://github.com/apache/datafusion/tree/073a3b110852f97ccb7085ce4bfd19473b8a3f4f).The following test reports run tests on the PR branch and compares them against the tests from the Sail main branch:
Note: Additional errors in
Test 2are expected. These occur because code using thedatafusion-functions-jsoncrate (fromdatafusion-contrib) had to be commented out. This was necessary since the crate depends on DataFusionv43, while the current release candidate usesv44. This wasn't an issue inTest 1because the commit frommainhadn't yet been updated to versionv44.Closing this issue now, but please feel free to re-open if necessary @alamb.
Also, @Omega359 please let me know regarding our earlier discussion (linked below). I haven't taken a look yet because of the holidays, but will create a new issue if I discover anything.
#13855 (comment)Happy Holidays!
Reacted by Andrew LambThank you very much @shehabgamin for the help and assistance. I love it when incentives are aligned like this
Reacted by Shehab AminAlso, @Omega359 please let me know regarding our earlier discussion (linked below). I haven't taken a look yet because of the holidays, but will create a new issue if I discover anything. #13855 (comment)
Thanks for reminding me about this - I've added it to my TODO list and I'll do my best to try and get to this in the next week.
Reacted by Shehab Amin
Is your feature request related to a problem or challenge?
We are trying to make sure the upgrade experience for DataFusion is better and to do so we are hoping to improve the testing of major downstream projects
For version 44.0.0 we would like to test with Sail prior to release.
Describe the solution you'd like
I would like to create a PR to test DataFusion Sail with a pre-released version of DataFusion (pinned to the main branch)
Describe alternatives you've considered
No response
Additional context
lakehq/sail#335