Repository navigation
[DISCUSS] More extensive pre-release testing #13661
Description
Activity
- changed the title
[-][DISCUSS] More deliberate pre-release testing[/-][+][DISCUSS] More extensive pre-release testing[/+]on Dec 5, 2024 BTW I often create WIP PRs to test releases of sqlparser-rs and arow-rs with DataFusion before the release
For example:
This caught at least one regression before release recently (apache/datafusion-sqlparser-rs#1556, fixed by @goldmedal 🙏 )
Reacted by Piotr FindeisenI would like to improve the testing / release process for DataFusion releases to reduce the number of regressions found after release.
I agree with the goal, but i am concerned about the cost/overhead involved. We don't have infinite bandwith at disposal.
Additionally, orchestrating testing across several downstream projects creates new problems we didn't have before- which downstream projects can stop the release train? what kind of problems are able to stop the release train?
- this is especially important question for closed-source downstream projects
- how long are we willing to wait for external teams to report back? Downstream project maintainers will obviously be willing and motivated to help, but their availability cannot be assumed
So what if we focused instead on:
- improving testing within DF itself? if a downstream project is concerned about stability of feature X, they can contribute to improve test coverage for feature X (eg [DISCUSSION] More SqlLogicTest test coverage for queries, including join queries #13470)
- easy low-ceremony low-overhead (automated) releases. If we see a big problem / regression after a major release, this can be patched on a maintenance branch. A maintenance branch can release daily without human intervention.
- which downstream projects can stop the release train? what kind of problems are able to stop the release train?
I agree with the goal, but i am concerned about the cost/overhead involved. We don't have infinite bandwith at disposal.
Additionally, orchestrating testing across several downstream projects creates new problems we didn't have beforeYes, this is true. I imagine an incremental rollout type approach -- where we start with one project (datafusion-python is a natural example, and maybe we can get delta-rs to help too). In my (likely naieve) thinking the downstream projects will be willing to help as they are directly affected
easy low-ceremony low-overhead (automated) releases. If we see a big problem / regression after a major release, this can be patched on a maintenance branch. A maintenance branch can release daily without human intervention
It is my understanding that the apache voting / approval process prevents automated builds
Creating a maintenance branch is also a compelling idea where we can focus on stability / shoring up test coverage 🤔
It is my understanding that the apache voting / approval process prevents automated builds
That's my understanding too, but i hope this process isn't nonnegotiable.
Processes are there to serve the project & the community after all, not the other way around.It is my understanding that the apache voting / approval process prevents automated builds
That's my understanding too, but i hope this process isn't nonnegotiable. Processes are there to serve the project & the community after all, not the other way around.
I think as long as we make it clear that nightly builds are not "official" releases from the ASF point of view, we could create / publish them. 🤔
That would work for me as long as these releases are the only once we publish. I would want automation for 'the releases' the people use. Especially if we have a maintenance branch, it would reasonable to release after every PR merge.
I don't know what problems the manual release process solves that cannot be solved with automated releases.anyway, did we hijack the thread?
anyway, did we hijack the thread?
Yeah, we should probably file a separate discussion about more frequent releases if we want to pursue that option
@alamb I did some personal testing with Cargo mutants. We can have a manually runnable pipeline, or something scheduled every day, that could run mutation testing and inform us about the weakness of our tests
I am happy to issue a PR if you think it's a good idea
@alamb I did some personal testing with Cargo mutants. We can have a manually runnable pipeline, or something scheduled every day, that could run mutation testing and inform us about the weakness of our tests
I am happy to issue a PR if you think it's a good idea
I am not sure what mutation testing means -- I would be happy to check out a PR that had such a thing
As for running, we could add it to the "extended" test maybe: https://github.com/apache/datafusion/blob/main/.github/workflows/extended.yml
Is your feature request related to a problem or challenge?
Up to now, when we have made DataFusion releases, we have mostly focused on validated that DataFusion's own unit tests have passed, (see dev/release/README.md) but haven't tested the upgrade with other downstream projects (like ballista, ray, InfluxDB IOx, etc) until after we have release the code
This results sometimes in downstream users finding issues after release. Some recent examples
Invalid comparison operation: Utf8 == Utf8Viewerror during LEFT ANTI JOIN #13510Describe the solution you'd like
I would like to improve the testing / release process for DataFusion releases to reduce the number of regressions found after release.
This would likely take the form of updating the dev/release/README.md
Describe alternatives you've considered
One idea mentioned by @Omega359 and @andygrove on #13525 (comment)
Additional context
No response