Repository navigation
[MNG-6265] Controlling reactor order of modules #7598
Description
August Shi opened MNG-6265 and commented
We have a use case for controlling the reactor order in Maven, specifically for running tests on Travis. The common pattern is to run "mvn install -DskipTests; mvn test". After modules are installed (in the default order), the modules can be run in any order during testing, e.g., running the failed modules first, as in -Dsurefire.runOrder=failedfirst for Surefire.
Would this feature be useful for others? I am attaching a draft patch that implements this feature by modifying DefaultMaven.java. I see no clean way of implementing this feature through an extension, unless some new callback, e.g., afterProjectsOrdered, is implemented first. Would it be worthwhile to improve this draft?
One can test the feature by applying the patch, building Maven, and running tests with the modules in a random order like this:
$ mvn -DdistributionTargetDir=$(pwd)/apache-maven-3.5.x-SNAPSHOT clean install -DskipTests
$ apache-maven-3.5.x-SNAPSHOT/bin/mvn test -Dmaven.project.order.file=<(ls -d maven* | shuf)
Affects: 3.5.0
Attachments:
- p.patch (3.55 kB)
Remote Links:
1 votes, 4 watchers
Activity
jira-importer commentedon Jul 28, 2017
Karl Heinz Marbaise commented
Tests are running on each module and not on a reactor base or do you mean the tests in different modules? Furthermore what do you mean by
mvn install -DskipTests; mvn test? Is the reactor order different from themvn install..than frommvn testafter ? The order of running tests within the modules can be controlled by use the option-Dsurefire.runOrder...as you mentioned...Do you have defined correct dependencies between your modules in your reactor ? It sounds like you didn't? Cause the reactor order is defined by the dependencies...?Can you give a concrete example ? What is the real problem you are trying to solve?
jira-importer commentedon Jul 28, 2017
August Shi commented
Thanks for replying so quickly. Our overall goal is to find a failing test as soon as possible (if any test fails). We need to control the order in which modules are tested from the reactor base (and we further control the order in which classes are tested inside each module). Our use case (on Travis) always has two steps: first, "mvn install -DskipTests" compiles and installs the modules without running tests (and this is done in the reactor order, following the dependencies), and then "mvn test" runs the tests after install is successful (maybe a more correct way to express our use case is "mvn install -DskipTests && mvn test"). Since everything is installed in the first step, the modules can be run in any order in the second step, so we want to control that order for the second step.
As an example, assume a project has modules A, B, and C. Based on the dependencies, the reactor order is [A, B, C]. In one run, "mvn install -DskipTests" builds modules in that order [A, B, C] and "mvn test" runs tests in that order [A, B, C]. However, if tests fail in module C, then for our next run, we still run "mvn install -DskipTests" in the order [A, B, C], but we want to control the order to run module C before others in the second, testing step, e.g., run "mvn test" in the order [C, A, B]. With this new order, if tests fail again in C, the build terminates faster, skipping testing in the remaining modules. This feature cannot be achieved with -Dsurefire.runOrder, because Surefire only controls the order of tests within one module, while we would like to control the order across several modules.
jira-importer commentedon Jul 29, 2017
Robert Scholte commented
I hope we can do this much smarter. There are already some options and there are some feature requests which can help to improve this.
There's this option
-rf / --resume-from <arg>where you can say you want to resume from C. This has one disadvantage: it cannot pick up results earlier in the reactor. To do so you now need to add-am / --also-make, otherwise they will be pulled from the local repository (but you were doing this anyway).There's also the option
-fae / --fail-at-end. If you want to know all failing projects ASAP, this might be an option as wel, you don't have to wait for the first module to be fixed.Some other options outside the Maven project:
IIRC Karl Heinz has been working on an extension which detects scm changes compared to the previous build and can which will only build the minimal set.
Takari has a takari-smart-builder which should also help to let C build ASAP.
"Finding a failing test as soon as possible" is a valid requirement, but IMHO should work out-of-the-box instead of trying to manipulate the order of modules.
In the end I would really prefer that
mvn testormvn test -rfdoes exactly that.jira-importer commentedon Jul 29, 2017
August Shi commented
We are familiar with most of the options you listed, but none of them does what we want: we want to run all the modules (not only some) but run them in a prioritized order (that may differ from the reactor order). Note that supporting our feature may not require big changes in the core. If just one callback is added, we can implement the feature as an extension and not affect the core further.
In our example, suppose that we run "mvn install -DskipTests" first (as commonly done on Travis), and then we want to run "mvn test <something>" to run tests from all modules A, B, and C but in the order [C, A, B]. For example, we may have changed code in all three modules, but we expect that C is the most likely to fail.
This is what the default "mvn test" conceptually (not exactly) does:
mvn -pl A test
mvn -pl B test
mvn -pl C test
This is what our patch "mvn test -Dmaven.project.order.file=<(echo C; echo A; echo B)" would allow:
mvn -pl C test # this is the key difference in order
mvn -pl A test
mvn -pl B test
If we run "mvn test -rf C":
mvn -pl C test # only runs C, does not run A and B
If we run "mvn test -pl C -am" (assuming C depends on both A and B):
mvn -pl A test
mvn -pl B test
mvn -pl C test # still runs C last
The option -fae does not change the order (and could not run C first) but continues even after failure. Also, a combination of -rf and -am, i.e., "mvn test -rf C -am", ends up the same as "mvn test -rf C", and the dependencies of C are not built.
That is indeed a useful feature. There is actually another Maven plugin we are aware of that is similar:
https://github.com/vackosar/gitflow-incremental-builder
However, that plugin only selects to run some modules, and it could end up selecting to run all modules in our example. It (1) does not prioritize in what order to run them and (2) does not run them all (although we may not even need to run all if the selection is safe). In the end, though, we still want to prioritize the order of modules.
That is indeed another useful extension for parallel builds. Some researchers have implemented similar parallelization for testing in Maven:
http://jonbell.net/preprint/releng15poster.pdf
http://jonbell.net/preprint/releng15abstract.pdf
Unfortunately, these features still do not prioritize the order of modules to allow for [C, A, B].
I don't see how that would work without manipulating the order of modules, but I may be missing something.
We propose a really small change mvn test -Dmaven.project.order.file=${file_with_order_of_modules}
jira-importer commentedon Jul 30, 2017
Robert Scholte commented
I was looking for options which would help to get the same reliability with the options that are already there. And I would prefer automatic solution when possible. And I am quite critical with every new option added: is it really required? And the number of lines changed/added is no criteria.
IMHO I don't think that order is really the key issue here. If C breaks the build and you only change C, then there's no need to build A and B again, unless you made changes there as well. So to me trying to run C, A, B is a sign there's something else wrong.
There is already one argument which can contain multiple projects:
-pl / --project-list. Right now AFAIK its order has no effect, but I can imagine we could make Maven respect that order, as long as the graph-order is respected! In other words: if A, B and C have no relations, there order can be changed. If C depends on A and B, it'll still be ABC.jira-importer commentedon Jul 30, 2017
August Shi commented
We understand your concern about "feature creep", especially that you mentioned user-facing options. However, for our purposes, we may not need a new command-line option to the core as in the proposed draft patch; adding just one API callback, say "afterProjectsOrdered", similar to "afterSessionStarted" or "afterProjectsRead", would suffice.
In our use case A and/or B also changes. For example, an integration test in C may have failed, requiring to change not only C but also A and/or B. We then need to run multiple modules anyhow, and we just want a different order for "mvn test" -- compilation (with "mvn install -DskipTests") still obeys the reactor order.
That would be already a step forward, although that would change an existing option, and some users may have relied on the order they get now. Adding a new option, "-opl", might be better, even if it respects both the given order and the graph. However, our use case really needs a way that does not respect the graph, only for testing purposes (after everything is compiled).
jira-importer commentedon Jul 30, 2017
Robert Scholte commented
If you don't want to respect the graph order, then a custom implementation seems to be the best solution. I'm not sure if
-b,--builder <arg>is sufficient enough or if that also requires an API change to be able to sort the projects.jira-importer commentedon Aug 1, 2017
August Shi commented
Thank you for pointing out the --builder option. We implemented a custom Builder and it seems to work for our use case; we can run modules for testing in an order different from the reactor order. The printing of modules is still in the reactor order, but it is something we can work on later.