Develop/Document multi-level parallelism policy

[rrnewton]

I can see that stack test defaults to parallel builds. But that refers to parallelism between different test-suites, right? In the case of both builds and tests (with test-framework or tasty) there's the issue of "inner loop" parallelism within each build or test target. I suppose we can't parallelize the tests until we have some way for the build-tool to know it's a tasty/test-framework suite, not just exitcode-stdio-1.0, but that still leaves build-parallelism. Does Stack currently pass -j to GHC?

I haven't seen comprehensive benchmark numbers, but a small value like -j2 or -j4 for GHC offers some benefit. The current implementation is not scalable however (for example, I haven't seen anything good come of -j32).

Nested parallelism of course raises the issue of N * N jobs being spawned for N cores. As long as its quadratic oversubscription and not exponential, I think this is not that big a problem for CPU usage, but it very often can create problems with memory usage (or hitting ulimits / max # of processes).

There are several related cabal issues, but it's a little hard to tell the current status with them spanning a lot of time and various merged and unmerged pull requests:

• build multiple packages in parallel haskell/cabal#440 -- mentions how Gentoo uses an estimate of current load to gate creation of new jobs
• Parallelise cabal build over modules haskell/cabal#976
• 'install -j': allow limiting the number of parallel linker invocations  haskell/cabal#1572 -- some talk of using OS semaphors to manage shared compute resources. A draft of the related code appears here, in @23Skidoo's unmerged work.
• -j should build package components in parallel haskell/cabal#2623

[snoyberg]

I'm seeing about three different things being raised here:

• Asking GHC to build a single package in parallel
• Telling a test suite to run its test cases in parallel
• Running multiple test suites from a single package in parallel

stack does none of these right now. The only thing stack does in parallel is process individual packages in parallel. Does that answer your question?

As to what should stack be doing... I don't see a downside to passing -j to GHC when available. I'd rather avoid running test suites in parallel, but there's really no strong reason for that. I don't see how stack can have any impact on the insides of the test suite itself, since that's entirely up to the test framework.

[borsboom]

One tricky thing is deciding how many threads GHC should be running if stack is running multiple builds (i.e. if you have 8 CPUs and stack is running 8 builds, each GHC shouldn't be running 8 of its own threads). Simplest might be to only pass -j if stack is only building a single package.

[rrnewton]

Given that neither stack nor cabal nor GHC has a notion of global resource management on the machine, I guess in the medium term what I'd like is enough knobs to experiment with this.

For example, it would be great to be able to "turn it up" to max parallelism -- parallel packages, parallel targets within a package, parallel modules within ghc --make plus parallel tests. And then do some benchmarking to see what speedups and memory usage look like.

We can already vary things pretty easily by generating .cabal/.stack files that pass the right arguments through to GHC and to the test-suites. I guess the critical thing on which to get help from stack itself is the third bullet @snoyberg mentioned -- multiple test-suites (plus profiling/non-profiling/documentation) in parallel within one package, which corresponds to haskell/cabal#2623.

By the way, as far as I know, no one's properly analyzed GHC builds from a parallel algorithms perspective. I.e. we need to profile the dependence graph and do a work/span analysis to figure out what is limiting our scaling. (We're working on a research project where we may be hacking on fine-grain parallelism inside certain GHC phases, but it only makes sense if there's a coherent picture for parallelism at the coarser grain too.)

In lieu of a GHC server mode (which has been discussed on various issues), we can't directly implement an inter-process work-stealing type policy that mimics the now-standard intra-process load balancing approach. But we can do what that Gentoo builder seems to be doing and simply delay some tasks to manage resource use. The more look-ahead or previous knowledge we have about the task DAG the smarter we can be in prioritizing the critical path.

[snoyberg]

I'm tempted to close this issue, and just add a link to my comment above to the FAQ. Any objection?

[rrnewton]

That's fine. Fixing this so that stack does something smart would be a big project not solved in a day, and that addresses the "document" part.

[snoyberg]

Added here: https://github.com/commercialhaskell/stack/wiki/FAQ#how-does-stack-handle-parallel-builds-what-exactly-does-it-run-in-parallel

[alexanderkjeldaas]

It would be great if this issue, since it's linked from the FAQ, tells me how to build at least my package in parallel using some ghc option, possibly specified in the cabal file or as an option to stack.

Alternatively, explicitly say that this cannot be done.

[mgsloan]

@alexanderkjeldaas No such flag necessary, stack will build your package in parallel with other things if it can.

Perhaps you mean having ghc build modules in parallel? Unfortunately in my experience this doesn't speed things up as much as I'd hoped. You can do --ghc-options -j5 (or whatever number).

[alexanderkjeldaas]

Yes that option is interesting - ghc seems to be able to use 6x the CPU and finish on exactly the sime time. Impressive!

[sjakobi]

Yes that option is interesting - ghc seems to be able to use 6x the CPU and finish on exactly the sime time.

Related GHC ticket: https://ghc.haskell.org/trac/ghc/ticket/9221

I did some simple timings on my machine (i3-2350M, 2 physical cores + hyperthreading) and always got the shortest build times with -j2 or -j3. The relative speedup varied a lot depending on the package, e.g. ~30% with vector-algorithms vs. ~10% with haskell-src-exts.

I was wondering how hard it would be to detect when stack doesn't use all of its build threads and in that case to pass -j2 or -j3 to the build jobs until all threads are used.
Build times would probably still be quite far from their optimum but I don't believe that this could result in build times that are worse than the status quo.

[Blaisorblade]

Related question (might need its own issue): how do I tell stack to set -j4 by default for itself, aside from ghc-options? I found nothing in http://docs.haskellstack.org/en/latest/yaml_configuration/#non-project-specific-config, http://docs.haskellstack.org/en/latest/faq/ or by googling.

Studying this issue suggests also builds are parallel, and stack's source suggests it defaults to -j $(getNumProcessors), already in 1.1.2, which sounds good (depending on answers to the other questions), and that this can be tuned through e.g. jobs: 4 in config.yaml.