32 codegen units may not always be better at -O0

[alexcrichton]

Continuing discussion from #44853 (comment)

[alexcrichton]

@mersinvald unfortunately the perf files you mentioned I can't read locally, and the perf diff output isn't as illuminating as I hoped it would be.

I wonder, could you try investigating if one crate in particular takes much longer? Maybe take some of the crates that take longest to compile (sorting the rlibs by size is typically a rough measurement for this) and see how long they take?

I'm hoping that this is one crate hitting an accidentally pathological case, but it may also just be a case where everything slows down with 32 codegen units (even though we don't intend that).

One other possiblility may be the linker? (like ld) That should show up though as the final crate taking much longer to finish in 32 cgus than 2 in theory.

[crumblingstatue]

I tested compiling regex with different number of codegen units on my 2 core Pentium E2200.
Here are the results:

1:  33.4
2:  28.76
4:  28.90
8:  29.57
16: 29.97
32: 31.0

It gets a decent win with the number of codegen units equal the number of cores, but gets increasingly slower with more than that.

Here are the results for rustfmt. These are a bit more wild. It's actually faster on 4 units than 2, but gets increasingly slower after that, up to the point where 32 units is actually slower than 1 unit.

1:  177.31
2:  172.14
4:  167.91
8:  170.75
16: 175.80
32: 179.49

[retep998]

Testing with winapi 0.3 on a Ryzen 7 1700X.

codegen-units	Time
1	15.17
2	14.63
4	15.18
8	15.44
16	14.68
32	14.89
64	15.22
128	15.22
256	15.92
512	17.34
1024	20.24
2048	27.12
4096	40.74
8192	69.23

[oyvindln]

There is a pretty noticable change in compile times on perf.rust-lang.org with the commit that changed the default number as well.

[alexcrichton]

I'm going to tag this as a regression until we can figure out what the spike is

[alexcrichton]

Ok looking more into the data from perf.rust-lang.org. We're measuring instructions executed and the "cpu-clock" measurement. Instructions certainly don't have much bearing on wall time (but serve as great indicators of when to look for a regression) and apparently cpu-clock is not the same as wall time. I believe the cpu-clock stat is the total sum of all threads on all cpus.

I'm trying to rationalize the numbers we're seeing and it's not quite making sense. The regressions shown on perf.rust-lang.org imply to me that if I pin the process to one CPU we'll see that much increase in compile time, but I see virtually no increase in compile time locally. In fact, if I pin cargo/rustc to one CPU it typically gets faster with more cgus presumably because the objects are a little quicker to create overall.

Basically I'm having a lot of difficulty reproducing any slowdowns here. Assistance investigating why things are slowing down as opposed to just "things are slowing down" would be much appreciated.

[alexcrichton]

I believe these are all the reported regressions from this change

[alexcrichton]

Some more investigation

I found locally I was able to reproduce some slowdowns with the syntex_syntax crate:

cgus	1	2	4	8	16	32
wall time	32.9	24.9	22.8	21.4	22.0	21.8
cpu-clock	33064	35218	40584	44581	48542	51788
1 core wall time	34.3	35.2	36.4	38.1	40.3	41.7

Here "wall time" is the wall time it took to compile the syntex_syntax crate on an 8 core cpu. The "cpu-clock" is the statistic reported by perf state, and "1 core wall time" is the amount of time the compilation takes when pinned to one cpu (taskset -c 0).

There's a clear tend here that parallelization is helping but also taking up more CPU time. Namely increasing the number of codegen units is not purely dividing the work, it's also increasing the total amount of work being done. This becomes particularly noticeable with more codegen units limited to one core on this big crate.

So on one hand there's certainly some overhead from more codegen units. We're shuffling more files around (not copying, just working with), spawning more threads, more messages, more synchronization, etc. I don't think that these factors would cause such a drastic increase in compile times, however. Instead what I think may be happening here (that I have just now remembered) is how we treat #[inline].

@michaelwoerister correct me if I'm wrong, but I believe that we'll inline any #[inline] function into all codegen units instead of just one. This means that if you're using String you're probably going to get a copy of each of its functions in all of your codegen units. This to me seems like the root cause of the timing regressions here.

As a result my conclusion is that increasing the number of codegen units increases the amount of work the compiler does, and hence parallelism isn't always enough to recover this loss in the increase of work done. In the worst case with N codegen units you can probably do N times more work, and with less than N cores you'll notice the increase in compile times.

@michaelwoerister do you agree with the conclusions here? If so, do you think we could treat #[inline] differently in debug mode? Is it worth it to solve this? Can you think of another possible solution?

I'm personally hesitant to set the number of codegen units based on the amount of cores on the machine. I think that may return optimal performance (perhaps 2x the number of cores cgus) because the extra work needed for each codegen unit I think should come out in the wash because of parallelization. This, however, would make builds across machines nondeterministic if they've got different numbers of cpus, forcing deterministic compiles to explicitly specify the number of codegen units.

My personal preference for a solution right now is:

• Don't inline #[inline] into all codegen units in debug mode, only inline into one codegen unit.
• Reduce the number of codegen units from 32 to 16 here if we continue to see problems. I think most machines compiling Rust today have fewer than 16 cores so we should still get some benefit of using as much as possible, while also this should help reduce any extra costs from extra work needed for each codegen unit.

[michaelwoerister]

@alexcrichton, Yes I think your assumptions and conclusions are correct. I also think that #[inline] functions are the main source of work duplication here. There's also debuginfo, which cannot be shared between codegen units, but that is usually rather cheap afaik.

Don't inline #[inline] into all codegen units in debug mode, only inline into one codegen unit.

We probably mean the same thing here: I would suggest treating them the same as regular polymorphic functions:

• Instantiate only "on-demand", and
• have exactly one instance and link to that.

That might require some special handling in the symbol hash to avoid conflicts between crates, just as we do for generic functions at the moment:

rust/src/librustc_trans/back/symbol_names.rs

Lines 194 to 201 in ed1cffd

	// If this is an instance of a generic function, we also hash in
	// the ID of the instantiating crate. This avoids symbol conflicts
	// in case the same instances is emitted in two crates of the same
	// project.
	if substs.types().next().is_some() {
	hasher.hash(tcx.crate_name.as_str());
	hasher.hash(tcx.sess.local_crate_disambiguator().as_str());
	}

If performance turns out to be unbearable, we maybe could have a heuristic to do some inlining:

• if the function body is small, and
• if the function does not call other inline functions.

Reduce the number of codegen units from 32 to 16 here if we continue to see problems.

Seems like a sensible approach.

Other solutions?

I wonder if ThinLTO would allow us to completely get rid of duplicating bodies of inline functions across codegen units during trans. That would be awesome for incremental compilation and it would also help in the non-incremental case.

[alexcrichton]

Some excellent news!

I implemented the strategy of "only inline functions into one codegen unit" and the updating timings I get are:

cgus	1	2	4	8	16	32
wall time	30.7	25.2	21.0	19.0	18.4	18.7
cpu-clock	30864	31387	31876	32525	34341	35370
1 core wall time	31.2	30.8	30.8	30.6	30.7	30.6

In other words, I think this indeed makes a huge dent in the issue of "more cgus causes rustc to do more work" problem. I'm going to send a PR shortly implementing this.

@michaelwoerister I also think that you're correct in that ThinLTO should eliminate the need for inlining functions into all codegen units unconditionally. Just gotta land ThinLTO and test it out first :)