Repository navigation
rustbuild: Dist on cross builds produce way too many tarballs #38365
Description
Activity
A closer look at the list shows we're certainly building target-agnostic components over and over, which is everything! (I incorrectly assumed the std is different and require full cross product, but upon looking at the dist directory layout at static.rust-lang.org I immediately realized it's not the case.)
So we only have to skip the execution of steps that have
s.host != s.targetfordistrules? It seems creation of the steps are not as straightforward to work around. I could prepare a patch for this.I think that's along the right lines, yeah. I'm not sure if that's precisely the strategy that we want, but it sounds about right.
In general the build triple builds everything, so we should only package from there. I think for packaging the rustc package, however, there may be some snags. In any case would be great to test out!
FYI it seems that the
make distchecksteps on buildbot are also affected by the same symptom, which is very awful as here every target takes ~3h instead of a few minutes... Just check out the buildbot waterfall. I could integrate a fix into #38468 hopefully before the next nightly build triggers.(Or you could directly push into my branch instead, the fix would just be an additional test for
kind == Kind::Test. It's 3 am here and I need some rest, maybe couldn't get up in time. I'd personally appreciate seeing this land just after the i128 PR as a quickened nightly update benefits everyone.)@alexcrichton Take this build for an example.
distcheckafter #38367 essentially equals to adistfollowed by atest; you can see in the logs where although thediststeps are taking somewhat longer, the real waste of time comes in the form of multiple compilations of LLVM and multiple runs of the full testsuite, but the host list asconfigured is as short as 2 hosts! Therefore I now believetestshould be covered too.Ah yeah distcheck takes forever in its own right, but we hope to fix that soon!
The dist step on cross builds takes over half an hour, and that's a crazy amount of time for something which should take at most 5 minutes.
The list of tarballs created is massively duplicated. Namely we create around 60 source tarballs (!!). Looks like we're also packaging up the standard library dozens of times for the same target.
My guess as to what's happening is that by default rustbuild is doing a cross product of (host x target) and doing all the builds. We should skip a massive number of them for distribution and ideally not even put them into the step list at all!
Note that this isn't a fire at the moment as everything's just overwriting what was previously created, but this sounds like something that could become a fire!
Should also be a relatively easy bug to knock out for anyone who wants to become familiar with rustbuild!