Generate kinds concurrently or in parallel

[ahal]

Currently we generate each kind one after the other, which can take awhile (we're approaching 5 min in Gecko now). Instead, we should generate the tasks for kinds in parallel.

We'll have to take kind-dependencies into account and come up with a way to synchronize workers such that we only generate a kind one all of its dependencies have also been generated.

[bhearsum]

Do we have any idea how much this would speed up Gecko taskgraph generation (I presume that would be the longest one).

[ahal]

No I'm not sure :(. It would be a lot of work to get the full benefit out of asyncio (e.g, we'd need to move from requests to aiohttp, use async file system access, figure out how to yield control when accessing the build system, etc). All this work could happen incrementally over time, but it's hard to say what the end result would be.

[JohanLorenzo]

I love async and I could see the benefits of it in scriptworker for instance. Like you raised in the team's meeting @ahal, going async may raise the entry barrier even higher. A year ago, I checked how many people contributed to the taskcluster folder in Gecko. The numbers were amazing: 300 contributors in total and about 30 core contributors.

However, outside of Gecko, the numbers aren't that high. I get the feeling Releng owns many taskgraph configuration. If next year, we want to be more hands-off, I don't think we should go async yet.

For reference, the Android decision task takes 5 minutes too. It's mainly because we shell out to time-consuming process (bug 1788606). Going async wouldn't help this case.

[ahal]

It might still be possible to support both async and sync transforms. The problem is that async transforms return async generators, which can't be iterated over in a synchronous transform.

We could either say "no sync transforms allowed after any async ones" (I don't love this). Or we could convert the async generator to a list prior to passing it into a sync transform. This would negate the benefits of using a generator, but maybe that's a fair trade-off.

[ahal]

If we supported both, I'd envision all core transforms would be async (so contribution barrier would be higher there still), but then consumers could choose whether to bother with async or not.

It's also worth noting that we're not talking hugely more difficult here. Basically just s/def/async def and s/for/async for.

But I agree we can sit on this for now.

[escapewindow]

It might still be possible to support both async and sync transforms. The problem is that async transforms return async generators, which can't be iterated over in a synchronous transform.

We could either say "no sync transforms allowed after any async ones" (I don't love this). Or we could convert the async generator to a list prior to passing it into a sync transform. This would negate the benefits of using a generator, but maybe that's a fair trade-off.

I think @djmitche took an approach of creating synchronous wrappers to all outward-facing async functions, e.g.

async def do_stuff_async(*args, **kwargs):
  ...

def do_stuff_sync(*args, **kwargs):
    return asyncio.run(do_stuff_async(*args, **kwargs)

which allows for all of the library to be async, but for downstream consumers to use functions synchronously.

[ahal]

I also played around with generating kinds in parallel rather than concurrently:
https://phabricator.services.mozilla.com/D239805

It didn't seem to improve things at all. I'm not sure if this is overhead from serializing / deserializing the loaded_tasks, or process overhead or what. I even made an optimization to only pass in kind_dependencies_tasks rather than all tasks down to the worker processes. But this only got us back to par.

[ahal]

Ok, I found a bottleneck in my logic, after fixing that I can shave off about ~10s of generation time on my machine. That's about a 23% improvement with ./mach taskgraph full --fast and a 10% improvement with ./mach taskgraph full. This number makes sense to me because tests are by far the long poll here, so we can only improve by however long it takes to generate non-test tasks, which are typically quite fast.

Still, a 10s improvement is nothing to sneeze at for the developer sitting at their terminal waiting for their ./mach try to finish...

[bhearsum]

Not to move tho goalposts, but this also makes me wonder if there are ways we can (safely) concurrently process entries in an individual kind? Offhand, I don't think there are any dependencies or data passed from one task in a kind to the next? (There's a ton of tasks in Gecko's test kind, so that would presumably be pretty impactful if it's possible.)

[escapewindow]

I bet there are test harnesses/flavors that don't depend on each other that might be able to be split out. There might be side effects to this, but hopefully detectable via a taskgraph diff (and maybe a search of the test code to make sure there aren't any unexpected hardcodes?)

Though I'm now thinking mach might have some test kind hardcodes... maybe create a TEST_KINDS tuple that specifies which kinds are tests.

[djmitche]

Is the bottleneck IO or CPU?

[ahal]

IO, reading the moz.build/test manifest files to generate a list of all tests and waiting for the ML service to say which tests to run are the two main things. Neither of these things are enabled with --fast, so they don't affect people waiting at the cli too much.

I think the holy grail here is going full asyncio and using Python 3.13t (free threaded aka no GIL). That's probably still a few years away from being stable enough for our use case (but maybe worth trying just to see?). But then we wouldn't need to worry about explicitly parallelizing things within a kind. It would just be automagic.

So personally I think steering towards that is probably the best bet rather than bothering to parallelize within a kind.. Though if we were going to parallelize within a kind, doing so across each tasks-from (e.g taskcluster/kinds/test/mochitest.yml) would be an obvious hook point to do it.