feat(run-task): add git sparse checkout support - #1039
Open
ahochheiden wants to merge 3 commits into
Open
Conversation
ahochheiden
force-pushed
the
sparse-checkout
branch
from
September 11, 2026 04:20
7f6bc32 to
29d0422
Compare
…ronment The patterns travel in `<REPO>_SPARSE_PATTERNS` as a JSON list of native git patterns, so the cloned revision does not need to contain the profile. A JSON list because generic-worker on Windows sets task environment variables through a batch `set` line, which cuts a value at its first newline. The option and the variable have to come together, and a checkout is required for either. Patterns go through stdin so a name starting with a dash is not taken for an option. A full task on a sparse cache drops the sparse checkout first. On the Mercurial path the option goes to robustcheckout as --sparseprofile. A sparse clone is blobless (--filter=blob:none) and defers its checkout until the patterns are set, so only files inside the profile are written or fetched. Git keeps that filter on the origin remote and a fetch only inherits it when it names that remote, so run-task fetches through origin whenever the head repository is the repository it cloned, and points origin at the task's repository first so a cached clone follows a repository that moved. git 2.49 keeps the filter for a fetch by matching URL as well, but that is not documented and the workers run other versions. Refs taskcluster#1025
Sparse and full tasks in a worker pool share one checkout cache, and this change decides what happens when a task finds a cache another kind of task left behind: the cache only ever grows. A full task on a sparse cache widens it to a full checkout, and every later task on that worker runs full. A sparse task on a full cache runs on it as is. A sparse task on a cache with another sparse profile appends its own patterns with `sparse-checkout add`, so the cache ends up covering both profiles. Appending is why every translated pattern has to be positive. Whatever grows the cache runs after the checkout of the target revision, so a widen or an addition fetches that revision's files and not the old revision's. A cache in cone mode is treated as foreign and widened for now. The next commit starts creating cone caches and takes them over. Refs taskcluster#1025
Git tests a no cone pattern against every index entry on every checkout. Cone mode accepts only directories, and in return git matches by prefix and keeps the index itself small with the sparse index. When every pattern is an anchored literal path, which is true for twelve of the eighteen profiles, run-task applies them with `set --cone --sparse-index`. Locally repackage-msi went from 15 to 1 second on a warm cache. A path to a file works as a cone directory because cone mode includes the files of every ancestor directory. The same rule means a cone checkout has a few hundred more files than the profile names, the files directly inside every ancestor up to the root. Never fewer, so tasks are unaffected. Mixing follows the widen only rule. A cone cache takes further cone profiles with `add`. A glob profile on a cone cache converts it to no cone mode and keeps every directory it had plus the files of their ancestors, listed with `git ls-tree` at the target revision, so nothing that was checked out disappears. Refs taskcluster#1025
ahochheiden
force-pushed
the
sparse-checkout
branch
from
September 11, 2026 04:23
29d0422 to
8c33f29
Compare
bhearsum
reviewed
Sep 11, 2026
bhearsum
left a comment
bhearsum
left a comment
Contributor
There was a problem hiding this comment.
I think I need to take a second pass on this with fresh brains; here's a few thoughts to start.
A full task widens a sparse cache.
Does this mean that tasks that use sparse profiles that run after a full checkout is cached are less performant than those that run with nothing cached, or after a different task that used a sparse profile?
Patterns go through stdin so a name starting with a dash is not taken for an
option.
This also avoids the possibility of hitting any limits on the length of arguments, presumably?
| "safe.directory", | ||
| Path(destination_path).as_posix(), | ||
| ] | ||
| retry_required_command(b"vcs", args, extra_env=env) |
Contributor
There was a problem hiding this comment.
This is moving from being measured, to not being measured. It's arguably more correct this way (git config is not part of the clone), but it does change this metric. Is that on purpose?
| args.append("--filter=blob:none") | ||
|
|
||
| if shallow or sparse_patterns: | ||
| args.append("--no-checkout") |
Contributor
There was a problem hiding this comment.
Why does sparse_patterns imply --no-checkout?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1025
run-task gains sparse checkouts for git clones. The task supplies native git patterns in
<REPO>_SPARSE_PATTERNSas a JSON list, and names the profile with--<repo>-sparse-profile. A sparse clone is blobless and applies the patterns before its first checkout, so only the profile's files are written or fetched.Checkout caches are shared between sparse and full tasks and only ever widen:
A profile made only of paths uses cone mode with the sparse index.
On the Firefox side this is supported by Bug 2071081, which translates the Mercurial sparse profiles into these patterns, and Bug 2071092, which keys checkout caches on the run-task a task actually runs. Delivering the patterns to git tasks lands after a release with this change is pinned.