feat(run-task): enable parallel git checkout - #1041
Open
ahochheiden wants to merge 4 commits into
Open
Conversation
…ronment The patterns travel in `<REPO>_SPARSE_PATTERNS` as a JSON list of native git patterns, so the cloned revision does not need to contain the profile. A JSON list because generic-worker on Windows sets task environment variables through a batch `set` line, which cuts a value at its first newline. The option and the variable have to come together, and a checkout is required for either. Patterns go through stdin so a name starting with a dash is not taken for an option. A full task on a sparse cache drops the sparse checkout first. On the Mercurial path the option goes to robustcheckout as --sparseprofile. A sparse clone is blobless (--filter=blob:none) and defers its checkout until the patterns are set, so only files inside the profile are written or fetched. Git keeps that filter on the origin remote and a fetch only inherits it when it names that remote, so run-task fetches through origin whenever the head repository is the repository it cloned, and points origin at the task's repository first so a cached clone follows a repository that moved. git 2.49 keeps the filter for a fetch by matching URL as well, but that is not documented and the workers run other versions. Refs taskcluster#1025
Sparse and full tasks in a worker pool share one checkout cache, and this change decides what happens when a task finds a cache another kind of task left behind: the cache only ever grows. A full task on a sparse cache widens it to a full checkout, and every later task on that worker runs full. A sparse task on a full cache runs on it as is. A sparse task on a cache with another sparse profile appends its own patterns with `sparse-checkout add`, so the cache ends up covering both profiles. Appending is why every translated pattern has to be positive. Whatever grows the cache runs after the checkout of the target revision, so a widen or an addition fetches that revision's files and not the old revision's. A cache in cone mode is treated as foreign and widened for now. The next commit starts creating cone caches and takes them over. Refs taskcluster#1025
Git tests a no cone pattern against every index entry on every checkout. Cone mode accepts only directories, and in return git matches by prefix and keeps the index itself small with the sparse index. When every pattern is an anchored literal path, which is true for twelve of the eighteen profiles, run-task applies them with `set --cone --sparse-index`. Locally repackage-msi went from 15 to 1 second on a warm cache. A path to a file works as a cone directory because cone mode includes the files of every ancestor directory. The same rule means a cone checkout has a few hundred more files than the profile names, the files directly inside every ancestor up to the root. Never fewer, so tasks are unaffected. Mixing follows the widen only rule. A cone cache takes further cone profiles with `add`. A glob profile on a cone cache converts it to no cone mode and keeps every directory it had plus the files of their ancestors, listed with `git ls-tree` at the target revision, so nothing that was checked out disappears. Refs taskcluster#1025
git writes the working tree with one thread unless checkout.workers is set. With one worker per core the cold checkout write of the Firefox tree went from 27.9s to 13.3s on a GCP c3d-standard-8 docker worker and from 111.6s to 36.0s on an Azure Standard_F8alds_v7 Windows worker. Warm updates and blobless sparse checkouts did not change, since there the fetch dominates. The setting sits next to safe.directory in the global config, needs git 2.32, and older gits ignore the key. The git docs note that parallel checkout can be slower on spinning disks or machines with very few cores. Fixes taskcluster#1040
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1040
Stacked on #1039, only the top commit is new here.
Cold checkouts of the Firefox tree went from 27.9s to 13.3s on Linux and 111.6s to 36.0s on Windows. Git older than 2.32 ignores the settings.