Skip to content

Seeding a worktree copy copies a 48 GB target/debug: three worktrees took 140 GB of disk in an hour #419

Closed

Description

@alex09x

Seeding a new worktree's copy copies the main copy's whole target/debug/deps, however large it has grown. For this repository on a Linux build node that is 48 GB in 571,304 files per worktree, so three short-lived worktrees took 140 GB within an hour and brought the disk from 22% to 16% free. The only guard is "the disk has twice the size free", which lets seeding go on until the janitor has to prune.

On the build node that holds this repository's copies:

$ du -sh target/debug/deps target/debug/build target/debug/.fingerprint   # main copy
48G	deps
352M	build
60M	.fingerprint
$ ls target/debug/deps | wc -l
571304
$ du -sh prod-code--wt-*                                                  # worktree copies
27G  30G  38G  38G  43G  46G  46G  48G
$ df -h .
/dev/...  913G  717G  150G  83% /

Three of those copies (46 G, 46 G and 48 G) came from worktrees that existed only for a cargo update and a report test.

seed_build_cache_within (#278) copies deps, build and .fingerprint whenever free space is at least twice their size. On a 980 GB filesystem with 150 GB free it would still copy 48 GB, leaving 102 GB (10%), below the janitor's 15% prune line (#386). The next copy then waits on pruning. seed_dependency_trees (#412, #414) uses the same rule.

Expected:

Environment

  • gateway 0.3.17 on a Linux x86_64 build node, 980 GB storage filesystem
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinggatewayServer daemonworktreeMulti-worktree isolation and sync

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions