Skip to content

A worktree seed leaves a fifth of the disk free and logs a skipped copy (#419) - #420

Merged
alex09x merged 1 commit into
mainfrom
fix/419-seed-keeps-disk-free
Sep 26, 2026
Merged

alex09x merged 1 commit into
mainfrom
fix/419-seed-keeps-disk-free

Conversation

@alex09x

@alex09x alex09x commented Sep 26, 2026

Copy link
Copy Markdown
Owner

Problem

A seed copied the main copy's whole target/debug into every new worktree copy whenever twice its size was free (#419). For this repository on a build node that was 48 GB in 571,304 files. Three short-lived worktrees (46 GB, 46 GB, 48 GB) took 140 GB within an hour and left the 980 GB disk at 16% free, near the janitor's 15% prune line (#386). On a disk with 150 GB free the rule would still copy 48 GB and leave 10%. seed_dependency_trees (#412, #414) used the same rule.

Change

  • seed_fits(what, size, space) decides for both seeds. It requires twice the size free, as before, and at least 20% of the filesystem (SEED_MIN_FREE_SHARE) free after the copy. That keeps seeding above the janitor's 15%, so a seed never forces pruning or the disk pressure that placement avoids (Placement ignores memory and disk: a node out of space or memory still gets new workspaces #396).
  • A skipped copy is logged: 🌱 [SEED] skipped: the copy would leave too little disk free, with what, size_mb, free_mb and total_mb. A slow first build in a worktree can then be told apart from a bug.
  • free_disk_bytes becomes disk_space, which returns free and total bytes from statvfs.

With the numbers above (980 GB, 290 GB free after the cleanup), the next 48 GB seed still happens (242 GB, 24.7% left) and the one after it is skipped (194 GB would be 19.8%).

Reproduce and verify

$ du -sh target/debug/deps && ls target/debug/deps | wc -l     # main copy on the node
48G
571304
$ du -sh prod-code--wt-*
27G  30G  38G  38G  43G  46G  46G  48G

The three copies of worktrees that no longer existed were removed by hand (150 GB free before, 290 GB after).

a_seeded_copy_goes_without_a_build_cache_it_has_no_room_for now also checks the share rule. A 1,000-byte cache with 10,000 free is skipped on a 46,000-byte filesystem (9,000 left, under a fifth) and copied on a 44,000-byte one. It also checks that disk_space reports a total at least as large as the free space. On the old rule the 46,000 case would have copied. The other seed tests use roomy() / space(free, total).

Most of the 48 GB is stale split-DWARF files (565,488 .dwo, 26.2 GB), filed separately.

Checks

On a Linux build node:

cargo fmt --all -- --check                               # clean
cargo clippy --workspace --all-targets -- -D warnings    # Finished, no warnings
cargo test -p prod-code-gateway --lib                    # 80 passed, 1 ignored

prod-code validate --diff --compile: 0 errors, 0 warnings.

Closes #419

@alex09x alex09x added bug Something isn't working gateway Server daemon worktree Multi-worktree isolation and sync labels Sep 26, 2026
@alex09x
alex09x merged commit 0caf79a into main Sep 26, 2026
@alex09x
alex09x deleted the fix/419-seed-keeps-disk-free branch September 26, 2026 08:36
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-26T08:38:13.776497Z a7350e3 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a7350e3217

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +547 to +549
let after = space.free.saturating_sub(size);
let fits = space.free >= size.saturating_mul(2)
&& after as f64 >= SEED_MIN_FREE_SHARE * space.total as f64;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Serialize seed reservations before enforcing the free-space floor

When multiple fresh worktrees are seeded concurrently, each SyncProbe runs in its own connection task and can read the same pre-copy free-space value, so every copy can pass this check even though their combined allocation violates the 20% floor. With the commit's 980 GB example, three simultaneous 48 GB seeds can all observe 290 GB free, each compute 242 GB remaining, and ultimately leave only 146 GB (14.9%), crossing the janitor threshold this change is intended to avoid. Reserve the bytes atomically or serialize the capacity check and copy across seed operations.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working gateway Server daemon worktree Multi-worktree isolation and sync

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Seeding a worktree copy copies a 48 GB target/debug: three worktrees took 140 GB of disk in an hour

1 participant