Skip to content

[WIP] Fix cleanup and isolation for self-hosted job runtimes - #75444

Merged
lpcox merged 8 commits into
mainfrom
copilot/self-hosted-isolate-job-runtime
Nov 7, 2026
Merged

lpcox merged 8 commits into
mainfrom
copilot/self-hosted-isolate-job-runtime

Conversation

Copilot AI commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor
  • Trace shared runtime paths, teardown, and Docker resource collisions.
  • Allocate private per-invocation runtime roots and propagate host/container paths.
  • Scope setup, installation, Docker resources, and teardown without sudo cleanup.
  • Add concurrent-job, interruption, and unprivileged cleanup regressions.
  • Run targeted validation, security review, and final repository checks.

Actions review follow-up

Installer staging uses native mktemp -d under the canonical runner scratch directory, with device/inode/UID-checked, unprivileged cleanup. Rootless Copilot installation now fails explicitly if ~/.copilot is unusable or unwritable; it does not repair ownership or reclaim existing state. Concurrent installer and error-path regressions cover staging cleanup, and a non-root permissions test plus a root-independent invalid-directory test cover preflight failure.

Legacy /tmp/gh-aw and AWF chroots lack trustworthy per-job ownership records. Both post-step implementations leave them untouched rather than using sudo or recursively deleting another runner's state. Hosts must manage these legacy leftovers outside active jobs; full runtime-path migration remains a separate unchecked item above. No unconditional AWF-chroot warning is emitted.

The optional Docker fixture and its private cleanup framework were removed because no repository CI job configured its image. The retained unprivileged failure regression checks warning/exit behavior without claiming live Docker ownership coverage. BSD stat field-order fixtures and a real macOS stat regression cover ownership snapshots; actual installer failure tests verify the EXIT cleanup survives setup.

Copilot AI and others added 3 commits November 6, 2026 16:55
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
…egressions

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
@lpcox
lpcox marked this pull request as ready for review November 6, 2026 18:17
Copilot AI dynamic review requested due to automatic review settings November 6, 2026 18:17
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · routed: gpt56 max · ~1 AIC · ⌖ ~1 AIC · ⊞ 6.3K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · routed: opus55 xhigh · ~13 AIC · ⌖ ~2 AIC · ⊞ 7.7K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Generated by Ponytail Reviewer for #75444

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Specification Reviewer completed successfully!

PR Specification Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from PR Specification Reviewer · gpt56 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K · ◷

Generated by PR Specification Reviewer for #75444

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

No actionable correctness, security, performance, or maintainability issues were identified in the changed lines during the manual review. The specialist pass was unavailable because its file-access call was denied by workflow permissions; this is not treated as evidence of a clean specialist review.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~2 AIC · ⌖ ~1 AIC · ⊞ 1.6K
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What this change does: It removes broad sudo sweeps of legacy AWF chroots and gives each installer a private staging directory with identity-checked cleanup. It also adds a large JavaScript regression suite for concurrency, interruption, rootless setup, and Docker-written leftovers.

Should fix:

  1. The staging allocator reimplements mktemp (actions/setup/sh/release_integrity.sh:L37-L64)

    • What this is: The shared helper generates random bytes, validates them, and retries directory creation five times.
    • Problem: mktemp -d already atomically creates a private unique directory, and the repository already uses it for installer staging. The custom allocator adds code and a portability/test surface without changing the identity snapshot or cleanup requirement.
    • Fix: Keep parent canonicalization, identity recording, and traps, but replace the nonce and retry block with mktemp -d "${parent}/${prefix}-install-XXXXXX".
    • If we skip it: Both installers carry a hand-rolled allocator that must be maintained around behavior the platform already provides.
  2. The Docker regression is never exercised by repository CI (actions/setup/js/release_staging.test.cjs:L452-L531)

    • What this is: This is an 80-line Docker fixture for root-owned leftovers, plus a cleanup helper and two helper-only tests.
    • Problem: No workflow or Make target sets GH_AW_TEST_DOCKER_IMAGE, so skipIf skips this code in every repository CI run while adding a second cleanup framework and Docker-specific maintenance burden.
    • Fix: Remove this optional block and its private cleanup helper until a real CI job can execute it, or wire it into one such job before keeping it.
    • If we skip it: The largest specialized part of the staging test can drift while every normal run remains green.

Not checked: live Docker execution; the repository does not configure the image variable.
Verdict: fix 1 and 2 first.
Lean: -170 lines possible.

Generated by ✂️ Ponytail Reviewer for #75444 · codex · routed: gpt56 max · ~7 AIC · ⌖ ~1 AIC · ⊞ 118
Comment /ponytail to run again

Comment thread actions/setup/sh/release_integrity.sh Resolved

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs and /tdd. Requesting changes because the shared-/tmp cleanup this PR targets is still partly sudo-based.

📋 Key Themes & Highlights

Key Themes

  • Incomplete isolation fix: actions/setup/post.js:109 and actions/setup/clean.sh:35 still run sudo rm -rf /tmp/gh-aw. That can delete state belonging to another runner sharing the host, which contradicts the goal of scoping teardown without sudo. The matching checklist item in the PR body is still unchecked, so this is expected for WIP, but it blocks merge.
  • Test coverage is weaker than it looks: chroot_home_cleanup.test.js checks source text, not behavior. The root-skipped and Docker-gated cases in release_staging.test.cjs may not run in CI.
  • Unverified edge paths: the BSD stat -f fallback, EXIT trap ordering, and the rootless hard-fail behavior change have no direct regression test.

Positive Highlights

  • ✅ create_release_staging allocates a 0700 private directory and checks device:inode:uid before cleanup. It refuses to remove a replaced directory.
  • ✅ The concurrency, symlink-replacement, interruption, and fake-sudo tests cover the real failure modes well.
  • ✅ bash -n and shellcheck pass on all four changed shell scripts.

Not verified in this review

  • I did not run the vitest or Go suites. Vitest discovery for actions/setup/js/*.cjs and whether CI sets GH_AW_TEST_DOCKER_IMAGE are unconfirmed.

@copilot please address the review comments above, starting with the remaining sudo rm -rf on /tmp/gh-aw.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~14 AIC · ⌖ ~1 AIC · ⊞ 5.4K
Comment /matt to run again

Comment thread actions/setup/sh/install_copilot_cli.sh Resolved
Comment thread actions/setup/sh/release_integrity.sh Resolved
Comment thread actions/setup/post.js Resolved
Comment thread actions/setup/sh/release_integrity.sh Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The change regresses temporary-directory fallback and leaves AWF chroots without scoped cleanup, while macOS test defects and missing Docker CI coverage leave key behavior insufficiently validated.

5 open issues
What changed in this PR

This PR begins addressing #75441 by isolating installer staging and removing cross-job AWF chroot cleanup from gh-aw’s setup action.

Changes:

  • Add private installer staging with collision-resistant names, ownership checks, and scoped cleanup.
  • Remove shared /tmp chroot sweeps and recursive Copilot ownership changes in rootless mode.
  • Add concurrency, interruption, path-portability, and optional Docker ownership regressions.
File Description
pkg/​cli/​install_copilot_cli_test.go Prepare temporary directories and assert rootless installation avoids chown.
pkg/​cli/​install_copilot_cli_cooldown_test.go Update prerequisites and temporary-directory fixtures.
actions/​setup/​sh/​release_integrity.sh Add private staging allocation and ownership-checked cleanup.
actions/​setup/​sh/​release_integrity_test.sh Create the runner temporary-directory fixture.
actions/​setup/​sh/​install_copilot_cli.sh Adopt private staging, remove chroot sweeps, and adjust rootless preflight.
actions/​setup/​sh/​install_awf_binary.sh Adopt private staging and remove shared chroot cleanup.
actions/​setup/​post.js Replace broad chroot cleanup with a warning.
actions/​setup/​js/​release_staging.test.cjs Add staging lifecycle, concurrency, portability, and optional Docker tests.
actions/​setup/​js/​chroot_home_cleanup.test.js Replace sweep tests with source checks enforcing cleanup boundaries.
actions/​setup/​clean.sh Replace script-mode chroot sweeps with a warning.

Review effort: Max (auto)

path.join(bin, "stat"),
`#!/bin/sh
if [ "$1" = "-c" ]; then exit 1; fi
exec "$REAL_STAT" -c "$2" "$3"
expect(fs.existsSync(job.dir)).toBe(false);
});

it.skipIf(!dockerImage || !process.getuid || process.getuid() === 0)(
Comment on lines +124 to +125
# Other invocations can share both /tmp and the runner UID. Neither a filename
# pattern nor the filesystem owner proves that a chroot belongs to this install.
Comment thread actions/setup/sh/release_integrity.sh Outdated
echo "ERROR: Invalid release staging prefix" >&2
return 1
fi
parent="${RUNNER_TEMP:-$PWD}"
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@lpcox

lpcox commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

Addressed the Actions-bot reviews in 9aebaac. Ponytail review 5805003897: replaced the custom allocator with mktemp and removed the Docker fixture that CI never exercised. Skills review 5805082164: removed shared /tmp/gh-aw sudo/recursive cleanup from both post steps, replaced source-string checks with behavior tests, added actual-installer EXIT cleanup tests, BSD snapshot coverage, and a root-independent preflight failure test. The existing unwritable-directory test also passes. All nine Actions threads have direct replies and are resolved. Code-quality review 5804965042 reported no actionable findings; its specialist-pass limitation remains acknowledged.

The 20 focused JavaScript tests, Copilot installer Go tests, shell integrity/AWF suites, build, and change-scoped lint/drift checks pass. The with-tests gate initially lacked lint tooling; after restoring dependencies, its lint gate passed and the focused Go installer tests passed after updating the restricted PATH fixture for mktemp.

The PR description records that legacy shared state is intentionally left untouched, live Docker-created ownership coverage is not established, and full runtime-path migration remains incomplete. This does not claim the new Copilot review findings are all addressed, or that the outstanding changes-requested review has been superseded.

@lpcox

lpcox commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

@copilot address all review feedback

Copilot AI and others added 2 commits November 6, 2026 23:33
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>

Copilot AI commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

@copilot address all review feedback

Addressed the outstanding review findings in c7aa551: portable TMPDIR fallback, native BSD stat coverage, a pinned non-root Docker CI job, and invocation-scoped AWF temp allocation/teardown. Workspace canonicalization was already present. Focused Docker and shell/JS tests pass.

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
@lpcox
lpcox dismissed github-actions[bot]’s stale review November 7, 2026 00:27

feedback addresseed

@lpcox
lpcox merged commit 239d819 into main Nov 7, 2026
63 checks passed
@lpcox
lpcox deleted the copilot/self-hosted-isolate-job-runtime branch November 7, 2026 00:27
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.2.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Self-hosted: isolate each job's runtime directory and scope cleanup without sudo

3 participants