Skip to content

A dev-lead PR a few commits behind main is held for a human: Test-Deletion Guard and the test-tamper guard both read base-side test changes as the PR's #2254

Closed

Description

@don-petry

Problem

A dev-lead PR whose branch is a few commits behind main gets held for a human by two guards that both misread changes that happened on main as changes made by the PR. Neither is a real finding, and together they stop the PR.

Evidence: PR #2252 (issue #1960), 2026-10-11

The branch was cut at 63c03b4f; main had moved 3 commits to b6bf0f2e. The PR itself adds two test files' worth of cases and deletes nothing (git diff origin/main...HEAD -- tests: 124 insertions, 0 deletions).

  1. Test-Deletion Guard reported a deleted test. tests/dev-lead/unit/test_budget_breaker_writer.bats was added on main by feat(budget-breaker): default-off writer that pauses and resumes the fleet from observed rate limits (slice 3b) #2248 after the branch point. The guard compared base against head directly, so a file that exists on main and not on the branch read as deleted by the PR. Runs 38104498680 and 38104666687.
  2. dev-lead's test-tamper guard then refused dev-lead's own fix. A review-changes pass rebased onto main to clear the first failure. The tamper guard reported "Existing test changed without justification" for tests/dev-lead/unit/test_dev_lead_retry_budget_hold.bats and tests/test_persona_pr_pregate.bats. Those files changed on main, not in the PR. dev-lead did not push, disabled auto-merge, and the PR was labelled needs-human-review three minutes after it opened.

A plain "update branch" (merge main into the branch) cleared the first guard. The label had to be removed by hand.

Acceptance criteria

  1. Test-Deletion Guard compares the PR head against the merge base with the base branch, so a file added to the base after the branch point is not reported as deleted. Test: a branch behind a base that gained a test file passes.
  2. dev-lead's test-tamper guard ignores test changes that a rebase or merge of the base branch brings in. Only changes in the pass's own commits relative to the updated base count. Test: a pass that only rebases onto a base with changed tests is clean; a pass that rebases and also edits an existing test is still flagged.
  3. A rebase or update-branch pass that changes no file of its own does not apply needs-human-review or disable auto-merge.

Related

Activity

  1. github-actions commented on Oct 11, 2026

    @github-actions
    Contributor

    Thanks for the detailed report and for including the run links and the PR #2252 evidence, which makes the failure easy to follow. To confirm the scope of the fix, can you say whether the tamper guard in the rebase case compares against the pre-rebase base SHA or the current tip of the base branch? Knowing which ref it uses will show whether a merge-base comparison, as in the acceptance criteria, is enough or whether the pass needs to record the base SHA before it rebases.

  2. don-petry commented on Oct 11, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #2254 — A dev-lead PR a few commits behind main is held for a human: Test-Deletion Guard and the test-tamper guard both read base-side test changes as the PR's

    Scope

    Make both guards judge only the PR's (or the pass's) own test changes. Changes that main gained after the branch point, or that a rebase or merge of main brings in, no longer count.

    Implementation Checklist

    • scripts/test-deletion-guard.sh (new): move the guard logic out of the workflow. It diffs the PR head against merge-base(base, head) instead of the base tip. If no merge base resolves, it fails closed.
    • .github/workflows/test-deletion-guard.yml: call the script (same env, same ack-test-deletion label semantics).
    • scripts/lib/test-tamper-guard.sh: ttg_scan_pass takes an optional 4th argument, the updated merge base (pass result vs. origin/<base>). When it differs from the pre-pass merge base, the pass brought base-side changes in. For each touched test file, the guard rebuilds what a plain rebase or merge would give: a 3-way git merge-file of pre-pass head, old merge base and new merge base. It then judges only the pass's diff against that expected content. A line still counts as "existing" if it is on the updated base and was not added by the PR. Skips are counted the same way. A test file that main added after the branch point is protected too.
    • scripts/dev-lead-fix-reviews.sh: resolve the updated merge base (git merge-base HEAD origin/<base>) next to the existing one and pass it to ttg_scan_pass. If it cannot be resolved, the old strict behaviour stays (fail closed).

    Tests to Write

    • tests/test_deletion_guard.bats: a branch behind a base that gained a test file passes; a PR that really deletes a test fails; the ack-test-deletion label passes it; a missing merge base fails closed.
    • tests/dev-lead/unit/test_test_tamper_guard.bats: a pass that only rebases onto a base with changed tests is clean; rebase plus an edit to an existing test is tampered; rebase plus an edit to a test file main added is tampered; rebase plus a skip is tampered; a merge of the base (instead of a rebase) is clean; an unresolvable updated merge base gives unknown.
    • tests/dev-lead/unit/test_fix_reviews.bats: a review-changes pass that only rebases onto a moved main (whose test changed) is pushed, with no needs-human-review label and no auto-merge disable (AC 3). Rebase plus an existing-test edit is still refused.

    Test Command

    bats tests/test_deletion_guard.bats tests/dev-lead/unit/test_test_tamper_guard.bats tests/dev-lead/unit/test_fix_reviews.bats, plus bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Oct 11, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #2300

    The implementation for issue #2254 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions