Skip to content

feat: implement issue #2141 — dev-lead test-tamper guard treats tests the PR itself added as existing: a pass cannot revise its own new test, and its real fixes get retracted - #2149

Merged
don-petry merged 5 commits into
mainfrom
dev-lead/issue-2141-20261009-1927
Oct 9, 2026
Merged

don-petry merged 5 commits into
mainfrom
dev-lead/issue-2141-20261009-1927

Conversation

@don-petry

@don-petry don-petry commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

dev-lead test-tamper guard treats tests the PR itself added as existing: a pass cannot revise its own new test, and its real fixes get retracted

From the issue: The test-tamper guard from #2013 (scripts/lib/test-tamper-guard.sh) decides what an "existing" test is by comparing against the pre-pass head. A test file the PR itself added in an earlier commit therefore counts as existing. When a reviewer asks for a change to that new test, dev-lead cannot make it without a Test-Change-Justification: trailer; if the model forgets the trailer, the push is refused, the PR is flagged for a human, and every "Fixed" reply the pass posted is retracted, including replies for fixes that had nothing to do with the test.

Risk

Low — changes automation shell logic under scripts/, covered by shellcheck (--severity=warning) and the bats suite.

Test plan

Tests added/updated: tests/dev-lead/unit/test_fix_reviews.bats,tests/dev-lead/unit/test_test_tamper_guard.bats. Verification: bash scripts/dev-lead-lint.sh (shellcheck --severity=warning) ran pre-commit; the bats suite runs in CI.

Rollback

Revert this PR. No non-revertible side effects (no tags, migrations, or external state).

Monitoring

This PR's Lint (shellcheck) and bats checks show pass/fail; watch subsequent dev-lead / pr-review runs for behavioral regressions.

Closes #2141

View guided diff Turn on auto-fix

… the PR itself added as existing: a pass cannot revise its own new test, and its real fixes get retracted
@don-petry
don-petry requested a review from a team as a code owner October 9, 2026 19:59
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@qodo-code-review

Copy link
Copy Markdown

ⓘ Qodo reviews are paused because your trial has ended. Ask your workspace admin to add credits to resume reviews. Manage billing

@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 22 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 85136cb6-9740-4191-996f-bbf8f6cf089f

📥 Commits

Reviewing files that changed from the base of the PR and between 1ba2304 and aa607a8.


📒 Files selected for processing (4)
  • scripts/dev-lead-fix-reviews.sh
  • scripts/lib/test-tamper-guard.sh
  • tests/dev-lead/unit/test_fix_reviews.bats
  • tests/dev-lead/unit/test_test_tamper_guard.bats


  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — waiting on PR blockers (intent: review-changes)

PR: #2149
No changes were committed, but the PR still can't be marked done: required check SonarCloud is still pending. The retry cron will re-attempt automatically. Next attempt after: 2026-10-09T20:30:10Z

@don-petry

Copy link
Copy Markdown
Collaborator Author

Note

@don-petry I reviewed this PR and no code changes were needed, but I can't mark it done yet: required check SonarCloud is still pending. I'll re-check automatically.
Next attempt after: 2026-10-09T20:30:10Z

@don-petry
don-petry enabled auto-merge (squash) October 9, 2026 20:00
@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-09T21:01:24Z.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refines the test-tamper guard mechanism to only protect tests that existed prior to the pull request (i.e., present at the merge base with the base branch). This allows PRs to freely modify or revise tests that they themselves introduced. The changes include updating the scanning logic to compute preexisting removals, resolving the merge base, and adding comprehensive unit tests to verify the new behavior. I have no feedback to provide as there are no review comments.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread scripts/dev-lead-fix-reviews.sh Resolved
Comment thread scripts/lib/test-tamper-guard.sh Resolved
Comment thread scripts/dev-lead-fix-reviews.sh Resolved
@codeant-ai

This comment has been minimized.

@codeant-ai

This comment has been minimized.

don-petry and others added 2 commits October 9, 2026 20:07
…ath in skip diff

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-reviews (applied)

Changes committed and pushed. Requested items addressed:

  • scripts/dev-lead-fix-reviews.sh:2763 — applied
  • scripts/lib/test-tamper-guard.sh:258 — applied
  • scripts/dev-lead-fix-reviews.sh:2763 — applied

@don-petry
don-petry enabled auto-merge (squash) October 9, 2026 20:16
@don-petry
don-petry disabled auto-merge October 9, 2026 20:32
@don-petry

Copy link
Copy Markdown
Collaborator Author

Codex usage-limit notice; no code finding, nothing to change.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Qodo billing/trial-ended notice; no code finding, nothing to change.

@don-petry

Copy link
Copy Markdown
Collaborator Author

CodeRabbit rate-limit summary; it has no actionable, nitpick, outside-diff, or architecture-review findings, so nothing to change.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Valid but narrow: a test file renamed earlier in the same PR is absent at the merge base, so later edits/skips on the new path escape the guard in ttg_scan_pass. Fixing it needs rename-aware merge-base mapping, which is beyond this PR's scope (#2141). Tracked in #2047.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Already handled: commit bc2473d makes dev-lead-fix-reviews.sh look up the PR's real base ref via gh api repos/$REPO/pulls/$PR_NUMBER (.base.ref) before the tamper guard runs, so BASE_REF is the PR's actual target and only falls back to main if the lookup fails. Covered by the stub added in 0cf6d84.

@don-petry

Copy link
Copy Markdown
Collaborator Author

SonarCloud quality gate passed; no action needed.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-reviews (no-changes)

Agent reasoning
- The `cancelled` runs are dev-lead `ci-relay`, `dispatch` and `resume`, plus one `review / review`. The matching runs for the same jobs are `skipped` or still running, so I treated these as superseded runs, not failures. I didn't check why they were cancelled.
- Several checks are still queued or running (`bats`, `unit`, `Analyze`, and others), so I couldn't confirm the full suite passed.
**Phase 1b — dispositions posted on the PR's issue comments**
- **Informational:** the Codex usage-limit notice, the Qodo trial-ended notice, the CodeRabbit rate-limit summary and the SonarCloud quality-gate comment. The CodeRabbit summary has no finding-bearing section.
- **Answered:** the CodeAnt "Medium Risk" note about `BASE_REF`. Commit `bc2473d` already looks up the PR's real base ref before the tamper guard runs, and `0cf6d84` stubs that lookup in the tests.
- **Out-of-scope, tracked in #2047:** the CodeAnt nitpick that a test renamed earlier in the PR is absent at the merge base. Later edits or skips on the renamed path would then bypass `ttg_scan_pass`. The finding is real but needs rename-aware merge-base mapping. I added it to the deferred-findings issue #2047.
- **Skipped:** our own automation comments (`don-petry` and `donpetry-bot`).
I accidentally posted one junk test comment on the PR (a call with a placeholder id) and deleted it right away.
**Test verification:** I didn't run the suite because I changed no code.
**Files changed:** none.

@don-petry
don-petry enabled auto-merge (squash) October 9, 2026 20:34
@sonarqubecloud

sonarqubecloud Bot commented Oct 9, 2026

Copy link
Copy Markdown

@don-petry

Copy link
Copy Markdown
Collaborator Author

No description provided.

@donpetry-bot

Copy link
Copy Markdown
Contributor

No description provided.

@don-petry

Copy link
Copy Markdown
Collaborator Author

No description provided.

@don-petry
don-petry disabled auto-merge October 9, 2026 21:18
@don-petry

Copy link
Copy Markdown
Collaborator Author

CodeRabbit rate-limit notice — no code findings to address. The comment indicates the free OSS review quota was consumed; this is an operational notice, not a code defect.

@don-petry
don-petry enabled auto-merge (squash) October 9, 2026 21:18
@don-petry
don-petry merged commit d747070 into main Oct 9, 2026
69 checks passed
@don-petry
don-petry deleted the dev-lead/issue-2141-20261009-1927 branch October 9, 2026 21:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dev-lead test-tamper guard treats tests the PR itself added as existing: a pass cannot revise its own new test, and its real fixes get retracted

2 participants