Skip to content

feat: implement issue #2013 — dev-lead fix-reviews: posts "Fixed" replies for changes that were never pushed, and edits tests to match its own code - #2019

Merged
don-petry merged 9 commits into
mainfrom
dev-lead/issue-2013-20261002-1249
Oct 3, 2026
Merged

don-petry merged 9 commits into
mainfrom
dev-lead/issue-2013-20261002-1249

Conversation

@don-petry

@don-petry don-petry commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

dev-lead fix-reviews: posts "Fixed" replies for changes that were never pushed, and edits tests to match its own code

From #2013 (observed on combsmith/.github#1220, three incidents):

  1. "Fixed" replies posted before a push that was then refused; the claim named the PR's first commit (a stale pre-pass head) and was never retracted.
  2. 15a919e was pushed with the existing suite red: it added a new test and broke an existing one without editing it. CodeRabbit then confirmed the reply as addressed, so a wrong fix cleared a review gate.
  3. 13927fc rewrote an existing test to match the code, applying a bot's "update the older test so the suite can pass" literally.

Fix (ADR-0011): verify what landed and retract claims that did not (claim-landing.sh, addressed-claim-verify.sh); refuse an unjustified existing-test rewrite (test-tamper-guard.sh); refuse a push when a test that passed on the pre-pass head fails on the result (test-regression-guard.sh). Applies to fix-reviews, fix-bot-comment, review-changes and human-pr.

Risk

Medium — changes GitHub Actions workflow behavior, which is exercised only post-merge; verify via the affected workflow runs.

Test plan

Tests added/updated: tests/dev-lead/unit/test_addressed_claim_verify.bats,tests/dev-lead/unit/test_claim_landing.bats tests/dev-lead/unit/test_fix_reviews.bats,tests/dev-lead/unit/test_test_tamper_guard.bats, tests/dev-lead/unit/test_test_regression_guard.bats. Verification: bash scripts/dev-lead-lint.sh (shellcheck --severity=warning) ran pre-commit; the bats suite runs in CI.

Rollback

Revert this PR. No non-revertible side effects (no tags, migrations, or external state).

Monitoring

Watch the affected workflow run(s) in the Actions tab and this PR's Lint check for regressions.

Closes #2013

Review in cubic

@chatgpt-codex-connector

This comment has been minimized.

@qodo-code-review

This comment has been minimized.

@coderabbitai

This comment has been minimized.

Comment thread scripts/dev-lead-fix-reviews.sh Resolved
Comment thread scripts/dev-lead-fix-reviews.sh Resolved
@codeant-ai

This comment has been minimized.

@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-02T14:14:09Z.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements ADR-0010, introducing mechanisms to verify pushed commits and retract unlanded claims for write-mode personas on shared branches, alongside a test-tamper guard to prevent unauthorized test rewrites. The feedback suggests improving the robustness of jq parsing in scripts/lib/claim-landing.sh by using literal string matching (contains and split/join) instead of regex functions (test and gsub), and safely handling potentially null fields. Additionally, it is recommended to assert specific non-zero exit codes in the unit tests to prevent false positives.

Comment thread scripts/lib/claim-landing.sh Resolved
Comment thread scripts/lib/claim-landing.sh Resolved
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — review-changes (partial)

A commit was pushed, but not every requested change was applied. Per requested item:

  • scripts/dev-lead-fix-reviews.sh:1992 — applied
  • scripts/dev-lead-fix-reviews.sh:2017 — applied
  • scripts/lib/claim-landing.sh:93 — not applied
  • scripts/lib/claim-landing.sh:125 — not applied
  • tests/dev-lead/unit/test_claim_landing.bats:271 — applied

The unaddressed items above still need work.

@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-02T14:19:55Z.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed against the latest diff

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread scripts/lib/test-tamper-guard.sh Resolved
Comment thread scripts/lib/test-tamper-guard.sh Resolved
Comment thread scripts/dev-lead-fix-reviews.sh Resolved
Comment thread scripts/dev-lead-fix-reviews.sh Resolved
Comment thread scripts/lib/addressed-claim-verify.sh Resolved
Comment thread scripts/dev-lead-fix-reviews.sh Resolved
Comment thread prompts/dev-lead/fix-bot-comment.md Resolved
Comment thread scripts/dev-lead-fix-reviews.sh Resolved
@donpetry-bot

Copy link
Copy Markdown
Contributor

Review — fix requested (cycle 1/3)

The automated review identified the following issues. Please address each one:

Findings to fix

Automated review — NEEDS HUMAN REVIEW

Risk: MEDIUM
Reviewed commit: 4c9463fa2e95fa7492b5e7d894e7c699b96708b4
Cascade: triage → audit (triage: haiku 4.5 [sonnet 5.5, sonnet 5] → deep: opus 5.5 [opus 4.8, sonnet 5.5] + duck: gemini-3.8-flash [sonnet 5.5] → audit: opus 5.5 [opus 4.8, opus 4.7])

Summary

This PR adds harness controls that verify a claim landed, retract claims that did not, and guard against test tampering. It tightens the security posture: retraction PATCHes only this bot's own comments and passes the body as argv, the retraction reason is sanitised to [a-z-], and the removed trailing try_enable_auto_merge used to re-enable auto-merge after a guard abort (cp_rc==3). CI_WEAKENING_DETECTED is a false positive (the skip strings are the guard's regex and quoted bats fixtures, and the existing-test edits only re-point claim SHAs and add assertions), but required checks are not finished (bats and unit are in progress, several validate-* jobs are queued) and the merge state is BLOCKED, so this escalates instead of approving.

Downstream impact

This change is consumed by 8 downstream repo(s) that pin the affected reusable workflow / lib / prompt. Impacted consumers:

Impacted shared surfaces:
  - .github/workflows/dev-lead-reusable.yml

Impacted consumers (8, fetching up to 10):
  - petry-projects/.github (pins .github/workflows/dev-lead-reusable.yml)
      .github/workflows/dev-lead.yml
  - petry-projects/.github-private (pins .github/workflows/dev-lead-reusable.yml)
      .github/workflows/dev-lead.yml
  - petry-projects/ContentTwin (pins .github/workflows/dev-lead-reusable.yml)
      .github/workflows/dev-lead.yml
  - petry-projects/TalkTerm (pins .github/workflows/dev-lead-reusable.yml)
      .github/workflows/dev-lead.yml
  - petry-projects/bmad-bgreat-suite (pins .github/workflows/dev-lead-reusable.yml)
      .github/workflows/dev-lead.yml
  - petry-projects/broodly (pins .github/workflows/dev-lead-reusable.yml)
      .github/workflows/dev-lead.yml
  - petry-projects/google-app-scripts (pins .github/workflows/dev-lead-reusable.yml)
      .github/workflows/dev-lead.yml
  - petry-projects/markets (pins .github/workflows/dev-lead-reusable.yml)
      .github/workflows/dev-lead.yml

Findings

  • major: Required checks are not green. bats and unit are IN_PROGRESS; check-model-pins, validate-caller-inputs, validate-ingress-if, template-drift, validate-agent-profiles, validate-eval-cases, validate-interaction-model, verify-persona-teams, persona-reach-check and gh-aw-compile are QUEUED; validate-personas is IN_PROGRESS. mergeStateStatus is BLOCKED. The new bats suites (test_claim_landing, test_test_tamper_guard) must pass before merge.
  • minor (scripts/lib/test-tamper-guard.sh:96): The same model whose edits the guard checks can author the Test-Change-Justification trailer. Any 20+ character line satisfies it, and a 'justified' verdict only logs a ::notice::. A prompt-injected bot comment could tell the model to add the trailer. This does not regress anything (no control existed before), but consider adding needs-human-review on a 'justified' result, or requiring the trailer to cite a resolvable comment or issue URL.
  • minor (scripts/dev-lead-fix-reviews.sh:2353): Confirmed: when retract_unlanded_claims fails after a successful push, cp_rc becomes 1. resolution_gate_open only falls back to cp_rc when a SHA is missing; otherwise ri_may_resolve(pre, cur) opens the gate for an advanced head. resolve_addressed_bot_threads therefore still runs and a 'no-changes' marker is posted. Each claim is re-checked in-pass against HEAD, which limits the impact.
  • minor (scripts/lib/test-tamper-guard.sh:38): The test-tamper guard covers only the fix-reviews and fix-bot-comment intents (review-changes and human are not guarded). The skip regex misses Jest .only/fit/fdescribe and C# [Skip]. A rename with content loss can escape numstat's removed-line count under rename detection. These are follow-ups, not blockers.
  • info (scripts/dev-lead-fix-reviews.sh:2391): The trailing try_enable_auto_merge in fix-bot-comment was removed. It ran even when cp_rc==3 (the no-op guard, and now the test-tamper guard), which undid flag_noop_pr's auto-merge disable. Removing it is a safety fix. The inner call still runs whenever cp_rc != 3. The PR body should mention it.
  • info (scripts/dev-lead-fix-reviews.sh:2012): The retraction sweep selects comments by our bot login (plus [bot] variants) and created_at >= PASS_START_ISO, then PATCHes them through gh api with -f body (no shell interpolation). GitHub enforces authorship on PATCH, so a look-alike app login cannot get another user's comment rewritten. The retraction reason is sanitised to ^[a-z][a-z-]*$, so it cannot break out of the HTML comment. No injection surface was found.
  • info (tests/dev-lead/unit/test_fix_reviews.bats:1239): CI_WEAKENING_DETECTED is cleared as a false positive. The matches are the _TTG_SKIP_RE constant and skip strings inside quoted bats fixtures passed to ttg_count_added_skips. The four modified existing tests in test_fix_reviews.bats change fixture claim SHAs from base_sha to the in-pass HEAD, as the new in-pass rule requires, and add stricter assertions that each test is skipped for its original reason. No assertion was weakened or removed.
  • info (scripts/dev-lead-fix-reviews.sh:109): PASS_START_ISO comes from the runner clock at 1-second resolution, and GitHub created_at is compared lexicographically. Clock skew could leave an early reply unretracted. A small negative margin would fail safer.

Reviewed by the PR-review cascade (triage: haiku 4.5 [sonnet 5.5, sonnet 5] → deep: opus 5.5 [opus 4.8, sonnet 5.5] + duck: gemini-3.8-flash [sonnet 5.5] → audit: opus 5.5 [opus 4.8, opus 4.7]). Reply if you need a human review.

Additional tasks

  1. Resolve all unresolved review thread comments from other reviewers
  2. Ensure all CI checks pass after your changes
  3. Rebase on the target branch if behind
  4. Do NOT modify files unrelated to the findings above

The review cascade will automatically re-review after new commits are pushed.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 12 files

Requires human review: Auto-approval blocked because this review re-detected 6 unresolved issues already reported by Cubic.

Re-trigger cubic

Comment thread prompts/dev-lead/fix-bot-comment.md Resolved
Comment thread scripts/lib/test-tamper-guard.sh Resolved
Comment thread scripts/lib/claim-landing.sh Resolved
Comment thread scripts/lib/claim-landing.sh Resolved
Comment thread prompts/dev-lead/fix-bot-comment.md Resolved
Comment thread scripts/lib/test-tamper-guard.sh Resolved
@donpetry-bot donpetry-bot added the needs-human-review Flagged by automated PR review agent label Oct 2, 2026
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-reviews (partial)

A commit was pushed, but not every requested change was applied. Per requested item:

  • scripts/lib/claim-landing.sh:93 — not applied
  • scripts/lib/claim-landing.sh:125 — applied
  • scripts/lib/test-tamper-guard.sh:141 — applied
  • scripts/lib/test-tamper-guard.sh:38 — applied
  • scripts/dev-lead-fix-reviews.sh:1898 — applied
  • scripts/dev-lead-fix-reviews.sh:1991 — not applied
  • scripts/lib/addressed-claim-verify.sh:421 — not applied
  • prompts/dev-lead/fix-reviews.md:58 — not applied
  • scripts/lib/test-tamper-guard.sh:102 — applied
  • scripts/lib/addressed-claim-verify.sh:453 — applied
  • scripts/lib/claim-landing.sh:126 — applied
  • scripts/dev-lead-fix-reviews.sh:109 — not applied
  • scripts/dev-lead-fix-reviews.sh:1937 — not applied
  • docs/architecture/adr/0010-write-mode-personas-on-a-shared-branch.md:35 — applied
  • tests/dev-lead/unit/test_addressed_claim_verify.bats:481 — applied
  • prompts/dev-lead/fix-bot-comment.md:82 — applied
  • scripts/dev-lead-fix-reviews.sh:1898 — applied
  • prompts/dev-lead/fix-bot-comment.md:98 — applied
  • scripts/lib/test-tamper-guard.sh:38 — applied
  • scripts/lib/claim-landing.sh:97 — applied
  • scripts/lib/claim-landing.sh:133 — applied
  • prompts/dev-lead/fix-bot-comment.md:82 — applied
  • scripts/lib/test-tamper-guard.sh:144 — applied
  • tests/dev-lead/unit/test_claim_landing.bats:56 — not applied
  • docs/architecture/adr/0010-write-mode-personas-on-a-shared-branch.md:36 — applied
  • tests/dev-lead/unit/test_test_tamper_guard.bats:76 — not applied

The unaddressed items above still need work.

@don-petry

don-petry commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator Author
Resolved — the `needs-human-review` hold was lifted; dev-lead has picked this item up. Click to expand the prior hold notice.

dev-lead is withholding action on this item.

It is labeled needs-human-review (flagged for human review — this label is applied by automation as well as by people, so an item can become held without anyone noticing), so dev-lead will not pick it up while that label is present. This notice is posted once so the withhold is visible rather than looking like a stalled run.

To re-enable automated pickup: remove the needs-human-review label.

@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-02T14:54:11Z.

@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-02T15:06:31Z.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 10 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread scripts/dev-lead-fix-reviews.sh Resolved
@don-petry don-petry removed the needs-human-review Flagged by automated PR review agent label Oct 2, 2026
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-reviews (partial)

A commit was pushed, but not every requested change was applied. Per requested item:

  • scripts/lib/claim-landing.sh:93 — not applied
  • scripts/lib/claim-landing.sh:125 — not applied
  • scripts/lib/test-tamper-guard.sh:144 — not applied
  • scripts/dev-lead-fix-reviews.sh:2001 — not applied
  • scripts/lib/addressed-claim-verify.sh:421 — not applied
  • scripts/dev-lead-fix-reviews.sh:109 — not applied
  • prompts/dev-lead/fix-bot-comment.md:98 — not applied
  • scripts/dev-lead-fix-reviews.sh:1946 — applied

The unaddressed items above still need work.

@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-02T18:42:26Z.

@don-petry don-petry left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review against the corrected body of issue #2013 ("dev-lead fix-reviews: posts "Fixed" replies for changes that were never pushed, and edits tests to match its own code"). Criteria 1, 4 and 5 are met for fix-reviews and fix-bot-comment. Criteria 2 and 3 are not met; the inline comments say what is required. The issue body's "Remaining work on PR #2019" section has the same list.

Also: the PR description still quotes the old issue text ("From the issue: (observed on ...)"). Update it from the current issue body.

Comment thread scripts/dev-lead-fix-reviews.sh Resolved
Comment thread scripts/dev-lead-fix-reviews.sh Resolved
Comment thread scripts/lib/claim-landing.sh Resolved
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-reviews (partial)

A commit was pushed, but not every requested change was applied. Per requested item:

  • scripts/lib/claim-landing.sh:93 — not applied
  • scripts/lib/claim-landing.sh:125 — not applied
  • scripts/lib/test-tamper-guard.sh:144 — not applied
  • scripts/lib/addressed-claim-verify.sh:421 — not applied
  • scripts/dev-lead-fix-reviews.sh:113 — not applied
  • prompts/dev-lead/fix-bot-comment.md:98 — not applied
  • scripts/dev-lead-fix-reviews.sh:2000 — applied
  • scripts/dev-lead-fix-reviews.sh:2021 — applied
  • docs/architecture/adr/0010-write-mode-personas-on-a-shared-branch.md:22 — not applied
  • scripts/lib/claim-landing.sh:10 — applied

The unaddressed items above still need work.

@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-02T23:32:09Z.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 8 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread scripts/dev-lead-fix-reviews.sh Resolved
Comment thread scripts/lib/test-regression-guard.sh Resolved
Comment thread scripts/lib/test-regression-guard.sh Resolved
Comment thread scripts/lib/test-regression-guard.sh Resolved
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-reviews (partial)

A commit was pushed, but not every requested change was applied. Per requested item:

  • scripts/lib/claim-landing.sh:96 — not applied
  • scripts/lib/claim-landing.sh:128 — not applied
  • scripts/lib/test-tamper-guard.sh:144 — not applied
  • scripts/lib/addressed-claim-verify.sh:421 — not applied
  • scripts/dev-lead-fix-reviews.sh:115 — not applied
  • prompts/dev-lead/fix-bot-comment.md:98 — not applied
  • scripts/dev-lead-fix-reviews.sh — not applied
  • scripts/dev-lead-fix-reviews.sh:2040 — not applied
  • docs/architecture/adr/0010-write-mode-personas-on-a-shared-branch.md — not applied
  • scripts/lib/claim-landing.sh — not applied
  • scripts/dev-lead-fix-reviews.sh:2026 — not applied
  • scripts/lib/test-regression-guard.sh:78 — not applied
  • scripts/lib/test-regression-guard.sh:46 — applied
  • scripts/lib/test-regression-guard.sh:48 — applied
  • tests/dev-lead/unit/test_fix_reviews.bats:5124 — not applied

The unaddressed items above still need work.

@don-petry
don-petry force-pushed the dev-lead/issue-2013-20261002-1249 branch from da692ef to 809b3d9 Compare October 3, 2026 13:47
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — rebase (applied)

Rebase completed and pushed.

@don-petry
don-petry enabled auto-merge (squash) October 3, 2026 13:47
@sonarqubecloud

sonarqubecloud Bot commented Oct 3, 2026

Copy link
Copy Markdown

@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-03T14:49:04Z.

@donpetry-bot donpetry-bot added the needs-human-review Flagged by automated PR review agent label Oct 3, 2026
@donpetry-bot

Copy link
Copy Markdown
Contributor

Automated review — escalated to human

The automated review cascade escalated this PR to a human reviewer at review cycle 1/3 (risk: HIGH, reviewed commit 809b3d9ff5e315d11ca5ebc4f6759186cf6a679f).

Why: the cascade could neither approve the PR nor auto-request fixes, so it requested human review via CODEOWNERS and set the needs-human-review label. A human should review the PR, or remove the needs-human-review label to re-engage the automated cascade.

Reviewer summary: The new test-regression guard runs PR-tree test commands (npm test, make test, pytest, bats, or $DEV_LEAD_TEST_CMD) deterministically inside the dev-lead job. That job has GH_PAT_DON_PETRY (write+workflows) in its job-level GH_TOKEN, plus CLAUDE_CODE_OAUTH_TOKEN and GOOGLE_API_KEY*. The code comment says env -i keeps tokens away from the test process, but that is false: same-uid code can read ancestor environments via /proc//environ. The CI-weakening hard stop is a confirmed false positive (the matches are regex fixtures only), but the secret-isolation problem, the bats/unit checks still in progress, and the reusable-workflow impact on 8 pinned consumers block approval.

This note is updated in place on re-escalation; it is not re-posted.

@don-petry

Copy link
Copy Markdown
Collaborator Author

dev-lead is withholding action on this item.

It is labeled needs-human-review (flagged for human review — this label is applied by automation as well as by people, so an item can become held without anyone noticing), so dev-lead will not pick it up while that label is present. This notice is posted once so the withhold is visible rather than looking like a stalled run.

To re-enable automated pickup: remove the needs-human-review label.

@don-petry don-petry removed the needs-human-review Flagged by automated PR review agent label Oct 3, 2026
@don-petry
don-petry merged commit f67bd98 into main Oct 3, 2026
71 of 74 checks passed
@don-petry
don-petry deleted the dev-lead/issue-2013-20261002-1249 branch October 3, 2026 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dev-lead fix-reviews: posts "Fixed" replies for changes that were never pushed, and edits tests to match its own code

3 participants