Skip to content

feat(github-evidence-guard): warn before redoing work an existing PR covers - #6305

Draft
john-the-dev wants to merge 3 commits into
mainfrom
feat/dup-work-check
Draft

john-the-dev wants to merge 3 commits into
mainfrom
feat/dup-work-check

Conversation

@john-the-dev

@john-the-dev john-the-dev commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Draft until a 12-hour trial reports. This PR becomes ready for review only after a 12h trial on a live core, due 07:24 PT, has been summarised with python3 hooks/_dup_work.py --summary and its false-alarm count is added here.

What changed, and why

This automates the existing manual check. The check that works by hand is gh pr list --state all --search <ticket id>, run on its own, before starting work. The documented misses all come from skipping it or diluting it:

That makes six documented incidents. Duplicate work comes up in the agent's notes on 11 different days since July.

So the ticket-id query runs on its own, in every state, and its hits are always listed first and never cut by a result cap. Keyword and file signals come after it.

Where it hooks in

There is no new hook file. hooks/github-evidence-guard.py already reads gh pr … commands, so this PR adds a --dup-check mode to it. The core registers the same file once more under PreToolUse[Bash] as python3 <hook> --dup-check --repo <checkout>. That registration runs only this check. The hook's existing behaviour, and any other registration of it, is unchanged.

This follows the registration pattern of #6304, but does not depend on it. #6304 is still open.

Read first:

  1. hooks/_dup_work.py: the policy, a helper beside the hook in the _shell_scan.py style. It covers trigger parsing, the single search, matching and ranking, the log, the off switch and --summary.
  2. hooks/github-evidence-guard.py: dup_check() and the --dup-check branch in decide() and main(), about 25 lines.
  3. src/agent/claude/cli/build-core-settings.mjs (arg 9) and session-launch.sh: the registration.
  4. hooks/_shell_scan.py: heredocs(). shell-trap-guard.py and stale-head-post-guard.py each carried their own copy of this heredoc splitter, and this check needs a third. Both now delegate to the shared one. Their behaviour is unchanged and both suites pass.

Behaviour

Triggers:

  • git checkout -b|-B, git switch -c|-C|--create|--force-create and git worktree add -b|-B. It follows cd X && and git -C X.
  • gh pr create.
  • gh pr review <n|url>.

Heredoc bodies are data, so a note that mentions git checkout -b is not a trigger. A heredoc body passed to gh pr create --body is read for ticket ids.

One gh api graphql read per trigger (5 s budget). It carries every search as an alias:

Alias What Scope
tk ticket ids: rm[a-z0-9]{10,14} from the branch, title or body; #NNNN from the branch or title, or after closes/fixes/resolves in the body all states, up to 100 hits
ko / kc keywords from the branch name and title open / closed in the last 30 days
ro the 25 most recently updated open PRs, with files pr create and pr review only

For pr create, the local side is git diff --name-only origin/<base>...HEAD plus git diff -U0 hunk headers. For pr review, it is the reviewed PR's files, read in the same query, and its gh pr diff, read in parallel.

Ranking, in this order:

  1. Ticket-id hits. Every hit is printed. If more exist than were returned, a line gives the total and the search to run.
  2. The same function changed. File-overlap PRs whose @@ hunk-header function matches one of yours. Only the top 3 by shared files are diffed (gh pr diff), inside the same 5 s budget.
  3. Keyword hits. At least two keywords, and at least half of them, must appear in the PR's title or branch. A shared prefix of 4+ characters counts.

File-only overlap is printed only as a count, and only beside another hit. On a busy file it flags many unrelated PRs (see the Replay section).

Excluded: the PR being reviewed, and any PR whose head is the branch being created or pushed.

Weak-input line. When the branch name carries no ticket id, the output adds one line: "No ticket id in the branch name, so the duplicate check is weaker." That is exactly where the one replay miss came from.

Output mechanism. The repo has two kinds of hook output. A denial is permissionDecision: deny. Advice is additionalContext, with exit 0, and the command still runs. This check only ever emits additionalContext. When nothing matches and the branch has a ticket id, it prints nothing.

Never silent on failure. These all log UNKNOWN and print one line, Duplicate-work check could not run (<reason>); search open and recent PRs yourself before continuing.:

  • a gh error or timeout
  • an unreadable response
  • no GitHub origin
  • an unnamed review target
  • an exception in the hook

Logging. Each trigger appends one row to <workspace>/logs/dup-work.jsonl, with the path resolved through workspace_default. Each row has ts, trigger, repo, branch, pr, verdict (FLAG / OK / UNKNOWN), ticket, function, keyword, file_only and elapsed_ms. python3 hooks/_dup_work.py --summary prints counts by verdict and trigger, and p50/p95.

Config. env.DUP_WORK_CHECK in sutando.config.local.json, read through config_get, the same convention as #6304. off (or 0/false/no) turns the check off. Any other value, or no value, leaves it on.

Can this get in the way?

Can it block legitimate work? No. It has no deny path. In the hook, --dup-check returns from decide() before the existing deny branch can run. In main(), an exception in this mode becomes the one-line advisory instead of a denial.

The worst case is a noisy warning that the model reads and ignores. That noise is bounded:

  • Keyword rows are capped at 5.
  • File-only overlap is one count line.
  • Ticket hits are uncapped by design, because truncating them caused the miss above.

The worst case for time is the 5 s budget on a trigger command, and only on a trigger command. The off switch is env.DUP_WORK_CHECK: off.

What existing mechanism already sees this event, and why not extend it?

  • github-evidence-guard.py is the GitHub-evidence hook, so this PR extends it. On main it is not auto-registered for the core, and its gh (pr|run) (view|checks|merge)|gh api filter matches none of these triggers. That is why it needs a registration and a pre-filter of its own.
  • stale-head-post-guard.py sees gh pr review but owns post staleness, and it never sees branch creation.
  • shell-trap-guard.py sees every Bash command, but owns shell portability.

Folding GitHub reads into either of those would mix policies. The trade-off is one more hook process per Bash command (see Cost).

Merge order with #6304. Both PRs register the same file as arg 9. Whichever lands second should fold into one registration, --merge-gate --dup-check, so that both run in one process. decide() would then need to run both branches instead of returning on the first.

Cost

Path Subprocess calls File reads Time
Non-trigger Bash command, in the hook (8,000 decisions) 0 0 1.0 µs/command
Non-trigger, the whole hook process (30 runs) the hook process itself interpreter start p50 25.3 ms, p95 26.5 ms
Branch trigger, stubbed gh 1 gh + 1 local git config, log 1.6 ms excluding the network
Branch trigger, live GitHub (5 branch names, read-only) 1 gh + 1 local git config, log p50 932 ms, max 967 ms
pr review, live (3 PRs) 1 search + up to 4 gh pr diff, in parallel config, log p50 1,942 ms, max 2,354 ms
pr create, live (1, empty local diff) 1 gh + 4 local git config, log 794 ms

The in-hook numbers come from tests/github-evidence-guard-dup-check.test.py (Cost). The live numbers were measured on a laptop against sonichi/sutando.

A non-trigger command returns on a substring check, before _dup_work is imported. The 25 ms process cost is what any new PreToolUse[Bash] registration pays, and #6304 pays it too. Folding the two registrations removes one of them.

The pr review and pr create paths go beyond the single search: they read up to 3 candidate diffs, plus the reviewed PR's own diff for a review. These are bounded by count and by the same 5 s deadline.

Replay of the recorded incidents

Source: shadow-dup-replay.json (generated 2026-11-08T03:36Z). It replayed each incident against GitHub as it stood at the time: only PRs created before as_of, each in the state it had then.

It ran the workspace prototype of this check, not this exact code. This PR keeps the prototype's keyword and ticket logic, and adds the function-overlap ranking the replay showed was needed.

Case Trigger Expected Outcome Matched on
sutando #3964 review review #3877 caught files, but 8 PRs flagged on file overlap alone
sutando #3964 branch branch #3877 caught keywords
sutando #3964 pr-create pr-create #3877 caught keywords + files
web client #1808, branch from ticket title branch #1808 caught keywords
web client #1808, branch with ticket id branch #1808 caught ticket + keywords
desktop app #750, branch from ticket title branch #750 missed 1 of 2 required keywords (usage)
desktop app #750, branch with ticket id branch #750 caught ticket
3 cases that were not PRs (issue / review-guide lesson / scheduled job) – – not replayable the check searches PRs only

The tally: 6 of 7 caught, 1 missed, 3 not replayable.

Two changes came out of the replay:

  • Precision. On the review case, file overlap alone flagged 8 PRs. Only the _one_line function overlap isolated fix(gateway): flatten field values on the separator set readers actually split on #3877. This PR therefore ranks function overlap above keywords, and shows file-only overlap only as a count.
  • The miss. The 2-keyword threshold is kept, because 1 keyword is noise ("desktop", "localhost"). Instead, a branch without a ticket id now gets the weak-input line. The same incident is caught when the branch carries the ticket id.

The Replay test class pins each recorded case as a fixture of what GitHub returned. That includes the #3964 review, where #3877 has to win on function overlap over seven same-file PRs.

Tests

  • python3 tests/github-evidence-guard-dup-check.test.py: 46 tests, OK (also under /usr/bin/python3 3.9 and scripts/hermetic-workspace-guard.py run). Its environment is built with tests/_helpers/hermetic_env.py, so tests/hermetic-env-structural.test.py passes. Line coverage of hooks/_dup_work.py is 100%.
    • Ticket-id match → flagged. The ticket search is all states and separate from the keyword searches.
    • A ticket-id hit survives 20 keyword hits. Ticket hits are listed first, all 30 are listed, and keyword rows are capped at 5.
    • Keyword match → flagged. One keyword is not enough.
    • Function overlap → flagged. File-only overlap is a count, and alone it prints nothing.
    • No match → prints nothing.
    • The reviewed PR and the own branch are excluded. Closed keyword hits older than 30 days are dropped.
    • gh failure → UNKNOWN plus exactly one line. A timeout → UNKNOWN. No GitHub origin → UNKNOWN.
    • Heredocs are data, but feed ticket ids to pr create. The trigger parser handles cd, -C, --create=, worktree, review URLs and env prefixes.
    • One search per trigger. The off switch is silent and logs nothing.
    • Replay of the four replayable incidents.
    • Cost: a non-trigger makes 0 subprocess calls and 0 file reads.
    • Delegation to _dup_work.check. The check runs only before the tool, and the hook is unchanged without the flag. Subprocess end to end with a failing gh, in a hermetic fake checkout. The switch comes from sutando.config. main() never blocks when decide() raises. The --summary CLI.
    • Registration, in both the launcher and the builder.
    • Edges: parser corners, gh or git that cannot start, unreadable responses, an unreadable local diff, a missing review target, an unresolvable log, a config that cannot be read, and ticket totals beyond the first page.
  • node --test tests/agent/claude/cli/build-core-settings.test.ts: 26/26. This adds registration and omission cases.
  • Existing suites, all OK:
    • tests/shell-trap-guard.test.py (53) and tests/stale-head-post-guard.test.py (31), the two hooks whose heredoc splitter moved.
    • tests/pr-decision-evidence.test.py and tests/pr-outcome-evidence.test.py.
    • tests/gdocs-write-guard.test.py and tests/claude-hook-paths-repair.test.py.
    • The launcher suites that source session-launch.sh: claude-plugin-launch-args, start-cli-claude-config-dir, session-launch-named-pids and launch-worker-session-absolute-paths.
  • Lints:
    • bash scripts/review-checks.sh --diff on the full PR diff: PASS.
    • lint-workspace-resolution.sh --diff: clean.
    • gen-src-map.py --check: up to date.
    • ruff==0.15.22: clean.
    • docs_audit.py: PASS.

Feature/documentation consistency

  • hooks/README.md has a new section.
  • docs/github-verification.md has a new paragraph, and its last_verified in docs/catalog.json is bumped.

Known limits

  • It sees one command at a time. A branch made by a script, an alias, or the GitHub web UI is not seen.
  • Keyword matching is literal English tokens. A branch named after the symptom, where the existing PR is named after the cause, needs a ticket id to be caught (the feat(audit-summary): CLI summarizer over voice/call/task/cost ledgers (§5 Next) #750 case).
  • File and function overlap only cover the 25 most recently updated open PRs, and only the top 3 are diffed. On a very busy repo, an older open duplicate can fall outside that window.
  • Function overlap depends on git's hunk-header function context. Files where git names no function (data, Markdown) fall back to file-only overlap.
  • #NNNN in a body counts only after closes/fixes/resolves. A bare "see Notes browser: add markdown rendering #123" is not searched, to keep prior-art mentions from becoming hits.
  • It does not see duplicates that are not PRs: issues, review-guide lessons, or scheduled jobs. Three of the six incidents are of that kind.
  • Codex-core sessions do not load Claude hooks.

Trial plan

Run a live core with this registration for 12 hours. Then post the --summary output here: counts by verdict and trigger, p50/p95, and a hand count of false alarms among the FLAG rows. The PR leaves draft only after that count is in.

Sutando-Task: 20261107-032

🤖 Generated with Claude Code

…covers

Registers github-evidence-guard.py a second time under PreToolUse[Bash]
with --dup-check. Before a new branch, gh pr create or gh pr review it
makes one GitHub search and names PRs that may already cover the work:
ticket-id hits in any state first and untruncated, then PRs changing the
same function, then keyword hits. Advisory only; never denies. A failed
search prints one line and logs UNKNOWN to <workspace>/logs/dup-work.jsonl.

The heredoc splitter that shell-trap-guard and stale-head-post-guard each
carried moves into _shell_scan.heredocs, which the new check also uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sutando-Task: 20261107-032
@john-the-dev john-the-dev added the ag2product AG2 product related (packages, PyPI, task-relay) label Nov 8, 2026

@sonichi-dot sonichi-dot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 2e795bc7de2c0c55955812e88918ff3a0830e7ff.

Not ready to merge: two reproducible silent-miss paths and the new test suite's hermetic-environment CI failure need fixing. The PR also remains draft pending its stated live trial.

Validation: inspected all 12 changed files and their activated paths. Locally passed the 31 duplicate-check tests (also under the workspace guard), 53 shell-trap tests (1 skipped), 31 stale-head tests, 26 settings-builder tests, decision/outcome evidence and related hook suites, four launcher suites, and the full-diff review scanner. Additional probes confirmed the two runtime findings below. Source/test files were verified against this head's Git blob hashes. I did not run the full repository suite or a live-core trial.

The current CI job python standalone tests (2/15) failed in hermetic-env-structural.test.py, explicitly naming the new test at lines 28 and 358. Other CI jobs were still running; CLA was successful.

Worst-case user impact: this always-on hook can silently miss duplicate work, or add noisy/slow advice to branch/PR commands. It has no denial path, and env.DUP_WORK_CHECK: off provides an escape hatch, but failed reads must remain distinguishable from clean results.

Comment thread hooks/_dup_work.py
Comment on lines +230 to +237
def pr_diff(run: Run, repo: str, number: int, timeout: float) -> Optional[str]:
if timeout < 0.3:
return None
try:
p = run(["gh", "pr", "diff", str(number), "-R", repo], capture_output=True, text=True, timeout=timeout)
except (OSError, subprocess.SubprocessError):
return None
return p.stdout if p.returncode == 0 else None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve failed diff reads as UNKNOWN

pr_diff() returns None on an error, timeout, or exhausted budget, and callers pass that to hunk_functions(), which turns it into an empty function set. In review mode there are no keyword/ticket searches, so _one() then reports OK with empty output even when the function comparison never ran. I reproduced this with a same-function candidate: successful reads produce FLAG; changing either the reviewed PR's diff or the candidate's diff to exit 1 with HTTP 502 produces OK and ''. Propagate an unavailable comparison separately from a successful empty diff, and emit/log UNKNOWN (or an explicit incomplete-check warning) rather than silently clearing it.

Comment on lines +61 to +68
DUP_TRIGGER_KEYS = ("checkout -b", "checkout -B", "switch -c", "switch -C", "switch --create",
"switch --force-create", "worktree add", "pr create", "pr review")


def dup_check(data: dict, command: str, argv) -> dict:
"""PreToolUse advice naming PRs that may already cover this work (`--dup-check` registration only)."""
if data.get("hook_event_name", "PreToolUse") != "PreToolUse" or not any(k in command for k in DUP_TRIGGER_KEYS):
return {}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Let all parser-supported commands reach the duplicate check

The literal substring prefilter drops valid triggers before _dup_work.triggers() can recognize them. For example, git checkout --quiet -b fix/rmabcdefghij12-test, git checkout -b fix/rmabcdefghij12-test (two spaces), and gh pr review 9 -R o/r --comment -b x all produce one trigger when parsed directly, but guard.decide() makes zero GitHub calls and returns {}. Thus ordinary flag placement or whitespace silently disables the check. Make this fast filter conservative enough to admit the parser's supported forms and test these through decide(), not only through triggers().

gh = Path(d, "gh")
gh.write_text("#!/bin/sh\necho 'HTTP 502' >&2\nexit 1\n")
gh.chmod(0o755)
env = dict(os.environ, PATH=f"{d}{os.pathsep}{os.environ['PATH']}")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Build the test environments with the existing hermetic helper

The current-head CI failure is from this new suite: hermetic-env-structural.test.py reports :28: unisolated for the module load and :358: copy for dict(os.environ, ...), failing both tree checks. The temporary workspace does not satisfy the repository's separate environment-isolation contract. Use tests/_helpers/hermetic_env.py to isolate the process before loading the hook and construct the child environment, then rerun the structural suite as well as this suite. The failing job's final summary identifies this as its one failed test file: https://github.com/sonichi/sutando/actions/runs/42444120450/job/128619386057

hermetic-env-structural refuses a test that loads modules or spawns
children on the outer environment; isolate at module top and hand the
end-to-end child isolated_child_env().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sutando-Task: 20261107-032

@sonichi-dot sonichi-dot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed cumulative changes at 73a64658fd881861f7bbf15487e7de39e5533572; this update changes only the test environment setup.

The environment-isolation finding is resolved: hermetic-env-structural.test.py passes all 28 tests, and the duplicate-check suite passes all 31 under the workspace guard.

Not ready to merge: the failed diff reads reported as OK and valid triggers skipped by the prefilter remain unchanged and still reproduce on this head. No new findings in this update; keeping those existing threads rather than duplicating them.

Also reran shell-trap (53 tests, 1 skipped), stale-head (31), settings-builder (26), and the full-diff review scanner successfully. Verified reviewed source/test blobs against this SHA. CI is still running, CLA is green, and the stated live trial remains pending. I did not run the full repository suite or a live core.

User impact remains missed duplicate-work advice in an always-on hook; the off switch mitigates noise, not those silent misses.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Gate

✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 93%.

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

  • hooks/_dup_work.py (100%)
  • hooks/_shell_scan.py (100%)
  • hooks/github-evidence-guard.py (100%)
  • hooks/shell-trap-guard.py (100%)
  • hooks/stale-head-post-guard.py (100%)

Summary

  • Total: 427 lines
  • Missing: 0 lines
  • Coverage: 100%

Diff coverage of hooks/_dup_work.py was 85%: unreadable responses,
gh or git that cannot start, an unreadable local diff, a missing review
target, the log resolver and the summary CLI had no test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sutando-Task: 20261107-032
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

@cla-assistant check

@sonichi-dot sonichi-dot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the cumulative 12-file diff at 1bd97c1a8417568c371eccd60c04898ec43b7f90. This update adds 15 tests; production behavior is unchanged.

Not ready to merge: the two existing runtime findings remain reproducible: failed diff reads become OK and valid commands miss the prefilter. No additional findings or duplicate inline threads.

The new test_pr_diff_unavailable_or_out_of_time_is_no_function_signal checks only that the helper returns None. It should also exercise the complete hook and require UNKNOWN plus advice on failure; the existing end-to-end repro still produces OK and empty output. Increased line coverage does not resolve that behavior.

Verified local results: duplicate-check 46 tests (under the workspace guard), environment-isolation 28, shell-trap 53 (1 skipped), stale-head 31, settings-builder 26, and the full-diff scanner all pass. Reviewed blobs match this SHA. Local coverage measurement was unavailable because the coverage module is not installed. Full CI and the live-core trial remain pending; CLA is green.

The user-facing risk remains silent missed warnings in the always-on hook. Its off switch mitigates noise, but the failure paths still need explicit incomplete-check advice.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ag2product AG2 product related (packages, PyPI, task-relay)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants