Skip to content

[bug] dev-lead posts a detailed 'Completed' claim before the work is durable, and never retracts it on timeout — #1407 reads as delivered but produced nothing #1445

Closed

Description

@don-petry

Story

As a maintainer or agent reading an issue to decide what still needs doing,
I want a "Completed" claim to be published only once the work is durable, and to be retracted if the run subsequently fails,
so that an issue can never read as delivered when nothing was produced.

Problem — observed on #1407

At 09:21:10Z dev-lead posted a detailed completion record on #1407 titled "## Completed — event-first resume for blocked/rate-limited dev-lead states". It asserts:

  • all 5 acceptance criteria complete, each checkmarked
  • specific test results — "test_dev_lead_resume.bats — 7/7 pass", "Full dev-lead unit suite — 726/726 pass", "validate-interaction-contracts.py (11 contracts) + bats (22) — pass", "shellcheck clean"
  • a files-changed table naming six files, two of them new

At 09:45:23Z — 24 minutes later — the same agent posted status=needs-human attempt=1 reason=timeout: "cannot implement issue #1407".

None of the claimed work exists. Verified against origin/main:

Claimed artifact Reality
scripts/dev-lead-resume.sh (new) absent
tests/dev-lead/unit/test_dev_lead_resume.bats (new) absent
resume_pr_event_first / resume-relay / resume_budget_exhausted zero matches anywhere in scripts/ or .github/workflows/
dev-lead-retry.yml → timer_role: safety-net 0 matches; file last touched by unrelated older PRs
branch or PR for #1407 neither exists

The likely mechanism is benign: the work was done in the runner's workspace, the summary was posted, and a later stage timed out before pushing — so the workspace was discarded. The escalation itself is honest. The defect is ordering and retraction, not dishonesty: a highly specific completion record was published before the work was durable, and nothing withdrew it when the run failed.

Why this is worse than it looks

The claim's specificity is what makes it dangerous. "726/726 pass" and a per-file change table are exactly the details a reader uses to conclude the work is real. #1407 currently reads as delivered while #1408 sits blocked behind work that does not exist.

This is adjacent to the premature-closure defect (#1077, premature-closure-audit.yml) but distinct: that guard catches issues closed as completed with no merged closing PR. Here the issue stays open — so the audit never fires — yet the issue body reads as done. The existing backstop does not cover this shape.

It also sits squarely under the standard's own posture on trustworthy signals: a report that cannot be relied upon is worse than no report, because it suppresses the follow-up that would otherwise happen.

Acceptance Criteria

  1. A completion summary is posted only after the work is durable — commits pushed and a PR opened. Until then, progress reporting must be explicitly provisional (e.g. "in progress", no AC checkmarks, no test tallies presented as results).
  2. Every completion claim references a verifiable artifact — the PR number and/or head commit SHA — so a reader can confirm it in one click and an auditor can check it mechanically.
  3. On terminal failure (timeout, exhaustion, needs-human), any prior completion claim on that issue is retracted — superseded in place with a visible marker (the strike-through-plus-dated-note convention already used in docs/metrics-baseline.md is a good model), not silently left standing.
  4. Test results are only reported as results if the run that produced them is the run that pushed the work. A tally from a discarded workspace must not be presented as evidence.
  5. An audit detects the standing case: an issue carrying a completion claim with no linked merged PR and no matching artifacts on main. Extend scripts/lib/premature-closure-detect.sh / premature-closure-audit.yml rather than adding a parallel workflow — the existing audit is close in shape but only inspects closed issues.
  6. [Phase 4] Demote dev-lead-retry from normal-path driver to a true backstop — resume blocked states event-first #1407's stale completion claim is retracted, and the issue re-enters implementation with a clean record.

Tasks / Subtasks

Dev Notes

Project Structure Notes

scripts/dev-lead-fix-issue.sh (completion/terminal marker posting), scripts/lib/premature-closure-detect.sh + premature-closure-audit.yml (the audit to extend), tests under tests/ registered in lint.yml's bats list.

References

Likely target surface

  • scripts/dev-lead-fix-issue.sh, scripts/lib/premature-closure-detect.sh

Filed from monitoring epic #1402. #1407 reads as fully delivered — five checkmarked ACs, "726/726 pass", a six-file change table — while no branch, PR, or file from that description exists anywhere.

Activity

  1. don-petry commented on Aug 3, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #1445 — [bug] dev-lead posts a detailed 'Completed' claim before the work is durable, and never retracts it on timeout

    Scope

    Make a completion claim publishable only once work is durable (pushed + PR open) and self-retracting on terminal failure, plus a detector that flags open issues carrying an unbacked completion claim.

    Implementation Checklist

    Tests to Write

    • tests/test_premature_closure_detect.bats — unbacked claim fires; a merged PR suppresses it; body_has_completion_claim matches marker + legacy heading; supersede_claim_body prepends dated banner + strikes heading + is idempotent.
    • tests/dev-lead/unit/test_fix_issue.bats — durable completion marker (pr=/sha=) posted only after PR create; on timeout a prior completion claim is retracted (PATCH to the comment); retryable failure does not retract.

    Test Command

    bats tests/test_premature_closure_detect.bats tests/dev-lead/unit/test_fix_issue.bats and bash scripts/dev-lead-lint.sh

    Notes

    AC #6 (retract #1407's stale claim + re-run) is a runtime action on a live issue handled by re-running dev-lead on #1407 with this code (its terminal path now retracts) — flagged for the operator, not performed from this implementation run (no live cross-issue writes here).

  2. don-petry commented on Aug 3, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Progress (in progress — not yet durable)

    Provisional. Nothing is pushed yet; the automation posts the authoritative
    completion record (with PR number + head SHA) after the push. Treat the notes
    below as intent, not delivered results — this is exactly the #1445 discipline
    being implemented here.

    Plan execution (intended)

    Local check output (provisional — NOT the pushed run, not final results)

    scripts/dev-lead-lint.sh              → all checks passed (shellcheck + frontmatter)
    tests/test_premature_closure_detect.bats  → green (new unbacked-claim + retraction cases)
    tests/dev-lead/unit/test_fix_issue.bats   → green (durable-claim / retraction / no-retract-on-retry)
    full tests/*.bats + tests/dev-lead/unit/*.bats + prompt-coverage → green locally
    

    Files touched (this workspace, not yet pushed)

    • prompts/dev-lead/fix-issue.md: Phase 6 made explicitly provisional.
    • scripts/dev-lead-fix-issue.sh: durable post_completion_claim (post-push) + retract_prior_completion_claims (terminal-failure).
    • scripts/lib/premature-closure-detect.sh: pure claim/retraction/report predicates.
    • scripts/premature-closure-audit.sh: open-issue unbacked-claim scan + report + env flags.
    • .github/workflows/premature-closure-audit.yml: header updated to describe the extended scan.
    • tests/test_premature_closure_detect.bats, tests/dev-lead/unit/test_fix_issue.bats: new failing-first tests.

    Notes

    AC #6's cross-issue retraction/re-run is deferred to an operator as a runtime action. The durable "Implementation Complete" record for THIS issue will be posted by the automation after it pushes and opens the PR.

  3. added a commit that references this issue on Aug 3, 2026
  4. don-petry commented on Aug 3, 2026

    @don-petry
    CollaboratorAuthor

    AC #6 was unmet at close — now satisfied manually

    This issue closed at 02:35:27Z with AC #6 outstanding: "#1407's stale completion claim is retracted, and the issue re-enters implementation with a clean record." It had not been.

    Why the shipped fix could not do it. retract_prior_completion_claims is wired into the terminal-failure branch only — the merged code says so explicitly: "Runs on every needs-human branch (missing-binary / timeout / retries-exhausted), never on the retryable path." It is forward-looking by design: it retracts a stale claim the next time dev-lead posts a terminal failure on that issue. #1407 is gated at dev-lead:needs-human, so no further run occurs and the trigger never fires. The claim would have stood indefinitely.

    Resolution. I applied the retraction by hand to comment 5156820419, using the shipped transform verbatim rather than inventing a format — PC_CLAIM_RETRACTED_MARKER banner, then supersede_claim_body's heading strike that preserves the markdown level:

    <!-- dev-lead-claim-retracted -->
    > **[Retracted 2026-08-03 — superseded]** …
    ## ~~Completed — event-first resume for blocked/rate-limited dev-lead states~~ [RETRACTED]
    

    So claim_is_retracted() will now recognise it, and the unbacked-claim audit will stop flagging it.

    The gap worth noting for the future. AC #5's detector (generate_unbacked_claim_report) finds standing unbacked claims, but nothing retracts them — the only retraction path requires a subsequent terminal-failure run on the same issue. Any pre-existing claim on an issue that is closed, abandoned, or human-gated is therefore permanently unreachable by the automation. Worth a follow-up: let the audit's live mode apply the retraction it already knows how to detect, reusing supersede_claim_body.

    Not reopening this issue — the shipped mechanism is correct and AC #6 is now factually satisfied. Flagging the residue instead of quietly closing over it, which is precisely what this issue exists to prevent.


    Generated with Claude Code

  5. added 12 commits that reference this issue on Aug 3, 2026
  6. added a commit that references this issue on Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickupinitiativeEpic / initiative tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions