Skip to content

The review Role's recall on planted faults #581

Open

Description

@don-petry

The review Role finds too few of the faults planted in the review Intent's eval cases. Its recall on the cases with a planted fault is measured well below the 80% per kind that decision 0120 asked, and not every case that expects a blocking Finding is caught yet. factory/v0.1.0 cuts on a bar set below today's measure, as a regression guard: the record replacing 0120's bar.

This story raises that recall without raising false positives on the clean cases. It is part of slice 1, #182.

Approaches to measure. Each is a candidate, not a plan. Any may be dropped if it does not move recall:

  • A pass per changed file. The Role judges each changed file in its own pass, then the passes are merged, so a fault in a file read late is not crowded out.
  • A fault checklist per criterion. Each criterion in the rubric carries a short list of the fault kinds it covers, and the Role answers each item before it judges the criterion met.
  • Reading the head before judging met. The Role reads the changed code at the head, not only the diff, before it may say a criterion is met.
  • Union across runs. The Role runs more than once on the same change, and the Findings are joined. This needs a rule for duplicates, and a cost bound.

How each is measured. A private replay of the held-out set measures each approach against the same commit without it. The replay's results stay private, as the set does; what is published is the approach kept and a rounded change in recall. A replay tunes, so it never stands in for the gate, and the fresh gate set (#562) is never replayed.

Done when:

  • Each approach above is measured by a private replay, and kept or dropped on its result.
  • Recall on the cases with a planted fault rises, and the clean cases still pass at least as the bar asks.
  • A change to the review Role's prompt, rubric or harness ships only with a full replay of the set at its head, never a partial one.

🤖 Addressed by Claude Code

Activity

  1. added 2 commits that reference this issue on Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions