You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The review Role finds too few of the faults planted in the review Intent's eval cases. Its recall on the cases with a planted fault is measured well below the 80% per kind that decision 0120 asked, and not every case that expects a blocking Finding is caught yet. factory/v0.1.0 cuts on a bar set below today's measure, as a regression guard: the record replacing 0120's bar.
This story raises that recall without raising false positives on the clean cases. It is part of slice 1, #182.
Approaches to measure. Each is a candidate, not a plan. Any may be dropped if it does not move recall:
A pass per changed file. The Role judges each changed file in its own pass, then the passes are merged, so a fault in a file read late is not crowded out.
A fault checklist per criterion. Each criterion in the rubric carries a short list of the fault kinds it covers, and the Role answers each item before it judges the criterion met.
Reading the head before judging met. The Role reads the changed code at the head, not only the diff, before it may say a criterion is met.
Union across runs. The Role runs more than once on the same change, and the Findings are joined. This needs a rule for duplicates, and a cost bound.
How each is measured. A private replay of the held-out set measures each approach against the same commit without it. The replay's results stay private, as the set does; what is published is the approach kept and a rounded change in recall. A replay tunes, so it never stands in for the gate, and the fresh gate set (#562) is never replayed.
Done when:
Each approach above is measured by a private replay, and kept or dropped on its result.
Recall on the cases with a planted fault rises, and the clean cases still pass at least as the bar asks.
A change to the review Role's prompt, rubric or harness ships only with a full replay of the set at its head, never a partial one.
Activity
Astra's third read of the #581 switches: a criterion one answer left …
Astra's fourth read of the #581 switches: coverage counted by answer,…