Repository navigation
Description
Problem
The weekly reviewer scorecard (scripts/reviewer_report.sh, posted to #1220) can't tell a refusal from a review. Each bot message is checked against one shared pattern, ADVISORY_RATE_LIMIT_RE (from scripts/lib/advisory-review-gate.sh). A message that matches is a refusal; every other message counts as a real review. For a bot with no formal PR review, the issue-comment fallback in _NORMALIZE_JQ also turns its status, skip and quota comments into "reviews".
So the Reviews, Total PRs and Refused columns are wrong for nearly every bot. Plan and quota decisions made from them are wrong too.
Evidence
The scorecard's own GraphQL collection was replayed for 2026-11-05 to 2026-11-11 (183 PRs), and each message was reclassified by hand-written rules:
| Bot | Scorecard "reviewed" PRs | PRs with findings | Clean pass | Refused (quota) | Skipped by config | Status messages only |
|---|---|---|---|---|---|---|
| chatgpt-codex-connector | 156 | 30 | 2 | 89 | 0 | 39 |
| coderabbitai | 183 | 23 | 0 | 1 | 136 | 23 |
| gemini-code-assist | 60 | 32 | 0 | 27 | 0 | 1 |
| codeant-ai | 46 | 17 | 0 | 1 | 3 | 25 |
| cubic-dev-ai | 36 | 24 | 12 | 0 | 0 | 0 |
| graphite-app | 34 | 5 | 29 | 0 | 1 | 0 |
| sonarqubecloud | 157 | 0 | 155 | 0 | 0 | 2 |
Messages the scorecard currently misreads:
| Bot | Message (start of body) | Counted as | Should be |
|---|---|---|---|
| chatgpt-codex-connector | Codex Review Summary (❌ Failed / ✅ Completed), posted next to each usage-limit notice |
review | status |
| gemini-code-assist | You have reached your daily quota limit (the shared pattern misses "daily quota limit") |
review | refused |
| coderabbitai | Review skipped / Auto reviews are disabled (a repo that turns auto review off on purpose) |
review | skipped |
| codeant-ai | Thanks for using CodeAnt, Review Status, CodeAnt PR Risk |
review | status |
| qodo-code-review | trial has ended |
review | refused |
| graphite-app | check run Graphite / AI Reviews, conclusion neutral, summary AI review skipped: hourly limit reached |
no response | refused |
The snapshot also collects threads_resolved and thumbs reactions, but the report never shows them.
Scope
Scorecard only. Don't change how the advisory review gate or the maintainer-comment gate behave. You can reuse their patterns, but their tests must pass unchanged.
The per-bot patterns can go in a new column in scripts/lib/reviewer-sources.tsv or in a scorecard-only sidecar file. Pick whichever fits the existing loader (scripts/lib/reviewer-sources.sh) better, and say why in the PR. Keep ADVISORY_RATE_LIMIT_RE as the shared fallback for "refused".
Out of scope (separate follow-ups): a bot × repo coverage matrix, and outage or quota-exhaustion alerts.
Acceptance
- Per-message classes. Every bot message (review, inline comment, issue comment, check run) gets exactly one class:
finding,clean,refused,skipped,statusorunclassified, from that bot's patterns first and the shared fallback second. A PR counts as reviewed by a bot only if the bot left at least onefindingorcleanmessage on it. A PR with only status messages, or only unclassified ones, never counts as reviewed. The report lists up to 3 truncated sample bodies per bot forunclassifiedmessages, so new message shapes show up. - Headline table. One row per bot:
Total PRs | With findings | Clean | Refused | Skipped | No response | Findings raised | Resolved % | Latency p50 | Latency p95. Findings raised = inline threads plus findings in review bodies; resolved % = resolved threads out of total threads. The old Reviews event count and the ✅ / 🔄 counts move to a details section. Week-over-week deltas cover the new columns, and a prior snapshot without the new fields shows no delta instead of failing. - Fixtures for every row of the second Evidence table, plus at least one real finding and one clean pass per bot, in
tests/reviewer_report.bats. Each must land in the class in its "Should be" column. The Graphite neutral hourly-limit check run counts asrefused. - Docs.
docs/reviewer-report.mddefines each class and each column, and says how to add patterns for a new bot.
Accepted changes (maintainer, 2026-11-13)
Recorded after the forced review of PR #2505 asked for a decision. Both changes are accepted for this issue:
- Acceptance 1: the shared
ADVISORY_RATE_LIMIT_REfallback is not applied to inline comments, so a finding that mentions a rate limit stays a finding. - Acceptance 2: Findings raised counts review-body findings only on a PR where the bot opened no inline thread; when it did, the body is treated as a summary of its threads. This undercounts CodeRabbit's outside-diff and nitpick findings and Codex's body-only suggestions. Counting those is tracked in Reviewer scorecard: count summary-only findings for CodeRabbit and Codex, and keep status/skip rows off inline comments #2527, together with keeping
statusandskippedsidecar rows off inline comments.
Activity
don-petry commentedon Nov 12, 2026
Dev-Lead Implementation Plan
Issue: #2504 — Reviewer scorecard: classify each bot message so refusals, skips and status notices stop counting as reviews
Scope
Give every bot message (review, inline comment, issue comment, completed check run) exactly one class (
finding/clean/refused/skipped/status/unclassified). The order is: that bot's patterns first, then the sharedADVISORY_RATE_LIMIT_REfallback (refused), then structural defaults. A PR counts as reviewed only when the bot left afindingorcleanmessage on it. The headline table gets rebuilt around those PR buckets. This is scorecard only: the advisory gate and the maintainer-comment gate are not touched.Design choice: sidecar file, not a new registry column
Patterns go in a new scorecard-only file,
scripts/lib/reviewer-message-classes.tsv(login class kind pattern, one row per pattern, first match wins). It is read by a newreviewer_sources_message_class_patternshelper in the existing loader. Reasons:reviewer-sources.tsvhas a fixed column layout (schema_version 3,rationaleread by position as$7) and five consumers, including both gates.info_status_pattern, the finding-section pattern) match markdownbody. The scorecard reads GraphQLbodyText, where HTML comments and markdown markup are already stripped, so those patterns can't be reused here as they are.Implementation Checklist
scripts/lib/reviewer-message-classes.tsv: per-bot patterns for all 9 bots, covering every message in the issue's "misreads" table plus a finding and a clean pass per botscripts/lib/reviewer-sources.sh:reviewer_sources_message_class_patternsloader that validates class and kindscripts/reviewer_report.sh:_NORMALIZE_JQrefusedfindings_raised, thread-levelthreads_total/threads_resolved, and up to 3 truncatedunclassifiedsamples per botaggregate_snapshot: findings / clean / refused / skipped / no-response PR buckets, plusschema_version: 2docs/reviewer-report.md: what each class and column means, and how to add patterns for a new botTests to Write (
tests/reviewer_report.bats)tests/fixtures/reviewer_message_classes.json): every row of the "misreads" table, plus a finding and a clean pass for each bot, each landing in its expected classrefusedtest_advisory_review_gate.bats,test_maintainer_comment_gate.bats,test_reviewer_sources.bats) run unchangedTest Command
bats tests/reviewer_report.bats tests/test_reviewer_sources.bats tests/dev-lead/unit/test_advisory_review_gate.bats tests/dev-lead/unit/test_maintainer_comment_gate.bats+bash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Nov 12, 2026
Dev-Lead: Progress (in progress — not yet durable)
Plan Execution (intended)
_NORMALIZE_JQ, in this order: the bot's sidecar rows, then the sharedADVISORY_RATE_LIMIT_REfallback, then structural defaults, elseunclassified. Completed check runs of any conclusion are classified.schema_version: 2. Headline deltas only compare against a v2 snapshot.Files touched (this workspace, not yet pushed)
scripts/lib/reviewer-message-classes.tsv(new): scorecard-only patterns, one per row:login class kind pattern, first match winsscripts/lib/reviewer-sources.sh:reviewer_sources_message_class_patternsloader (validates class and kind)scripts/reviewer_report.sh: classification, aggregation, renderingtests/reviewer_report.bats,tests/fixtures/reviewer_message_classes.json(new),tests/fixtures/reviewer_pr_node.jsondocs/reviewer-report.mdNotes
reviewer-sources.tsvhas a fixed column layout, and itsrationalecolumn is read by position ($7) by five consumers, both gates included. A bot needs many patterns across classes and message kinds. The registry patterns also match the markdownbody, but the scorecard readsbodyText.statusrecord (still No response). The "Codex discussing cubic's trial" test now asserts "not refused" rather than "reviewed", because an unclassified comment no longer counts as a review. The copilot body inreviewer_pr_node.jsonwas changed to a realistic overview, so its latency is still measured from the review.@mentionsnever notify anyone. They are not stored in the snapshot artifact. But they can still quote a few words from a bot comment on a private repo. If that is not acceptable, the samples could go only to the step summary.test_batch_fallback,test_model_sandbox,test_review_one_pr_*,test_pr_approval_diagnostic) fail the same way on unchangedmain.don-petry commentedon Nov 12, 2026
Dev-Lead: Implementation Complete — PR #2505
The implementation for issue #2504 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.
aa935b917c2a98d5bf4b6b20782880d047cca2eamain: 7 file(s), +1061/-163 linesReview happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.
fix(reviews): allow bot pattern rows of all classes to apply to inlin…
fix(reviews): classify CodeAnt Nitpicks code-suggestion lists as find…
feat: implement issue #2504 — Reviewer scorecard: classify each bot m…
fix(reviews): allow bot pattern rows of all classes to apply to inlin…
fix(reviews): classify CodeAnt Nitpicks code-suggestion lists as find…
feat: implement issue #2504 — Reviewer scorecard: classify each bot m…
feat(reviewer-report): count CodeRabbit and Codex body findings; keep…