Skip to content

Reviewer scorecard: classify each bot message so refusals, skips and status notices stop counting as reviews #2504

Closed

Description

@don-petry

Problem

The weekly reviewer scorecard (scripts/reviewer_report.sh, posted to #1220) can't tell a refusal from a review. Each bot message is checked against one shared pattern, ADVISORY_RATE_LIMIT_RE (from scripts/lib/advisory-review-gate.sh). A message that matches is a refusal; every other message counts as a real review. For a bot with no formal PR review, the issue-comment fallback in _NORMALIZE_JQ also turns its status, skip and quota comments into "reviews".

So the Reviews, Total PRs and Refused columns are wrong for nearly every bot. Plan and quota decisions made from them are wrong too.

Evidence

The scorecard's own GraphQL collection was replayed for 2026-11-05 to 2026-11-11 (183 PRs), and each message was reclassified by hand-written rules:

Bot Scorecard "reviewed" PRs PRs with findings Clean pass Refused (quota) Skipped by config Status messages only
chatgpt-codex-connector 156 30 2 89 0 39
coderabbitai 183 23 0 1 136 23
gemini-code-assist 60 32 0 27 0 1
codeant-ai 46 17 0 1 3 25
cubic-dev-ai 36 24 12 0 0 0
graphite-app 34 5 29 0 1 0
sonarqubecloud 157 0 155 0 0 2

Messages the scorecard currently misreads:

Bot Message (start of body) Counted as Should be
chatgpt-codex-connector Codex Review Summary (❌ Failed / ✅ Completed), posted next to each usage-limit notice review status
gemini-code-assist You have reached your daily quota limit (the shared pattern misses "daily quota limit") review refused
coderabbitai Review skipped / Auto reviews are disabled (a repo that turns auto review off on purpose) review skipped
codeant-ai Thanks for using CodeAnt, Review Status, CodeAnt PR Risk review status
qodo-code-review trial has ended review refused
graphite-app check run Graphite / AI Reviews, conclusion neutral, summary AI review skipped: hourly limit reached no response refused

The snapshot also collects threads_resolved and thumbs reactions, but the report never shows them.

Scope

Scorecard only. Don't change how the advisory review gate or the maintainer-comment gate behave. You can reuse their patterns, but their tests must pass unchanged.

The per-bot patterns can go in a new column in scripts/lib/reviewer-sources.tsv or in a scorecard-only sidecar file. Pick whichever fits the existing loader (scripts/lib/reviewer-sources.sh) better, and say why in the PR. Keep ADVISORY_RATE_LIMIT_RE as the shared fallback for "refused".

Out of scope (separate follow-ups): a bot × repo coverage matrix, and outage or quota-exhaustion alerts.

Acceptance

  1. Per-message classes. Every bot message (review, inline comment, issue comment, check run) gets exactly one class: finding, clean, refused, skipped, status or unclassified, from that bot's patterns first and the shared fallback second. A PR counts as reviewed by a bot only if the bot left at least one finding or clean message on it. A PR with only status messages, or only unclassified ones, never counts as reviewed. The report lists up to 3 truncated sample bodies per bot for unclassified messages, so new message shapes show up.
  2. Headline table. One row per bot: Total PRs | With findings | Clean | Refused | Skipped | No response | Findings raised | Resolved % | Latency p50 | Latency p95. Findings raised = inline threads plus findings in review bodies; resolved % = resolved threads out of total threads. The old Reviews event count and the ✅ / 🔄 counts move to a details section. Week-over-week deltas cover the new columns, and a prior snapshot without the new fields shows no delta instead of failing.
  3. Fixtures for every row of the second Evidence table, plus at least one real finding and one clean pass per bot, in tests/reviewer_report.bats. Each must land in the class in its "Should be" column. The Graphite neutral hourly-limit check run counts as refused.
  4. Docs. docs/reviewer-report.md defines each class and each column, and says how to add patterns for a new bot.

Accepted changes (maintainer, 2026-11-13)

Recorded after the forced review of PR #2505 asked for a decision. Both changes are accepted for this issue:

  • Acceptance 1: the shared ADVISORY_RATE_LIMIT_RE fallback is not applied to inline comments, so a finding that mentions a rate limit stays a finding.
  • Acceptance 2: Findings raised counts review-body findings only on a PR where the bot opened no inline thread; when it did, the body is treated as a summary of its threads. This undercounts CodeRabbit's outside-diff and nitpick findings and Codex's body-only suggestions. Counting those is tracked in Reviewer scorecard: count summary-only findings for CodeRabbit and Codex, and keep status/skip rows off inline comments #2527, together with keeping status and skipped sidecar rows off inline comments.

Activity

  1. don-petry commented on Nov 12, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #2504 — Reviewer scorecard: classify each bot message so refusals, skips and status notices stop counting as reviews

    Scope

    Give every bot message (review, inline comment, issue comment, completed check run) exactly one class (finding / clean / refused / skipped / status / unclassified). The order is: that bot's patterns first, then the shared ADVISORY_RATE_LIMIT_RE fallback (refused), then structural defaults. A PR counts as reviewed only when the bot left a finding or clean message on it. The headline table gets rebuilt around those PR buckets. This is scorecard only: the advisory gate and the maintainer-comment gate are not touched.

    Design choice: sidecar file, not a new registry column

    Patterns go in a new scorecard-only file, scripts/lib/reviewer-message-classes.tsv (login class kind pattern, one row per pattern, first match wins). It is read by a new reviewer_sources_message_class_patterns helper in the existing loader. Reasons:

    • reviewer-sources.tsv has a fixed column layout (schema_version 3, rationale read by position as $7) and five consumers, including both gates.
    • One bot needs many patterns across several classes and message kinds, and those don't fit in a single cell.
    • The registry patterns (info_status_pattern, the finding-section pattern) match markdown body. The scorecard reads GraphQL bodyText, where HTML comments and markdown markup are already stripped, so those patterns can't be reused here as they are.

    Implementation Checklist

    • scripts/lib/reviewer-message-classes.tsv: per-bot patterns for all 9 bots, covering every message in the issue's "misreads" table plus a finding and a clean pass per bot
    • scripts/lib/reviewer-sources.sh: reviewer_sources_message_class_patterns loader that validates class and kind
    • scripts/reviewer_report.sh:
      • per-message classification in _NORMALIZE_JQ
      • completed check runs of any conclusion are classified, so a neutral "hourly limit reached" run counts as refused
      • new per-class counts, findings_raised, thread-level threads_total / threads_resolved, and up to 3 truncated unclassified samples per bot
      • aggregate_snapshot: findings / clean / refused / skipped / no-response PR buckets, plus schema_version: 2
      • new headline table, a details section (Reviews events, ✅/🔄, thumbs), an unclassified-samples section, and WoW deltas that only compare against a prior snapshot carrying the new fields
      • coverage overlap counts only reviewing bots
    • docs/reviewer-report.md: what each class and column means, and how to add patterns for a new bot

    Tests to Write (tests/reviewer_report.bats)

    • Data-driven fixtures (tests/fixtures/reviewer_message_classes.json): every row of the "misreads" table, plus a finding and a clean pass for each bot, each landing in its expected class
    • A PR with only status or only unclassified messages is not reviewed
    • Graphite's neutral hourly-limit check run counts as refused
    • Headline columns; details section; unclassified samples (≤3, truncated)
    • WoW deltas on the new columns; a prior snapshot without the new fields shows no delta and does not fail
    • Sidecar validation: every login is registered, every pattern compiles, bad class or kind is rejected
    • Gate suites (test_advisory_review_gate.bats, test_maintainer_comment_gate.bats, test_reviewer_sources.bats) run unchanged

    Test Command

    bats tests/reviewer_report.bats tests/test_reviewer_sources.bats tests/dev-lead/unit/test_advisory_review_gate.bats tests/dev-lead/unit/test_maintainer_comment_gate.bats + bash .dev-lead/scripts/dev-lead-lint.sh

  2. don-petry commented on Nov 12, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Progress (in progress — not yet durable)

    Provisional. Nothing is pushed yet; the automation posts the authoritative
    completion record (with PR number + head SHA) after the push. Treat the notes
    below as intent, not delivered results.

    Plan Execution (intended)

    • Per-message classification in _NORMALIZE_JQ, in this order: the bot's sidecar rows, then the shared ADVISORY_RATE_LIMIT_RE fallback, then structural defaults, else unclassified. Completed check runs of any conclusion are classified.
    • New PR buckets and headline table, plus a Details section (Reviews, ✅/🔄, refusal events, 👍/👎) and an Unclassified-samples section.
    • Snapshot schema_version: 2. Headline deltas only compare against a v2 snapshot.

    Files touched (this workspace, not yet pushed)

    • scripts/lib/reviewer-message-classes.tsv (new): scorecard-only patterns, one per row: login class kind pattern, first match wins
    • scripts/lib/reviewer-sources.sh: reviewer_sources_message_class_patterns loader (validates class and kind)
    • scripts/reviewer_report.sh: classification, aggregation, rendering
    • tests/reviewer_report.bats, tests/fixtures/reviewer_message_classes.json (new), tests/fixtures/reviewer_pr_node.json
    • docs/reviewer-report.md

    Notes

    • Why a sidecar file and not a new column: reviewer-sources.tsv has a fixed column layout, and its rationale column is read by position ($7) by five consumers, both gates included. A bot needs many patterns across classes and message kinds. The registry patterns also match the markdown body, but the scorecard reads bodyText.
    • Existing test expectations changed where the issue changes behavior. The old header check now checks the Details table. Failed or empty-summary check runs used to produce no record and now produce a status record (still No response). The "Codex discussing cubic's trial" test now asserts "not refused" rather than "reviewed", because an unclassified comment no longer counts as a review. The copilot body in reviewer_pr_node.json was changed to a realistic overview, so its latency is still measured from the review.
    • Please review: unclassified samples go into a public issue comment. The report posts up to 3 bot message bodies per bot to 🔎 Third-Party Reviewer Scorecard — Weekly Report #1220, and this repo is public. The samples are cut to 100 characters, have URLs and backticks removed, and sit inside code spans so @mentions never notify anyone. They are not stored in the snapshot artifact. But they can still quote a few words from a bot comment on a private repo. If that is not acceptable, the samples could go only to the step summary.
    • The sidecar's starting patterns for message shapes not listed in the issue come from known bot wording, not from live data. The Unclassified section is there to show what they miss.
    • 33 local failures in unrelated suites (test_batch_fallback, test_model_sandbox, test_review_one_pr_*, test_pr_approval_diagnostic) fail the same way on unchanged main.
  3. don-petry commented on Nov 12, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #2505

    The implementation for issue #2504 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  4. added a commit that references this issue on Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupenhancementFeature requestsreviewer-reportWeekly Third-Party Reviewer Scorecard report

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions