Skip to content

Reviewer scorecard: bot × repo coverage matrix, outage alerts and quota timing #2506

Closed

Description

@don-petry

Problem

The weekly reviewer scorecard (scripts/reviewer_report.sh, posted to #1220) shows one org-wide row per bot. That hides two kinds of failure that went unnoticed for weeks:

  • A bot stops reviewing everywhere. After the 2026-10-14 org rename, the vendor-side links for gemini-code-assist, codeant-ai and cubic-dev-ai broke. Their reviewed-PR counts fell from about 150, 146 and 110 PRs (Oct 1–15) to 2, 2 and 1 (Oct 16 – Nov 9). The scorecard reported them as "No response", next to bots that are routinely quiet, and nobody acted for about four weeks.
  • A bot never runs on one repo. graphite-app never reviewed combsmith/software-factory from the repo's creation (2026-10-11) until it was enabled on 2026-11-11. Its check suite sat at queued on every head. The org-wide row hid this.

The report also can't show when a bot's quota runs out. gemini-code-assist hit its daily quota at 02:41 UTC on 2026-11-11 and posted 29 refusal notices after that, but the weekly totals can't show that time.

Some cells are off on purpose. For example, coderabbitai is on-demand only on software-factory (its .coderabbit.yaml turns auto review off). Those cells must not raise an alert.

Depends on

#2504 (Reviewer scorecard: classify each bot message so refusals, skips and status notices stop counting as reviews). This issue uses its per-message classes; "reviewed" below means at least one finding or clean message, as #2504 defines it.

Scope

Scorecard only. Don't change gate behaviour. Exclude Dependabot-authored PRs from every count here.

Acceptance

  1. Coverage matrix. A bot × repo table in the report. Each cell shows reviewed / opened PRs for the window. Expected-off cells are declared in the reviewer registry (scripts/lib/reviewer-sources.tsv) or the sidecar Reviewer scorecard: classify each bot message so refusals, skips and status notices stop counting as reviews #2504 chose, and render as off (by design). The software-factory × coderabbitai cell is declared as one.

  2. Outage alerts. An alerts section at the top of the report flags:

    • a bot with 0 reviewed PRs on 2 or more consecutive days, in repos where it is expected on, while at least 3 PRs were opened in those repos on those days;
    • a bot × repo cell (not expected-off) with at least 3 opened PRs and 0 reviewed PRs for the whole window.

    Each alert names the bot, the repo or repos, and the first silent day. With no alerts, the section says "none".

  3. Quota timing. For each bot with refused messages, the report shows refusals per day and the time of the first refusal on each day.

  4. Tests and docs. tests/reviewer_report.bats covers both outage rules with fixtures (an org-wide silence like the rename outage and a single-repo silence like software-factory × graphite-app), an expected-off cell that must not alert, and the quota-timing line. docs/reviewer-report.md explains the matrix, both alert rules, and how to declare an expected-off cell.

Activity

  1. don-petry commented on Nov 14, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #2506 — Reviewer scorecard: bot × repo coverage matrix, outage alerts and quota timing

    Scope

    Add three scorecard-only sections to scripts/reviewer_report.sh: an outage-alerts section at the top, a bot × repo coverage matrix, and per-day quota timing. Expected-off cells are declared in the #2504 sidecar. Gate behaviour does not change, and Dependabot PRs are left out of every new count.

    Implementation Checklist

    • scripts/lib/reviewer-message-classes.tsv: new expected-off row form <login> off repo <owner/repo or repo>; declare coderabbitai × combsmith/software-factory
    • scripts/lib/reviewer-sources.sh: reviewer_sources_message_class_patterns skips off rows; new reviewer_sources_expected_off_cells (validated <login>\t<repo> output). Only the scorecard calls either function.
    • scripts/reviewer_report.sh:
      • normalization: add refused_at (timestamps of refused messages) to bot_pr
      • window start for "opened": REVIEWER_WINDOW_START, else CUTOFF. main computes CUTOFF once so the renderer sees it.
      • _render_outage_alerts: rule 1 (org-wide silence on ≥2 consecutive active days with ≥3 opened PRs in expected-on repos) and rule 2 (a cell that is not expected-off, with ≥3 opened and 0 reviewed). Each alert names the bot, the repo(s) and the first silent day. When there are no alerts, the section says "none". Rendered right after Collection health.
      • _render_coverage_matrix: rows = repos, columns = bots, cells reviewed / opened or off (by design)
      • _render_quota_timing: refusals per day and the first refusal time (UTC) for each bot with refused messages
      • "opened" = non-draft, non-Dependabot PRs created in the window. "reviewed" = at least one finding/clean message (Reviewer scorecard: classify each bot message so refusals, skips and status notices stop counting as reviews #2504).
    • docs/reviewer-report.md: the matrix, both alert rules, and how to declare an expected-off cell

    Tests to Write (tests/reviewer_report.bats)

    • org-wide silence fixture (rename outage) → rule-1 alert naming bot, repos, first silent day
    • single-repo silence fixture (software-factory × graphite-app) → rule-2 alert
    • an expected-off cell (coderabbitai × software-factory) renders off (by design) and raises no alert
    • no alerts → "none"
    • Dependabot PRs excluded from the matrix and the alerts
    • quota-timing line (per-day count and first refusal time)
    • sidecar loader: off rows are skipped by the message-class loader and emitted by the expected-off loader; a bad off row is rejected

    Test Command

    bats tests/reviewer_report.bats tests/test_reviewer_sources.bats + bash .dev-lead/scripts/dev-lead-lint.sh

  2. don-petry commented on Nov 14, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #2544

    The implementation for issue #2506 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  3. added 2 commits that reference this issue on Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupenhancementFeature requestsreviewer-reportWeekly Third-Party Reviewer Scorecard report

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions