Repository navigation
Description
Problem
The weekly reviewer scorecard (scripts/reviewer_report.sh, posted to #1220) shows one org-wide row per bot. That hides two kinds of failure that went unnoticed for weeks:
- A bot stops reviewing everywhere. After the 2026-10-14 org rename, the vendor-side links for gemini-code-assist, codeant-ai and cubic-dev-ai broke. Their reviewed-PR counts fell from about 150, 146 and 110 PRs (Oct 1–15) to 2, 2 and 1 (Oct 16 – Nov 9). The scorecard reported them as "No response", next to bots that are routinely quiet, and nobody acted for about four weeks.
- A bot never runs on one repo. graphite-app never reviewed combsmith/software-factory from the repo's creation (2026-10-11) until it was enabled on 2026-11-11. Its check suite sat at
queuedon every head. The org-wide row hid this.
The report also can't show when a bot's quota runs out. gemini-code-assist hit its daily quota at 02:41 UTC on 2026-11-11 and posted 29 refusal notices after that, but the weekly totals can't show that time.
Some cells are off on purpose. For example, coderabbitai is on-demand only on software-factory (its .coderabbit.yaml turns auto review off). Those cells must not raise an alert.
Depends on
#2504 (Reviewer scorecard: classify each bot message so refusals, skips and status notices stop counting as reviews). This issue uses its per-message classes; "reviewed" below means at least one finding or clean message, as #2504 defines it.
Scope
Scorecard only. Don't change gate behaviour. Exclude Dependabot-authored PRs from every count here.
Acceptance
-
Coverage matrix. A bot × repo table in the report. Each cell shows
reviewed / openedPRs for the window. Expected-off cells are declared in the reviewer registry (scripts/lib/reviewer-sources.tsv) or the sidecar Reviewer scorecard: classify each bot message so refusals, skips and status notices stop counting as reviews #2504 chose, and render asoff (by design). The software-factory × coderabbitai cell is declared as one. -
Outage alerts. An alerts section at the top of the report flags:
- a bot with 0 reviewed PRs on 2 or more consecutive days, in repos where it is expected on, while at least 3 PRs were opened in those repos on those days;
- a bot × repo cell (not expected-off) with at least 3 opened PRs and 0 reviewed PRs for the whole window.
Each alert names the bot, the repo or repos, and the first silent day. With no alerts, the section says "none".
-
Quota timing. For each bot with
refusedmessages, the report shows refusals per day and the time of the first refusal on each day. -
Tests and docs.
tests/reviewer_report.batscovers both outage rules with fixtures (an org-wide silence like the rename outage and a single-repo silence like software-factory × graphite-app), an expected-off cell that must not alert, and the quota-timing line.docs/reviewer-report.mdexplains the matrix, both alert rules, and how to declare an expected-off cell.
Activity
don-petry commentedon Nov 14, 2026
Dev-Lead Implementation Plan
Issue: #2506 — Reviewer scorecard: bot × repo coverage matrix, outage alerts and quota timing
Scope
Add three scorecard-only sections to
scripts/reviewer_report.sh: an outage-alerts section at the top, a bot × repo coverage matrix, and per-day quota timing. Expected-off cells are declared in the #2504 sidecar. Gate behaviour does not change, and Dependabot PRs are left out of every new count.Implementation Checklist
scripts/lib/reviewer-message-classes.tsv: new expected-off row form<login> off repo <owner/repo or repo>; declarecoderabbitai × combsmith/software-factoryscripts/lib/reviewer-sources.sh:reviewer_sources_message_class_patternsskipsoffrows; newreviewer_sources_expected_off_cells(validated<login>\t<repo>output). Only the scorecard calls either function.scripts/reviewer_report.sh:refused_at(timestamps ofrefusedmessages) tobot_prREVIEWER_WINDOW_START, elseCUTOFF.maincomputesCUTOFFonce so the renderer sees it._render_outage_alerts: rule 1 (org-wide silence on ≥2 consecutive active days with ≥3 opened PRs in expected-on repos) and rule 2 (a cell that is not expected-off, with ≥3 opened and 0 reviewed). Each alert names the bot, the repo(s) and the first silent day. When there are no alerts, the section says "none". Rendered right after Collection health._render_coverage_matrix: rows = repos, columns = bots, cellsreviewed / openedoroff (by design)_render_quota_timing: refusals per day and the first refusal time (UTC) for each bot withrefusedmessagesfinding/cleanmessage (Reviewer scorecard: classify each bot message so refusals, skips and status notices stop counting as reviews #2504).docs/reviewer-report.md: the matrix, both alert rules, and how to declare an expected-off cellTests to Write (
tests/reviewer_report.bats)off (by design)and raises no alertoffrows are skipped by the message-class loader and emitted by the expected-off loader; a bad off row is rejectedTest Command
bats tests/reviewer_report.bats tests/test_reviewer_sources.bats+bash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Nov 14, 2026
Dev-Lead: Implementation Complete — PR #2544
The implementation for issue #2506 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.
74071925d6d2ec735722c6c0e38390b78b306ad0main: 5 file(s), +602/-4 linesReview happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.
fix(reviewer-report): keep the headline scorecard counts as they were (…
feat: implement issue #2506 — Reviewer scorecard: bot × repo coverage…