Skip to content

πŸ”Ž Third-Party Reviewer Scorecard β€” Weekly ReportΒ #1220

Open

Description

@github-actions

Tracking issue for the weekly Third-Party Reviewer Scorecard. Each run posts a new comment below measuring the external agentic PR reviewers (Copilot, Gemini, Codex, CodeRabbit, SonarCloud) across the org.

Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

Activity

  1. github-actions commented on Jul 14, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-07-14 Β· org petry-projects Β· 11 repos scanned Β· 191 PRs active (187 non-draft, review-eligible)

    Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Participation = distinct non-draft PRs a bot touched Γ· 187 review-eligible PRs. Latency = time from PR creation to the bot's first review/comment. Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer PRs reviewed Participation Reviews Comments/PR Latency p50 Latency p95 βœ… / πŸ”„ Rate-limited Resolved πŸ‘/πŸ‘Ž
    Gemini Code Assist 143 76% 85 1.2 63s 3m 0 / 0 9 86% 0/0
    GitHub Copilot 106 57% 106 1.8 3m 3.0h 0 / 0 6 56% 0/0
    SonarCloud 177 95% 0 0.0 6m 8.0d 0 / 0 0 n/a 0/0
    Codex 160 86% 30 0.4 5s 4m 0 / 0 154 86% 0/0
    CodeRabbit 177 95% 229 0.6 9s 22s 52 / 34 147 97% 0/0

    βœ… = APPROVED reviews Β· πŸ”„ = CHANGES_REQUESTED Β· Resolved = share of a bot's inline threads later resolved Β· Comments/PR = inline findings per reviewed PR.

    Reliability & no-shows

    Reviewer Eligible PRs missed Coverage gap Rate-limited PRs
    Gemini Code Assist 44 24% 9
    GitHub Copilot 81 43% 6
    SonarCloud 10 5% 0
    Codex 27 14% 154
    CodeRabbit 10 5% 147

    Coverage gap = eligible PRs a bot did not touch. External SaaS reviewers only sample a subset of PRs; this is a participation signal, not necessarily a fault.

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 177 of 187 eligible (95%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate requires judgment and is intentionally out of this deterministic report. It is a separate, human-approved LLM add-on.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  2. github-actions commented on Jul 14, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-07-14 Β· org petry-projects Β· 11 repos scanned Β· 181 PRs active (177 non-draft, review-eligible)

    Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Rate-limited count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Rate-limited No response Latency p50 Latency p95
    Gemini Code Assist 177 132 (β–² +47) 0 / 0 8 42 42s 3m
    GitHub Copilot 177 116 (β–² +10) 0 / 0 11 56 3m 2.8h
    SonarCloud 177 167 (β–² +167) 0 / 0 0 12 7m 8.1d
    Codex 177 19 (β–Ό -11) 0 / 0 192 26 4m 10m
    CodeRabbit 177 190 (β–Ό -39) 53 / 23 148 13 5m 1.6h
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Rate-limited β€” count of out-of-quota / rate-limit refusals, each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 168 of 177 eligible (95%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate requires judgment and is intentionally out of this deterministic report. It is a separate, human-approved LLM add-on.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  3. github-actions commented on Jul 20, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-07-20 Β· org petry-projects Β· 11 repos scanned Β· 157 PRs active (153 non-draft, review-eligible)

    Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Rate-limited count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Rate-limited No response Latency p50 Latency p95
    Gemini Code Assist 153 122 0 / 0 1 31 12s 2m
    GitHub Copilot 153 119 0 / 0 2 34 4m 34m
    SonarCloud 153 141 0 / 0 0 15 10m 7.2d
    Codex 153 5 0 / 0 162 28 4m 8m
    CodeRabbit 153 98 27 / 11 118 20 3m 84m
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Rate-limited β€” count of out-of-quota / rate-limit refusals, each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 138 of 153 eligible (90%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate requires judgment and is intentionally out of this deterministic report. It is a separate, human-approved LLM add-on.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  4. github-actions commented on Jul 27, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-07-27 Β· org petry-projects Β· 11 repos scanned Β· 207 PRs active (206 non-draft, review-eligible)

    Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Rate-limited count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Rate-limited No response Latency p50 Latency p95
    Gemini Code Assist 206 180 0 / 0 3 26 7s 3m
    GitHub Copilot 206 122 1 / 0 2 84 5m 32m
    SonarCloud 206 189 0 / 0 0 17 8m 2.8d
    Codex 206 0 0 / 0 185 24 β€” β€”
    CodeRabbit 206 156 31 / 12 156 30 4m 25m
    Qodo Merge 206 84 0 / 0 3 144 62s 2m
    CodeAnt 206 79 0 / 0 0 130 4s 3m
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Rate-limited β€” count of out-of-quota / rate-limit refusals, each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 184 of 206 eligible (89%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate requires judgment and is intentionally out of this deterministic report. It is a separate, human-approved LLM add-on.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  5. github-actions commented on Aug 3, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-08-03 Β· org petry-projects Β· 12 repos scanned Β· 35 PRs active (35 non-draft, review-eligible)

    Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Rate-limited count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Rate-limited No response Latency p50 Latency p95
    Gemini Code Assist 35 30 0 / 0 1 5 50s 2m
    GitHub Copilot 35 14 0 / 0 0 21 7m 13m
    SonarCloud 35 30 0 / 0 0 5 7.0d 12.7d
    Codex 35 0 0 / 0 30 5 β€” β€”
    CodeRabbit 35 53 8 / 4 17 5 56s 11.8d
    Qodo Merge 35 21 0 / 0 4 20 57s 2m
    CodeAnt 35 38 0 / 0 0 3 5s 11.8d
    Graphite 35 2 0 / 0 0 33 4m 12.7d
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Rate-limited β€” count of out-of-quota / rate-limit refusals, each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 31 of 35 eligible (89%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Agent comment noise

    • Agent comments: 0

    • No-action comments: 0 (n/a of agent comments)

    • Active PRs (window): 35

    • PRs with agent comments: 0

    • Affected PRs (β‰₯1 no-action comment): 0

    • No-action comments per PR: 0.00

    • Noise = agent comments that ask nothing of a human β€” a reported no-change, a no-op engine run, or a clean/repeat approval. Actionable agent comments (applied fixes, escalated findings, human-attention flags) are excluded. Classification is deterministic (scripts/lib/comment-noise.sh), computed from the automation markers each comment already carries β€” no LLM.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate for the third-party reviewers requires judgment and is intentionally out of this deterministic report (a separate, human-approved LLM add-on). Note the Agent comment noise section above is a different, deterministic metric β€” it scores OUR automation's no-action share from the markers each comment already carries.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  6. github-actions commented on Aug 10, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-08-10 Β· org petry-projects Β· 12 repos scanned Β· 32 PRs active (32 non-draft, review-eligible)

    Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Rate-limited count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Rate-limited No response Latency p50 Latency p95
    Gemini Code Assist 32 31 0 / 0 0 1 50s 2m
    GitHub Copilot 32 9 0 / 0 0 23 7m 13m
    SonarCloud 32 30 0 / 0 0 2 68m 16.2d
    Codex 32 0 0 / 0 31 1 β€” β€”
    CodeRabbit 32 48 5 / 4 16 2 20s 7.0d
    Qodo Merge 32 33 0 / 0 2 6 11s 14.1d
    CodeAnt 32 41 0 / 0 0 2 5s 12.8d
    Graphite 32 5 0 / 0 0 27 12.7d 14.1d
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Rate-limited β€” count of out-of-quota / rate-limit refusals, each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 31 of 32 eligible (97%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Agent comment noise

    • Agent comments: 0

    • No-action comments: 0 (n/a of agent comments)

    • Active PRs (window): 32

    • PRs with agent comments: 0

    • Affected PRs (β‰₯1 no-action comment): 0

    • No-action comments per PR: 0.00

    • Noise = agent comments that ask nothing of a human β€” a reported no-change, a no-op engine run, or a clean/repeat approval. Actionable agent comments (applied fixes, escalated findings, human-attention flags) are excluded. Classification is deterministic (scripts/lib/comment-noise.sh), computed from the automation markers each comment already carries β€” no LLM.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate for the third-party reviewers requires judgment and is intentionally out of this deterministic report (a separate, human-approved LLM add-on). Note the Agent comment noise section above is a different, deterministic metric β€” it scores OUR automation's no-action share from the markers each comment already carries.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  7. github-actions commented on Aug 17, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-08-17 Β· org petry-projects Β· 12 repos scanned Β· 8 PRs active (7 non-draft, review-eligible)

    Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Rate-limited count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Rate-limited No response Latency p50 Latency p95
    GitHub Copilot 7 0 0 / 0 0 7 β€” β€”
    Gemini Code Assist 7 6 0 / 0 0 1 67s 2m
    Codex 7 0 0 / 0 6 1 β€” β€”
    CodeRabbit 7 8 0 / 0 6 1 3m 9m
    SonarCloud 7 7 0 / 0 0 0 24.4h 10.1d
    Qodo Merge 7 6 0 / 0 0 1 9s 11s
    CodeAnt 7 8 0 / 0 0 0 5s 7s
    Graphite 7 1 0 / 0 0 6 16m 16m
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Rate-limited β€” count of out-of-quota / rate-limit refusals, each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 8 of 7 eligible (114%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Agent comment noise

    • Agent comments: 0

    • No-action comments: 0 (n/a of agent comments)

    • Active PRs (window): 8

    • PRs with agent comments: 0

    • Affected PRs (β‰₯1 no-action comment): 0

    • No-action comments per PR: 0.00

    • Noise = agent comments that ask nothing of a human β€” a reported no-change, a no-op engine run, or a clean/repeat approval. Actionable agent comments (applied fixes, escalated findings, human-attention flags) are excluded. Classification is deterministic (scripts/lib/comment-noise.sh), computed from the automation markers each comment already carries β€” no LLM.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate for the third-party reviewers requires judgment and is intentionally out of this deterministic report (a separate, human-approved LLM add-on). Note the Agent comment noise section above is a different, deterministic metric β€” it scores OUR automation's no-action share from the markers each comment already carries.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  8. github-actions commented on Aug 24, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-08-24 Β· org petry-projects Β· 12 repos scanned Β· 23 PRs active (23 non-draft, review-eligible)

    Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Rate-limited count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Rate-limited No response Latency p50 Latency p95
    GitHub Copilot 23 4 0 / 0 0 19 5m 13m
    Gemini Code Assist 23 22 0 / 0 0 1 38s 2m
    Codex 23 0 0 / 0 22 1 β€” β€”
    CodeRabbit 23 19 3 / 1 10 1 19s 2m
    SonarCloud 23 21 0 / 0 0 2 1.6h 28.2d
    Qodo Merge 23 24 0 / 0 0 3 7s 14.1d
    CodeAnt 23 43 0 / 0 0 1 5s 12.7d
    Graphite 23 2 0 / 0 0 21 9m 12.7d
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Rate-limited β€” count of out-of-quota / rate-limit refusals, each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 22 of 23 eligible (96%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Agent comment noise

    • Agent comments: 0

    • No-action comments: 0 (n/a of agent comments)

    • Active PRs (window): 23

    • PRs with agent comments: 0

    • Affected PRs (β‰₯1 no-action comment): 0

    • No-action comments per PR: 0.00

    • Noise = agent comments that ask nothing of a human β€” a reported no-change, a no-op engine run, or a clean/repeat approval. Actionable agent comments (applied fixes, escalated findings, human-attention flags) are excluded. Classification is deterministic (scripts/lib/comment-noise.sh), computed from the automation markers each comment already carries β€” no LLM.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate for the third-party reviewers requires judgment and is intentionally out of this deterministic report (a separate, human-approved LLM add-on). Note the Agent comment noise section above is a different, deterministic metric β€” it scores OUR automation's no-action share from the markers each comment already carries.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  9. github-actions commented on Aug 31, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-08-31 Β· org petry-projects Β· 12 repos scanned Β· 6 PRs active (6 non-draft, review-eligible)

    Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Rate-limited count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Rate-limited No response Latency p50 Latency p95
    GitHub Copilot 6 0 0 / 0 0 6 β€” β€”
    Gemini Code Assist 6 5 0 / 0 0 1 30s 3m
    Codex 6 0 0 / 0 5 1 β€” β€”
    CodeRabbit 6 6 0 / 0 2 1 19s 8m
    SonarCloud 6 6 0 / 0 0 0 2.7h 3.3d
    Qodo Merge 6 5 0 / 0 0 1 5s 6s
    CodeAnt 6 11 0 / 0 0 0 5s 6s
    Graphite 6 0 0 / 0 0 6 β€” β€”
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Rate-limited β€” count of out-of-quota / rate-limit refusals, each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 6 of 6 eligible (100%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Agent comment noise

    • Agent comments: 0

    • No-action comments: 0 (n/a of agent comments)

    • Active PRs (window): 6

    • PRs with agent comments: 0

    • Affected PRs (β‰₯1 no-action comment): 0

    • No-action comments per PR: 0.00

    • Noise = agent comments that ask nothing of a human β€” a reported no-change, a no-op engine run, or a clean/repeat approval. Actionable agent comments (applied fixes, escalated findings, human-attention flags) are excluded. Classification is deterministic (scripts/lib/comment-noise.sh), computed from the automation markers each comment already carries β€” no LLM.

    pr-review miss rate

    The false-NEGATIVE signal (#1596): accepted defects a trusted advisory bot found on a PR pr-review had already approved. Deterministic β€” computed with jq from review data, no LLM. Only findings resolved as accepted count; refuted false positives and ambiguous dispositions do not.

    • Miss rate: 0% β€” 0 missed vs 0 caught (accepted findings; miss = found first by a bot after approval).
    • Approvals on partial evidence: 0 β€” pr-review approved via the advisory-gate timeout fallback (before all bots reported).
    • Approved PRs in window: 0.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate for the third-party reviewers requires judgment and is intentionally out of this deterministic report (a separate, human-approved LLM add-on). Note the Agent comment noise section above is a different, deterministic metric β€” it scores OUR automation's no-action share from the markers each comment already carries.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  10. github-actions commented on Sep 7, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-09-07 Β· org petry-projects Β· 11 repos scanned Β· 15 PRs active (15 non-draft, review-eligible)

    Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Rate-limited count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Rate-limited No response Latency p50 Latency p95
    GitHub Copilot 15 0 0 / 0 0 15 β€” β€”
    Gemini Code Assist 15 14 0 / 0 0 1 38s 2m
    Codex 15 0 0 / 0 14 1 β€” β€”
    CodeRabbit 15 11 0 / 0 7 1 33s 8m
    SonarCloud 15 15 0 / 0 0 0 35m 10.2d
    Qodo Merge 15 14 0 / 0 0 1 5s 6s
    CodeAnt 15 33 0 / 0 0 0 5s 7s
    Graphite 15 0 0 / 0 0 15 β€” β€”
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Rate-limited β€” count of out-of-quota / rate-limit refusals, each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 15 of 15 eligible (100%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Agent comment noise

    • Agent comments: 0

    • No-action comments: 0 (n/a of agent comments)

    • Active PRs (window): 15

    • PRs with agent comments: 0

    • Affected PRs (β‰₯1 no-action comment): 0

    • No-action comments per PR: 0.00

    • Noise = agent comments that ask nothing of a human β€” a reported no-change, a no-op engine run, or a clean/repeat approval. Actionable agent comments (applied fixes, escalated findings, human-attention flags) are excluded. Classification is deterministic (scripts/lib/comment-noise.sh), computed from the automation markers each comment already carries β€” no LLM.

    pr-review miss rate

    The false-NEGATIVE signal (#1596): accepted defects a trusted advisory bot found on a PR pr-review had already approved. Deterministic β€” computed with jq from review data, no LLM. Only findings resolved as accepted count; refuted false positives and ambiguous dispositions do not.

    • Miss rate: 0% β€” 0 missed vs 0 caught (accepted findings; miss = found first by a bot after approval).
    • Approvals on partial evidence: 0 β€” pr-review approved via the advisory-gate timeout fallback (before all bots reported).
    • Approved PRs in window: 0.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate for the third-party reviewers requires judgment and is intentionally out of this deterministic report (a separate, human-approved LLM add-on). Note the Agent comment noise section above is a different, deterministic metric β€” it scores OUR automation's no-action share from the markers each comment already carries.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  11. github-actions commented on Sep 14, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-09-14 Β· org petry-projects Β· 11 repos scanned Β· 3 PRs active (3 non-draft, review-eligible)

    Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Rate-limited count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Rate-limited No response Latency p50 Latency p95
    GitHub Copilot 3 0 0 / 0 0 3 β€” β€”
    Gemini Code Assist 3 2 0 / 0 0 1 5s 38s
    Codex 3 0 0 / 0 2 1 β€” β€”
    CodeRabbit 3 4 0 / 0 1 0 31s 8m
    SonarCloud 3 2 0 / 0 0 1 11m 9.4d
    Qodo Merge 3 2 0 / 0 0 1 4s 4s
    CodeAnt 3 6 0 / 0 0 0 4s 4s
    Graphite 3 0 0 / 0 0 3 β€” β€”
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Rate-limited β€” count of out-of-quota / rate-limit refusals, each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 3 of 3 eligible (100%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Agent comment noise

    • Agent comments: 0

    • No-action comments: 0 (n/a of agent comments)

    • Active PRs (window): 3

    • PRs with agent comments: 0

    • Affected PRs (β‰₯1 no-action comment): 0

    • No-action comments per PR: 0.00

    • Noise = agent comments that ask nothing of a human β€” a reported no-change, a no-op engine run, or a clean/repeat approval. Actionable agent comments (applied fixes, escalated findings, human-attention flags) are excluded. Classification is deterministic (scripts/lib/comment-noise.sh), computed from the automation markers each comment already carries β€” no LLM.

    pr-review miss rate

    The false-NEGATIVE signal (#1596): accepted defects a trusted advisory bot found on a PR pr-review had already approved. Deterministic β€” computed with jq from review data, no LLM. Only findings resolved as accepted count; refuted false positives and ambiguous dispositions do not.

    • Miss rate: 0% β€” 0 missed vs 0 caught (accepted findings; miss = found first by a bot after approval).
    • Approvals on partial evidence: 0 β€” pr-review approved via the advisory-gate timeout fallback (before all bots reported).
    • Approved PRs in window: 0.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate for the third-party reviewers requires judgment and is intentionally out of this deterministic report (a separate, human-approved LLM add-on). Note the Agent comment noise section above is a different, deterministic metric β€” it scores OUR automation's no-action share from the markers each comment already carries.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  12. github-actions commented on Sep 21, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-09-21 Β· org petry-projects Β· 11 repos scanned Β· 7 PRs active (7 non-draft, review-eligible)

    Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Rate-limited count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Rate-limited No response Latency p50 Latency p95
    GitHub Copilot 7 0 0 / 0 0 7 β€” β€”
    Gemini Code Assist 7 4 0 / 0 0 3 6s 4m
    Codex 7 0 0 / 0 4 3 β€” β€”
    CodeRabbit 7 6 0 / 0 4 0 25s 8m
    SonarCloud 7 4 0 / 0 0 3 6m 5.0d
    Qodo Merge 7 4 0 / 0 0 3 4s 6s
    CodeAnt 7 27 0 / 0 0 2 5s 2m
    Graphite 7 0 0 / 0 0 7 β€” β€”
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Rate-limited β€” count of out-of-quota / rate-limit refusals, each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 6 of 7 eligible (86%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Agent comment noise

    • Agent comments: 0

    • No-action comments: 0 (n/a of agent comments)

    • Active PRs (window): 7

    • PRs with agent comments: 0

    • Affected PRs (β‰₯1 no-action comment): 0

    • No-action comments per PR: 0.00

    • Noise = agent comments that ask nothing of a human β€” a reported no-change, a no-op engine run, or a clean/repeat approval. Actionable agent comments (applied fixes, escalated findings, human-attention flags) are excluded. Classification is deterministic (scripts/lib/comment-noise.sh), computed from the automation markers each comment already carries β€” no LLM.

    pr-review miss rate

    The false-NEGATIVE signal (#1596): accepted defects a trusted advisory bot found on a PR pr-review had already approved. Deterministic β€” computed with jq from review data, no LLM. Only findings resolved as accepted count; refuted false positives and ambiguous dispositions do not.

    • Miss rate: 0% β€” 0 missed vs 0 caught (accepted findings; miss = found first by a bot after approval).
    • Approvals on partial evidence: 0 β€” pr-review approved via the advisory-gate timeout fallback (before all bots reported).
    • Approved PRs in window: 0.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate for the third-party reviewers requires judgment and is intentionally out of this deterministic report (a separate, human-approved LLM add-on). Note the Agent comment noise section above is a different, deterministic metric β€” it scores OUR automation's no-action share from the markers each comment already carries.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  13. added a commit that references this issue on Sep 23, 2026
  14. github-actions commented on Sep 28, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-09-28 Β· org petry-projects Β· 11 repos scanned Β· 108 PRs active (105 non-draft, review-eligible)

    Truncated connections

    These per-PR connections hit the GraphQL fetch cap, so the affected reviewer counts may undercount on these PRs (truncation is reported here, never silent β€” #1908):

    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1084 β€” comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1084 β€” reviews truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1098 β€” comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1116 β€” comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1159 β€” reviewThreads.comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1159 β€” reviews truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1167 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1730 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1742 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1742 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1770 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1770 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1793 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1794 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1796 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1799 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1873 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1873 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1887 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1920 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1928 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1961 β€” reviews truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/499 β€” comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/499 β€” reviewThreads.comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/499 β€” reviews truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/509 β€” reviewThreads.comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/509 β€” reviews truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/8 β€” comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/8 β€” reviewThreads.comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/8 β€” reviews truncated
    • petry-projects/bmad-bgreat-suite PR https://github.com/petry-projects/bmad-bgreat-suite/pull/472 β€” comments truncated
    • petry-projects/bmad-bgreat-suite PR https://github.com/petry-projects/bmad-bgreat-suite/pull/472 β€” reviewThreads.comments truncated
    • petry-projects/bmad-bgreat-suite PR https://github.com/petry-projects/bmad-bgreat-suite/pull/472 β€” reviews truncated
    • petry-projects/broodly PR https://github.com/petry-projects/broodly/pull/435 β€” comments truncated
    • petry-projects/broodly PR https://github.com/petry-projects/broodly/pull/51 β€” comments truncated
    • petry-projects/broodly PR https://github.com/petry-projects/broodly/pull/521 β€” comments truncated
    • petry-projects/broodly PR https://github.com/petry-projects/broodly/pull/521 β€” reviewThreads.comments truncated
    • petry-projects/broodly PR https://github.com/petry-projects/broodly/pull/521 β€” reviews truncated
    • petry-projects/broodminder-export PR https://github.com/petry-projects/broodminder-export/pull/135 β€” reviewThreads.comments truncated
    • petry-projects/broodminder-export PR https://github.com/petry-projects/broodminder-export/pull/135 β€” reviews truncated
    • petry-projects/broodminder-export PR https://github.com/petry-projects/broodminder-export/pull/60 β€” comments truncated
    • petry-projects/broodminder-export PR https://github.com/petry-projects/broodminder-export/pull/60 β€” reviews truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/548 β€” reviews truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/584 β€” reviewThreads truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/584 β€” reviews truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/585 β€” comments truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/585 β€” reviewThreads.comments truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/585 β€” reviews truncated
    • petry-projects/incubator PR https://github.com/petry-projects/incubator/pull/121 β€” reviewThreads truncated
    • petry-projects/incubator PR https://github.com/petry-projects/incubator/pull/121 β€” reviews truncated
    • petry-projects/repo-template PR https://github.com/petry-projects/repo-template/pull/188 β€” comments truncated
    • petry-projects/repo-template PR https://github.com/petry-projects/repo-template/pull/190 β€” comments truncated
    • petry-projects/repo-template PR https://github.com/petry-projects/repo-template/pull/205 β€” comments truncated
    • petry-projects/repo-template PR https://github.com/petry-projects/repo-template/pull/205 β€” reviewThreads.comments truncated
    • petry-projects/repo-template PR https://github.com/petry-projects/repo-template/pull/205 β€” reviews truncated
    • petry-projects/repo-template PR https://github.com/petry-projects/repo-template/pull/210 β€” comments truncated
      Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Refused count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Refused No response Latency p50 Latency p95
    GitHub Copilot 105 2 0 / 0 0 103 13m 14m
    Gemini Code Assist 105 93 0 / 0 6 10 87s 3m
    Codex 105 64 0 / 0 167 20 6m 10m
    CodeRabbit 105 270 22 / 27 114 0 4m 7.2d
    SonarCloud 105 89 0 / 0 0 22 22.2h 29.8d
    Qodo Merge 105 94 0 / 0 0 11 5s 6s
    CodeAnt 105 360 0 / 0 15 22 2m 3m
    Graphite 105 16 0 / 0 2 52 9.0h 22.5d
    cubic 105 69 0 / 0 29 71 10m 43.7d
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Refused β€” count of refusal events: out-of-quota / rate-limit notices AND a check-run reporter's own decline (a "too large" / "did not run" skip), each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 101 of 105 eligible (96%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Agent comment noise

    • Agent comments: 0

    • No-action comments: 0 (n/a of agent comments)

    • Active PRs (window): 108

    • PRs with agent comments: 0

    • Affected PRs (β‰₯1 no-action comment): 0

    • No-action comments per PR: 0.00

    • Noise = agent comments that ask nothing of a human β€” a reported no-change, a no-op engine run, or a clean/repeat approval. Actionable agent comments (applied fixes, escalated findings, human-attention flags) are excluded. Classification is deterministic (scripts/lib/comment-noise.sh), computed from the automation markers each comment already carries β€” no LLM.

    pr-review miss rate

    The false-NEGATIVE signal (#1596): accepted defects a trusted advisory bot found on a PR pr-review had already approved. Deterministic β€” computed with jq from review data, no LLM. Only findings resolved as accepted count; refuted false positives and ambiguous dispositions do not.

    • Miss rate: 0% β€” 0 missed vs 0 caught (accepted findings; miss = found first by a bot after approval).
    • Approvals on partial evidence: 0 β€” pr-review approved via the advisory-gate timeout fallback (before all bots reported).
    • Approved PRs in window: 0.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate for the third-party reviewers requires judgment and is intentionally out of this deterministic report (a separate, human-approved LLM add-on). Note the Agent comment noise section above is a different, deterministic metric β€” it scores OUR automation's no-action share from the markers each comment already carries.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  15. github-actions commented on Oct 5, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-10-05 Β· org petry-projects Β· 12 repos scanned Β· 179 PRs active (174 non-draft, review-eligible)

    Truncated connections

    These per-PR connections hit the GraphQL fetch cap, so the affected reviewer counts may undercount on these PRs (truncation is reported here, never silent β€” #1908):

    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1098 β€” comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1116 β€” comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1116 β€” reviews truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1124 β€” comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1167 β€” comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1167 β€” reviews truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1196 β€” comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1196 β€” reviews truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1238 β€” comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1238 β€” reviews truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1239 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1730 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1742 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1742 β€” reviewThreads truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1742 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1770 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1770 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1793 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1794 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1796 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1799 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1873 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1873 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1887 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1891 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1902 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1909 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1952 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1952 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1953 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1953 β€” reviewThreads truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1953 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1972 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1983 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1983 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2009 β€” reviewThreads truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2009 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2010 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2011 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2019 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2022 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2033 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2040 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2048 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2048 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2049 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2049 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2051 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2051 β€” reviews truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/468 β€” comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/499 β€” comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/499 β€” reviewThreads.comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/499 β€” reviews truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/509 β€” reviewThreads.comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/509 β€” reviews truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/8 β€” comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/8 β€” reviewThreads.comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/8 β€” reviews truncated
    • petry-projects/bmad-bgreat-suite PR https://github.com/petry-projects/bmad-bgreat-suite/pull/472 β€” comments truncated
    • petry-projects/bmad-bgreat-suite PR https://github.com/petry-projects/bmad-bgreat-suite/pull/472 β€” reviewThreads.comments truncated
    • petry-projects/bmad-bgreat-suite PR https://github.com/petry-projects/bmad-bgreat-suite/pull/472 β€” reviews truncated
    • petry-projects/broodly PR https://github.com/petry-projects/broodly/pull/36 β€” comments truncated
    • petry-projects/broodly PR https://github.com/petry-projects/broodly/pull/435 β€” comments truncated
    • petry-projects/broodly PR https://github.com/petry-projects/broodly/pull/51 β€” comments truncated
    • petry-projects/broodly PR https://github.com/petry-projects/broodly/pull/521 β€” comments truncated
    • petry-projects/broodly PR https://github.com/petry-projects/broodly/pull/521 β€” reviewThreads.comments truncated
    • petry-projects/broodly PR https://github.com/petry-projects/broodly/pull/521 β€” reviews truncated
    • petry-projects/broodminder-data PR https://github.com/petry-projects/broodminder-data/pull/135 β€” comments truncated
    • petry-projects/broodminder-data PR https://github.com/petry-projects/broodminder-data/pull/135 β€” reviewThreads.comments truncated
    • petry-projects/broodminder-data PR https://github.com/petry-projects/broodminder-data/pull/135 β€” reviews truncated
    • petry-projects/broodminder-data PR https://github.com/petry-projects/broodminder-data/pull/145 β€” comments truncated
    • petry-projects/broodminder-data PR https://github.com/petry-projects/broodminder-data/pull/60 β€” comments truncated
    • petry-projects/broodminder-data PR https://github.com/petry-projects/broodminder-data/pull/60 β€” reviews truncated
    • petry-projects/empower-personal-dashboard PR https://github.com/petry-projects/empower-personal-dashboard/pull/37 β€” reviews truncated
    • petry-projects/empower-personal-dashboard PR https://github.com/petry-projects/empower-personal-dashboard/pull/46 β€” comments truncated
    • petry-projects/empower-personal-dashboard PR https://github.com/petry-projects/empower-personal-dashboard/pull/46 β€” reviewThreads truncated
    • petry-projects/empower-personal-dashboard PR https://github.com/petry-projects/empower-personal-dashboard/pull/46 β€” reviews truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/548 β€” reviews truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/584 β€” comments truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/584 β€” reviewThreads truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/584 β€” reviews truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/585 β€” comments truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/585 β€” reviewThreads.comments truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/585 β€” reviews truncated
    • petry-projects/incubator PR https://github.com/petry-projects/incubator/pull/102 β€” comments truncated
    • petry-projects/incubator PR https://github.com/petry-projects/incubator/pull/121 β€” comments truncated
    • petry-projects/incubator PR https://github.com/petry-projects/incubator/pull/121 β€” reviewThreads truncated
    • petry-projects/incubator PR https://github.com/petry-projects/incubator/pull/121 β€” reviews truncated
    • petry-projects/repo-template PR https://github.com/petry-projects/repo-template/pull/205 β€” comments truncated
    • petry-projects/repo-template PR https://github.com/petry-projects/repo-template/pull/205 β€” reviewThreads.comments truncated
    • petry-projects/repo-template PR https://github.com/petry-projects/repo-template/pull/205 β€” reviews truncated
      Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Refused count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Refused No response Latency p50 Latency p95
    GitHub Copilot 174 2 0 / 0 0 172 13m 14m
    Gemini Code Assist 174 118 0 / 0 15 49 2m 4m
    Codex 174 74 0 / 0 249 33 7m 11m
    CodeRabbit 174 401 24 / 31 155 3 3m 7.2d
    SonarCloud 174 116 0 / 0 0 59 6.2h 14.7d
    Qodo Merge 174 143 0 / 0 0 31 5s 26s
    CodeAnt 174 453 0 / 0 21 64 2m 3m
    Graphite 174 22 0 / 0 8 101 3.3h 21.6d
    cubic 174 222 0 / 0 105 67 10m 19.8d
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Refused β€” count of refusal events: out-of-quota / rate-limit notices AND a check-run reporter's own decline (a "too large" / "did not run" skip), each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 161 of 174 eligible (93%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Agent comment noise

    • Agent comments: 0

    • No-action comments: 0 (n/a of agent comments)

    • Active PRs (window): 179

    • PRs with agent comments: 0

    • Affected PRs (β‰₯1 no-action comment): 0

    • No-action comments per PR: 0.00

    • Noise = agent comments that ask nothing of a human β€” a reported no-change, a no-op engine run, or a clean/repeat approval. Actionable agent comments (applied fixes, escalated findings, human-attention flags) are excluded. Classification is deterministic (scripts/lib/comment-noise.sh), computed from the automation markers each comment already carries β€” no LLM.

    pr-review miss rate

    The false-NEGATIVE signal (#1596): accepted defects a trusted advisory bot found on a PR pr-review had already approved. Deterministic β€” computed with jq from review data, no LLM. Only findings resolved as accepted count; refuted false positives and ambiguous dispositions do not.

    • Miss rate: 0% β€” 0 missed vs 0 caught (accepted findings; miss = found first by a bot after approval).
    • Approvals on partial evidence: 0 β€” pr-review approved via the advisory-gate timeout fallback (before all bots reported).
    • Approved PRs in window: 0.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate for the third-party reviewers requires judgment and is intentionally out of this deterministic report (a separate, human-approved LLM add-on). Note the Agent comment noise section above is a different, deterministic metric β€” it scores OUR automation's no-action share from the markers each comment already carries.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  16. github-actions commented on Oct 12, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-10-12 Β· org petry-projects Β· 14 repos scanned Β· 254 PRs active (246 non-draft, review-eligible)

    Truncated connections

    These per-PR connections hit the GraphQL fetch cap, so the affected reviewer counts may undercount on these PRs (truncation is reported here, never silent β€” #1908):

    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1116 β€” comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1116 β€” reviews truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1124 β€” comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1167 β€” comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1167 β€” reviews truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1196 β€” comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1196 β€” reviews truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1237 β€” comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1245 β€” comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1269 β€” reviews truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1273 β€” comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1273 β€” reviews truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1275 β€” comments truncated
    • petry-projects/.github PR https://github.com/petry-projects/.github/pull/1298 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1730 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1730 β€” reviewThreads.comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1742 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1742 β€” reviewThreads truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1742 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1770 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1770 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1796 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1796 β€” reviewThreads.comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1796 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1799 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1902 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1902 β€” reviewThreads truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1902 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1909 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1953 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1953 β€” reviewThreads truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/1953 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2031 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2033 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2040 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2048 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2048 β€” reviewThreads truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2048 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2049 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2049 β€” reviewThreads truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2049 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2051 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2051 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2052 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2054 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2054 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2084 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2084 β€” reviewThreads.comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2084 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2085 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2135 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2135 β€” reviewThreads truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2135 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2150 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2150 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2157 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2157 β€” reviews truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2158 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2172 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2182 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2190 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2218 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2239 β€” comments truncated
    • petry-projects/.github-private PR https://github.com/petry-projects/.github-private/pull/2308 β€” comments truncated
    • petry-projects/ContentTwin PR https://github.com/petry-projects/ContentTwin/pull/480 β€” comments truncated
    • petry-projects/ContentTwin PR https://github.com/petry-projects/ContentTwin/pull/480 β€” reviews truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/468 β€” comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/509 β€” comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/509 β€” reviewThreads.comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/509 β€” reviews truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/520 β€” comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/521 β€” comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/521 β€” reviews truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/8 β€” comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/8 β€” reviewThreads.comments truncated
    • petry-projects/TalkTerm PR https://github.com/petry-projects/TalkTerm/pull/8 β€” reviews truncated
    • petry-projects/broodly PR https://github.com/petry-projects/broodly/pull/36 β€” comments truncated
    • petry-projects/broodly PR https://github.com/petry-projects/broodly/pull/51 β€” comments truncated
    • petry-projects/broodly PR https://github.com/petry-projects/broodly/pull/521 β€” comments truncated
    • petry-projects/broodly PR https://github.com/petry-projects/broodly/pull/521 β€” reviewThreads.comments truncated
    • petry-projects/broodly PR https://github.com/petry-projects/broodly/pull/521 β€” reviews truncated
    • petry-projects/broodminder-data PR https://github.com/petry-projects/broodminder-data/pull/135 β€” comments truncated
    • petry-projects/broodminder-data PR https://github.com/petry-projects/broodminder-data/pull/135 β€” reviewThreads.comments truncated
    • petry-projects/broodminder-data PR https://github.com/petry-projects/broodminder-data/pull/135 β€” reviews truncated
    • petry-projects/broodminder-data PR https://github.com/petry-projects/broodminder-data/pull/145 β€” comments truncated
    • petry-projects/broodminder-data PR https://github.com/petry-projects/broodminder-data/pull/60 β€” comments truncated
    • petry-projects/broodminder-data PR https://github.com/petry-projects/broodminder-data/pull/60 β€” reviews truncated
    • petry-projects/empower-personal-dashboard PR https://github.com/petry-projects/empower-personal-dashboard/pull/67 β€” comments truncated
    • petry-projects/empower-personal-dashboard PR https://github.com/petry-projects/empower-personal-dashboard/pull/69 β€” comments truncated
    • petry-projects/empower-personal-dashboard PR https://github.com/petry-projects/empower-personal-dashboard/pull/69 β€” reviews truncated
    • petry-projects/empower-personal-dashboard PR https://github.com/petry-projects/empower-personal-dashboard/pull/72 β€” comments truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/548 β€” comments truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/548 β€” reviews truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/584 β€” comments truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/584 β€” reviewThreads truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/584 β€” reviews truncated
    • petry-projects/google-app-scripts PR https://github.com/petry-projects/google-app-scripts/pull/594 β€” comments truncated
    • petry-projects/incubator PR https://github.com/petry-projects/incubator/pull/121 β€” comments truncated
    • petry-projects/incubator PR https://github.com/petry-projects/incubator/pull/121 β€” reviewThreads truncated
    • petry-projects/incubator PR https://github.com/petry-projects/incubator/pull/121 β€” reviews truncated
    • petry-projects/markets PR https://github.com/petry-projects/markets/pull/502 β€” comments truncated
    • petry-projects/markets PR https://github.com/petry-projects/markets/pull/513 β€” comments truncated
    • petry-projects/markets PR https://github.com/petry-projects/markets/pull/513 β€” reviews truncated
    • petry-projects/markets PR https://github.com/petry-projects/markets/pull/517 β€” comments truncated
    • petry-projects/repo-template PR https://github.com/petry-projects/repo-template/pull/205 β€” comments truncated
    • petry-projects/repo-template PR https://github.com/petry-projects/repo-template/pull/205 β€” reviewThreads.comments truncated
    • petry-projects/repo-template PR https://github.com/petry-projects/repo-template/pull/205 β€” reviews truncated
    • petry-projects/repo-template PR https://github.com/petry-projects/repo-template/pull/226 β€” comments truncated
    • petry-projects/repo-template PR https://github.com/petry-projects/repo-template/pull/233 β€” comments truncated
      Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Refused count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Refused No response Latency p50 Latency p95
    GitHub Copilot 246 1 0 / 0 0 245 14m 14m
    Gemini Code Assist 246 190 0 / 0 13 51 85s 4m
    Codex 246 131 0 / 0 365 20 12s 9m
    CodeRabbit 246 507 40 / 44 243 0 2m 47.4h
    SonarCloud 246 190 0 / 0 0 58 1.9h 6.8d
    Qodo Merge 246 112 0 / 0 0 134 7s 10s
    CodeAnt 246 549 0 / 0 14 26 3m 7m
    Graphite 246 45 0 / 0 10 93 73m 21.6d
    cubic 246 173 0 / 0 57 163 7m 20.7d
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Refused β€” count of refusal events: out-of-quota / rate-limit notices AND a check-run reporter's own decline (a "too large" / "did not run" skip), each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 244 of 246 eligible (99%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Agent comment noise

    • Agent comments: 0

    • No-action comments: 0 (n/a of agent comments)

    • Active PRs (window): 254

    • PRs with agent comments: 0

    • Affected PRs (β‰₯1 no-action comment): 0

    • No-action comments per PR: 0.00

    • Noise = agent comments that ask nothing of a human β€” a reported no-change, a no-op engine run, or a clean/repeat approval. Actionable agent comments (applied fixes, escalated findings, human-attention flags) are excluded. Classification is deterministic (scripts/lib/comment-noise.sh), computed from the automation markers each comment already carries β€” no LLM.

    pr-review miss rate

    The false-NEGATIVE signal (#1596): accepted defects a trusted advisory bot found on a PR pr-review had already approved. Deterministic β€” computed with jq from review data, no LLM. Only findings resolved as accepted count; refuted false positives and ambiguous dispositions do not.

    • Miss rate: 0% β€” 0 missed vs 0 caught (accepted findings; miss = found first by a bot after approval).
    • Approvals on partial evidence: 0 β€” pr-review approved via the advisory-gate timeout fallback (before all bots reported).
    • Approved PRs in window: 0.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate for the third-party reviewers requires judgment and is intentionally out of this deterministic report (a separate, human-approved LLM add-on). Note the Agent comment noise section above is a different, deterministic metric β€” it scores OUR automation's no-action share from the markers each comment already carries.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  17. github-actions commented on Oct 19, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-10-19 Β· org combsmith Β· 14 repos scanned Β· 238 PRs active (235 non-draft, review-eligible)

    Truncated connections

    These per-PR connections hit the GraphQL fetch cap, so the affected reviewer counts may undercount on these PRs (truncation is reported here, never silent β€” #1908):

    • combsmith/.github PR https://github.com/combsmith/.github/pull/1116 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1116 β€” reviews truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1196 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1196 β€” reviews truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1245 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1298 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1730 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1730 β€” reviewThreads.comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1770 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1770 β€” reviews truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1796 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1796 β€” reviewThreads.comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1796 β€” reviews truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2052 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2135 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2135 β€” reviewThreads truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2135 β€” reviews truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2217 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2218 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2308 β€” comments truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/468 β€” comments truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/8 β€” comments truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/8 β€” reviewThreads.comments truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/8 β€” reviews truncated
    • combsmith/broodly PR https://github.com/combsmith/broodly/pull/51 β€” comments truncated
    • combsmith/broodminder-data PR https://github.com/combsmith/broodminder-data/pull/187 β€” comments truncated
    • combsmith/broodminder-data PR https://github.com/combsmith/broodminder-data/pull/194 β€” comments truncated
    • combsmith/broodminder-data PR https://github.com/combsmith/broodminder-data/pull/195 β€” comments truncated
    • combsmith/empower-personal-dashboard PR https://github.com/combsmith/empower-personal-dashboard/pull/69 β€” comments truncated
    • combsmith/empower-personal-dashboard PR https://github.com/combsmith/empower-personal-dashboard/pull/69 β€” reviews truncated
    • combsmith/empower-personal-dashboard PR https://github.com/combsmith/empower-personal-dashboard/pull/82 β€” comments truncated
    • combsmith/empower-personal-dashboard PR https://github.com/combsmith/empower-personal-dashboard/pull/86 β€” comments truncated
    • combsmith/google-app-scripts PR https://github.com/combsmith/google-app-scripts/pull/584 β€” comments truncated
    • combsmith/google-app-scripts PR https://github.com/combsmith/google-app-scripts/pull/584 β€” reviewThreads truncated
    • combsmith/google-app-scripts PR https://github.com/combsmith/google-app-scripts/pull/584 β€” reviews truncated
    • combsmith/google-app-scripts PR https://github.com/combsmith/google-app-scripts/pull/627 β€” comments truncated
    • combsmith/incubator PR https://github.com/combsmith/incubator/pull/173 β€” comments truncated
    • combsmith/markets PR https://github.com/combsmith/markets/pull/513 β€” comments truncated
    • combsmith/markets PR https://github.com/combsmith/markets/pull/513 β€” reviews truncated
    • combsmith/repo-template PR https://github.com/combsmith/repo-template/pull/237 β€” comments truncated
      Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Refused count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Refused No response Latency p50 Latency p95
    GitHub Copilot 235 1 0 / 0 0 234 3m 3m
    Gemini Code Assist 235 50 0 / 0 2 186 2m 8m
    Codex 235 218 0 / 0 209 23 11s 5m
    CodeRabbit 235 378 41 / 31 139 3 20s 3.7h
    SonarCloud 235 231 0 / 0 0 30 14m 2.1d
    Qodo Merge 235 16 0 / 0 0 219 5s 8s
    CodeAnt 235 170 0 / 0 3 163 3m 5m
    Graphite 235 21 0 / 0 5 166 7m 26.9d
    cubic 235 27 0 / 0 5 223 14m 26.8d
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Refused β€” count of refusal events: out-of-quota / rate-limit notices AND a check-run reporter's own decline (a "too large" / "did not run" skip), each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 228 of 235 eligible (97%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Agent comment noise

    • Agent comments: 0

    • No-action comments: 0 (n/a of agent comments)

    • Active PRs (window): 238

    • PRs with agent comments: 0

    • Affected PRs (β‰₯1 no-action comment): 0

    • No-action comments per PR: 0.00

    • Noise = agent comments that ask nothing of a human β€” a reported no-change, a no-op engine run, or a clean/repeat approval. Actionable agent comments (applied fixes, escalated findings, human-attention flags) are excluded. Classification is deterministic (scripts/lib/comment-noise.sh), computed from the automation markers each comment already carries β€” no LLM.

    pr-review miss rate

    The false-NEGATIVE signal (#1596): accepted defects a trusted advisory bot found on a PR pr-review had already approved. Deterministic β€” computed with jq from review data, no LLM. Only findings resolved as accepted count; refuted false positives and ambiguous dispositions do not.

    • Miss rate: 0% β€” 0 missed vs 0 caught (accepted findings; miss = found first by a bot after approval).
    • Approvals on partial evidence: 0 β€” pr-review approved via the advisory-gate timeout fallback (before all bots reported).
    • Approved PRs in window: 0.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate for the third-party reviewers requires judgment and is intentionally out of this deterministic report (a separate, human-approved LLM add-on). Note the Agent comment noise section above is a different, deterministic metric β€” it scores OUR automation's no-action share from the markers each comment already carries.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  18. github-actions commented on Oct 26, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-10-26 Β· org combsmith Β· 15 repos scanned Β· 275 PRs active (268 non-draft, review-eligible)

    Truncated connections

    These per-PR connections hit the GraphQL fetch cap, so the affected reviewer counts may undercount on these PRs (truncation is reported here, never silent β€” #1908):

    • combsmith/.github PR https://github.com/combsmith/.github/pull/1116 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1116 β€” reviews truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1196 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1196 β€” reviews truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1245 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1298 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1730 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1730 β€” reviewThreads.comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1770 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1770 β€” reviewThreads truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1770 β€” reviews truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2135 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2135 β€” reviewThreads truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2135 β€” reviews truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2217 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2218 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2369 β€” reviews truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/468 β€” comments truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/8 β€” comments truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/8 β€” reviewThreads truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/8 β€” reviewThreads.comments truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/8 β€” reviews truncated
    • combsmith/broodly PR https://github.com/combsmith/broodly/pull/36 β€” comments truncated
    • combsmith/broodly PR https://github.com/combsmith/broodly/pull/51 β€” comments truncated
    • combsmith/broodminder-data PR https://github.com/combsmith/broodminder-data/pull/194 β€” comments truncated
    • combsmith/empower-personal-dashboard PR https://github.com/combsmith/empower-personal-dashboard/pull/110 β€” comments truncated
    • combsmith/empower-personal-dashboard PR https://github.com/combsmith/empower-personal-dashboard/pull/114 β€” reviews truncated
    • combsmith/empower-personal-dashboard PR https://github.com/combsmith/empower-personal-dashboard/pull/69 β€” comments truncated
    • combsmith/empower-personal-dashboard PR https://github.com/combsmith/empower-personal-dashboard/pull/69 β€” reviews truncated
    • combsmith/google-app-scripts PR https://github.com/combsmith/google-app-scripts/pull/627 β€” comments truncated
    • combsmith/google-app-scripts PR https://github.com/combsmith/google-app-scripts/pull/649 β€” reviewThreads truncated
    • combsmith/incubator PR https://github.com/combsmith/incubator/pull/173 β€” comments truncated
    • combsmith/incubator PR https://github.com/combsmith/incubator/pull/182 β€” reviews truncated
    • combsmith/incubator PR https://github.com/combsmith/incubator/pull/188 β€” comments truncated
      Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Refused count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Refused No response Latency p50 Latency p95
    GitHub Copilot 268 1 0 / 0 0 267 5m 5m
    Gemini Code Assist 268 22 0 / 0 1 248 83s 4m
    Codex 268 282 0 / 0 253 21 9s 17s
    CodeRabbit 268 493 40 / 59 181 0 29s 1.9h
    SonarCloud 268 240 0 / 0 0 40 36m 3.0d
    Qodo Merge 268 12 0 / 0 0 256 5s 8s
    CodeAnt 268 79 0 / 0 1 240 3m 8m
    Graphite 268 17 0 / 0 3 195 14m 17.3d
    cubic 268 24 0 / 0 4 259 14m 28.7d
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Refused β€” count of refusal events: out-of-quota / rate-limit notices AND a check-run reporter's own decline (a "too large" / "did not run" skip), each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 262 of 268 eligible (98%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Agent comment noise

    • Agent comments: 0

    • No-action comments: 0 (n/a of agent comments)

    • Active PRs (window): 275

    • PRs with agent comments: 0

    • Affected PRs (β‰₯1 no-action comment): 0

    • No-action comments per PR: 0.00

    • Noise = agent comments that ask nothing of a human β€” a reported no-change, a no-op engine run, or a clean/repeat approval. Actionable agent comments (applied fixes, escalated findings, human-attention flags) are excluded. Classification is deterministic (scripts/lib/comment-noise.sh), computed from the automation markers each comment already carries β€” no LLM.

    pr-review miss rate

    The false-NEGATIVE signal (#1596): accepted defects a trusted advisory bot found on a PR pr-review had already approved. Deterministic β€” computed with jq from review data, no LLM. Only findings resolved as accepted count; refuted false positives and ambiguous dispositions do not.

    • Miss rate: 0% β€” 0 missed vs 0 caught (accepted findings; miss = found first by a bot after approval).
    • Approvals on partial evidence: 0 β€” pr-review approved via the advisory-gate timeout fallback (before all bots reported).
    • Approved PRs in window: 0.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate for the third-party reviewers requires judgment and is intentionally out of this deterministic report (a separate, human-approved LLM add-on). Note the Agent comment noise section above is a different, deterministic metric β€” it scores OUR automation's no-action share from the markers each comment already carries.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  19. github-actions commented on Nov 2, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-11-02 Β· org combsmith Β· 17 repos scanned Β· 226 PRs active (222 non-draft, review-eligible)

    Truncated connections

    These per-PR connections hit the GraphQL fetch cap, so the affected reviewer counts may undercount on these PRs (truncation is reported here, never silent β€” #1908):

    • combsmith/.github PR https://github.com/combsmith/.github/pull/1116 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1116 β€” reviews truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1196 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1196 β€” reviews truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1245 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1298 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1378 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1383 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1411 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1730 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1730 β€” reviewThreads.comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2135 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2135 β€” reviewThreads truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2135 β€” reviews truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2217 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2218 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2260 β€” comments truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/468 β€” comments truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/8 β€” comments truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/8 β€” reviewThreads truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/8 β€” reviewThreads.comments truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/8 β€” reviews truncated
    • combsmith/broodly PR https://github.com/combsmith/broodly/pull/51 β€” comments truncated
    • combsmith/broodminder-data PR https://github.com/combsmith/broodminder-data/pull/194 β€” comments truncated
    • combsmith/broodminder-data PR https://github.com/combsmith/broodminder-data/pull/207 β€” comments truncated
    • combsmith/broodminder-data PR https://github.com/combsmith/broodminder-data/pull/225 β€” reviews truncated
    • combsmith/empower-personal-dashboard PR https://github.com/combsmith/empower-personal-dashboard/pull/110 β€” comments truncated
    • combsmith/empower-personal-dashboard PR https://github.com/combsmith/empower-personal-dashboard/pull/128 β€” comments truncated
    • combsmith/google-app-scripts PR https://github.com/combsmith/google-app-scripts/pull/627 β€” comments truncated
    • combsmith/google-app-scripts PR https://github.com/combsmith/google-app-scripts/pull/649 β€” comments truncated
    • combsmith/google-app-scripts PR https://github.com/combsmith/google-app-scripts/pull/649 β€” reviewThreads truncated
    • combsmith/google-app-scripts PR https://github.com/combsmith/google-app-scripts/pull/649 β€” reviews truncated
    • combsmith/repo-template PR https://github.com/combsmith/repo-template/pull/256 β€” comments truncated

    Check-run fetch failures

    A check-run REST fetch failed on these PRs, so a check-run reporter clean pass may be missed and miscounted as No response (surfaced here, never silent β€” #1908):

    • combsmith/.github-private-ghsa-fx78-wx9r-525p PR https://github.com/combsmith/.github-private-ghsa-fx78-wx9r-525p/pull/1 β€” check-run fetch failed
    • combsmith/.github-private-ghsa-vxf5-2jf3-4jxr PR https://github.com/combsmith/.github-private-ghsa-vxf5-2jf3-4jxr/pull/1 β€” check-run fetch failed
      Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Refused count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Refused No response Latency p50 Latency p95
    GitHub Copilot 222 1 0 / 0 0 221 5m 5m
    Gemini Code Assist 222 16 0 / 0 0 208 83s 4m
    Codex 222 229 0 / 0 185 40 10s 18s
    CodeRabbit 222 389 15 / 45 91 2 10s 7.0h
    SonarCloud 222 180 0 / 0 0 50 3.1h 5.6d
    Qodo Merge 222 8 0 / 0 0 214 5s 19m
    CodeAnt 222 38 0 / 0 1 207 3m 5m
    Graphite 222 19 0 / 0 4 182 43m 17.3d
    cubic 222 12 0 / 0 4 216 14m 28.7d
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Refused β€” count of refusal events: out-of-quota / rate-limit notices AND a check-run reporter's own decline (a "too large" / "did not run" skip), each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 197 of 222 eligible (89%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Agent comment noise

    • Agent comments: 0

    • No-action comments: 0 (n/a of agent comments)

    • Active PRs (window): 226

    • PRs with agent comments: 0

    • Affected PRs (β‰₯1 no-action comment): 0

    • No-action comments per PR: 0.00

    • Noise = agent comments that ask nothing of a human β€” a reported no-change, a no-op engine run, or a clean/repeat approval. Actionable agent comments (applied fixes, escalated findings, human-attention flags) are excluded. Classification is deterministic (scripts/lib/comment-noise.sh), computed from the automation markers each comment already carries β€” no LLM.

    pr-review miss rate

    The false-NEGATIVE signal (#1596): accepted defects a trusted advisory bot found on a PR pr-review had already approved. Deterministic β€” computed with jq from review data, no LLM. Only findings resolved as accepted count; refuted false positives and ambiguous dispositions do not.

    • Miss rate: 0% β€” 0 missed vs 0 caught (accepted findings; miss = found first by a bot after approval).
    • Approvals on partial evidence: 0 β€” pr-review approved via the advisory-gate timeout fallback (before all bots reported).
    • Approved PRs in window: 0.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate for the third-party reviewers requires judgment and is intentionally out of this deterministic report (a separate, human-approved LLM add-on). Note the Agent comment noise section above is a different, deterministic metric β€” it scores OUR automation's no-action share from the markers each comment already carries.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  20. github-actions commented on Nov 9, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-11-09 Β· org combsmith Β· 20 repos scanned Β· 267 PRs active (262 non-draft, review-eligible)

    Truncated connections

    These per-PR connections hit the GraphQL fetch cap, so the affected reviewer counts may undercount on these PRs (truncation is reported here, never silent β€” #1908):

    • combsmith/.github PR https://github.com/combsmith/.github/pull/1116 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1116 β€” reviews truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1196 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1196 β€” reviews truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1245 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1298 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1299 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1378 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1383 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1418 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1421 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1437 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1463 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1730 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1730 β€” reviewThreads.comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1770 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1770 β€” reviewThreads truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1770 β€” reviews truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1796 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1796 β€” reviewThreads.comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1796 β€” reviews truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2135 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2135 β€” reviewThreads truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2135 β€” reviews truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2217 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2218 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2260 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2308 β€” comments truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/468 β€” comments truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/551 β€” comments truncated
    • combsmith/broodly PR https://github.com/combsmith/broodly/pull/51 β€” comments truncated
    • combsmith/broodly PR https://github.com/combsmith/broodly/pull/624 β€” comments truncated
    • combsmith/broodminder-data PR https://github.com/combsmith/broodminder-data/pull/194 β€” comments truncated
    • combsmith/broodminder-data PR https://github.com/combsmith/broodminder-data/pull/207 β€” comments truncated
    • combsmith/empower-personal-dashboard PR https://github.com/combsmith/empower-personal-dashboard/pull/110 β€” comments truncated
    • combsmith/google-app-scripts PR https://github.com/combsmith/google-app-scripts/pull/627 β€” comments truncated
    • combsmith/repo-template PR https://github.com/combsmith/repo-template/pull/256 β€” comments truncated

    Check-run fetch failures

    A check-run REST fetch failed on these PRs, so a check-run reporter clean pass may be missed and miscounted as No response (surfaced here, never silent β€” #1908):

    • combsmith/.github-ghsa-qpjq-vrh6-8wcg PR https://github.com/combsmith/.github-ghsa-qpjq-vrh6-8wcg/pull/1 β€” check-run fetch failed
    • combsmith/.github-private-ghsa-fx78-wx9r-525p PR https://github.com/combsmith/.github-private-ghsa-fx78-wx9r-525p/pull/1 β€” check-run fetch failed
    • combsmith/.github-private-ghsa-rw84-v39h-45cp PR https://github.com/combsmith/.github-private-ghsa-rw84-v39h-45cp/pull/1 β€” check-run fetch failed
    • combsmith/TalkTerm-ghsa-pc5f-cw67-7cmm PR https://github.com/combsmith/TalkTerm-ghsa-pc5f-cw67-7cmm/pull/1 β€” check-run fetch failed
      Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh); rate-limit/out-of-quota notices are detected by body text. Reviews and Refused count every event, not distinct PRs β€” a PR re-reviewed across N commits contributes N reviews. No response counts eligible PRs the reviewer never engaged with at all. Latency = time from PR creation to the bot's first real review (refusals excluded). Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs Reviews βœ… / πŸ”„ Refused No response Latency p50 Latency p95
    GitHub Copilot 262 0 0 / 0 0 262 β€” β€”
    Gemini Code Assist 262 22 0 / 0 1 242 73s 4m
    Codex 262 274 0 / 0 292 35 9s 80s
    CodeRabbit 262 498 25 / 43 76 4 9s 80m
    SonarCloud 262 229 0 / 0 0 41 57m 5.9d
    Qodo Merge 262 10 0 / 0 0 252 5s 19m
    CodeAnt 262 52 0 / 0 1 241 2m 5m
    Graphite 262 24 0 / 0 5 193 14m 32.5d
    cubic 262 20 0 / 0 4 253 2.9d 28.7d
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • Reviews β€” count of reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5). A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has that comment counted as its review. Rate-limit notices are never counted.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Refused β€” count of refusal events: out-of-quota / rate-limit notices AND a check-run reporter's own decline (a "too large" / "did not run" skip), each occurrence.
    • No response β€” eligible PRs the bot never engaged with at all (no review, comment, or refusal).

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 248 of 262 eligible (95%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Agent comment noise

    • Agent comments: 0

    • No-action comments: 0 (n/a of agent comments)

    • Active PRs (window): 267

    • PRs with agent comments: 0

    • Affected PRs (β‰₯1 no-action comment): 0

    • No-action comments per PR: 0.00

    • Noise = agent comments that ask nothing of a human β€” a reported no-change, a no-op engine run, or a clean/repeat approval. Actionable agent comments (applied fixes, escalated findings, human-attention flags) are excluded. Classification is deterministic (scripts/lib/comment-noise.sh), computed from the automation markers each comment already carries β€” no LLM.

    pr-review miss rate

    The false-NEGATIVE signal (#1596): accepted defects a trusted advisory bot found on a PR pr-review had already approved. Deterministic β€” computed with jq from review data, no LLM. Only findings resolved as accepted count; refuted false positives and ambiguous dispositions do not.

    • Miss rate: 0% β€” 0 missed vs 0 caught (accepted findings; miss = found first by a bot after approval).
    • Approvals on partial evidence: 0 β€” pr-review approved via the advisory-gate timeout fallback (before all bots reported).
    • Approved PRs in window: 0.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate for the third-party reviewers requires judgment and is intentionally out of this deterministic report (a separate, human-approved LLM add-on). Note the Agent comment noise section above is a different, deterministic metric β€” it scores OUR automation's no-action share from the markers each comment already carries.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

  21. github-actions commented on Nov 16, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report (2026-11-16): withdrawn

    Warning

    This report was wrong and its figures have been removed. It counted 0 reviews for every bot.
    Ignore this week's outage alerts, coverage matrix and quota timing as well.

    Use the corrected report instead: #1220 (comment) (generated 2026-11-19, covering the 7 days to that date).

    Why it was wrong. ubuntu-latest moved from Ubuntu 24.04 to 26.04 between the 2026-11-09 and 2026-11-16 runs. Ubuntu 26.04 ships jq 1.8, which parses the bot check in scripts/reviewer_report.sh differently, so no message was recognized as coming from a bot. The run itself succeeded, so nothing flagged the error. The fix is #2568 (issue #2547), merged 2026-11-19.

    Window not re-run. The corrected report covers 2026-11-12 to 2026-11-19, not this report's 2026-11-09 to 2026-11-16. Next Monday's scheduled report resumes the weekly series.


    Rewritten by Claude Code on behalf of the maintainer. The original all-zero report is in this comment's edit history.

  22. github-actions commented on Nov 19, 2026

    @github-actions
    ContributorAuthor

    πŸ”Ž Third-Party Reviewer Scorecard β€” 7-day Report

    Generated 2026-11-19 Β· org combsmith Β· 20 repos scanned Β· 198 PRs active (194 non-draft, review-eligible)

    Truncated connections

    These per-PR connections hit the GraphQL fetch cap, so the affected reviewer counts may undercount on these PRs (truncation is reported here, never silent β€” #1908):

    • combsmith/.github PR https://github.com/combsmith/.github/pull/1116 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1116 β€” reviews truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1117 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1218 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1298 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1378 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1383 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1426 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1498 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1503 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1503 β€” reviews truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1506 β€” comments truncated
    • combsmith/.github PR https://github.com/combsmith/.github/pull/1514 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1730 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1730 β€” reviewThreads.comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1730 β€” reviews truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1770 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1770 β€” reviewThreads truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1770 β€” reviews truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1796 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1796 β€” reviewThreads.comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/1796 β€” reviews truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2135 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2135 β€” reviewThreads truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2135 β€” reviews truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2217 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2218 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2260 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2290 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2308 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2308 β€” reviews truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2319 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2328 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2368 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2483 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2488 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2488 β€” reviews truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2494 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2505 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2505 β€” reviews truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2516 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2517 β€” comments truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2517 β€” reviews truncated
    • combsmith/.github-private PR https://github.com/combsmith/.github-private/pull/2536 β€” comments truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/549 β€” comments truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/551 β€” comments truncated
    • combsmith/TalkTerm PR https://github.com/combsmith/TalkTerm/pull/580 β€” comments truncated
    • combsmith/broodly PR https://github.com/combsmith/broodly/pull/51 β€” comments truncated
    • combsmith/broodly PR https://github.com/combsmith/broodly/pull/645 β€” comments truncated
    • combsmith/broodminder-data PR https://github.com/combsmith/broodminder-data/pull/194 β€” comments truncated
    • combsmith/broodminder-data PR https://github.com/combsmith/broodminder-data/pull/194 β€” reviews truncated
    • combsmith/broodminder-data PR https://github.com/combsmith/broodminder-data/pull/207 β€” comments truncated
    • combsmith/empower-personal-dashboard PR https://github.com/combsmith/empower-personal-dashboard/pull/110 β€” comments truncated
    • combsmith/empower-personal-dashboard PR https://github.com/combsmith/empower-personal-dashboard/pull/178 β€” comments truncated
    • combsmith/empower-personal-dashboard PR https://github.com/combsmith/empower-personal-dashboard/pull/180 β€” comments truncated
    • combsmith/google-app-scripts PR https://github.com/combsmith/google-app-scripts/pull/627 β€” comments truncated
    • combsmith/google-app-scripts PR https://github.com/combsmith/google-app-scripts/pull/680 β€” comments truncated
    • combsmith/repo-template PR https://github.com/combsmith/repo-template/pull/256 β€” comments truncated

    Outage alerts

    • GitHub Copilot β€” no PR reviewed on 8 consecutive days with PRs (117 PRs opened) in combsmith/.github, combsmith/.github-private, combsmith/broodly, combsmith/google-app-scripts, combsmith/software-factory, combsmith/software-factory-canary; first silent day 2026-11-12.
    • Codex β€” no PR reviewed on 3 consecutive days with PRs (6 PRs opened) in combsmith/.github-private, combsmith/broodly, combsmith/software-factory; first silent day 2026-11-15.
    • Qodo Merge β€” no PR reviewed on 8 consecutive days with PRs (117 PRs opened) in combsmith/.github, combsmith/.github-private, combsmith/broodly, combsmith/google-app-scripts, combsmith/software-factory, combsmith/software-factory-canary; first silent day 2026-11-12.
    • GitHub Copilot β€” 0 of 3 PRs opened in combsmith/.github reviewed in the whole window; first silent day 2026-11-13.
    • GitHub Copilot β€” 0 of 16 PRs opened in combsmith/.github-private reviewed in the whole window; first silent day 2026-11-12.
    • GitHub Copilot β€” 0 of 3 PRs opened in combsmith/broodly reviewed in the whole window; first silent day 2026-11-12.
    • GitHub Copilot β€” 0 of 84 PRs opened in combsmith/software-factory reviewed in the whole window; first silent day 2026-11-12.
    • GitHub Copilot β€” 0 of 10 PRs opened in combsmith/software-factory-canary reviewed in the whole window; first silent day 2026-11-13.
    • Codex β€” 0 of 10 PRs opened in combsmith/software-factory-canary reviewed in the whole window; first silent day 2026-11-13.
    • Qodo Merge β€” 0 of 3 PRs opened in combsmith/.github reviewed in the whole window; first silent day 2026-11-13.
    • Qodo Merge β€” 0 of 16 PRs opened in combsmith/.github-private reviewed in the whole window; first silent day 2026-11-12.
    • Qodo Merge β€” 0 of 3 PRs opened in combsmith/broodly reviewed in the whole window; first silent day 2026-11-12.
    • Qodo Merge β€” 0 of 84 PRs opened in combsmith/software-factory reviewed in the whole window; first silent day 2026-11-12.
    • Qodo Merge β€” 0 of 10 PRs opened in combsmith/software-factory-canary reviewed in the whole window; first silent day 2026-11-13.
    • Graphite β€” 0 of 10 PRs opened in combsmith/software-factory-canary reviewed in the whole window; first silent day 2026-11-13.

    Rules and how to mark a cell off by design: docs/reviewer-report.md (Outage alerts).

    Deterministic report β€” every figure is computed with jq/awk from GitHub review data; no LLM is involved. Bots are identified by their GraphQL App login (scripts/lib/advisory-review-gate.sh). Every bot message β€” review, inline comment, top-level comment, check run β€” gets exactly one class (finding, clean, refused, skipped, status or unclassified) from that bot's patterns (scripts/lib/reviewer-message-classes.tsv) first and the shared rate-limit pattern second. A PR counts as reviewed only when the bot left a finding or clean message on it; status notices and unclassified messages never count. Each eligible PR lands in exactly one bucket per bot, so the five PR columns sum to Total PRs. Latency = time from PR creation to the bot's first finding or clean message. Deltas (β–²/β–Ό) are vs the prior week and directional only.

    Scorecard

    Reviewer Total PRs With findings Clean Refused Skipped No response Findings raised Resolved % Latency p50 Latency p95
    GitHub Copilot 194 0 0 0 0 194 0 n/a β€” β€”
    Gemini Code Assist 194 78 45 28 0 43 164 82% 2m 25.2h
    Codex 194 35 0 53 0 106 149 88% 4m 47.4h
    CodeRabbit 194 60 33 0 100 1 261 84% 16s 5.9d
    SonarCloud 194 6 142 0 0 46 6 n/a 64m 7.7d
    Qodo Merge 194 0 0 7 0 187 0 n/a β€” β€”
    CodeAnt 194 91 0 0 0 103 143 79% 7m 21.5d
    Graphite 194 28 96 3 0 67 49 90% 24m 21.5d
    cubic 194 108 39 0 0 47 505 90% 4m 21.5d
    • Total PRs β€” review-eligible (non-draft) PRs in the window; the denominator each row is measured against.
    • With findings β€” PRs where the bot left at least one finding message (an inline thread, a review that requests changes, a body that reports issues).
    • Clean β€” PRs where the bot reviewed and reported nothing: at least one clean message and no finding.
    • Refused β€” PRs where the bot only declined for quota, rate limit, trial or plan reasons (no finding or clean message).
    • Skipped β€” PRs where the bot only reported that it skipped the review because of configuration (for example auto review turned off).
    • No response β€” every other eligible PR: the bot never appeared, or left only status or unclassified messages.
    • Findings raised β€” review threads the bot opened plus findings written in its review, comment or check-run bodies. CodeRabbit's outside-diff and nitpick items and Codex's body suggestions without a matching inline comment always count; any other finding body counts once, and only on a PR where the bot opened no thread (otherwise it summarizes those threads). The Details section splits the two.
    • Resolved % β€” threads the bot opened that are resolved, out of all threads it opened.

    Details

    Reviewer Reviews βœ… / πŸ”„ Refusal events πŸ‘ / πŸ‘Ž Status messages Unclassified messages Findings: threads / bodies
    GitHub Copilot 0 0 / 0 0 0 / 0 0 0 0 / 0
    Gemini Code Assist 144 0 / 0 37 0 / 0 0 102 164 / 0
    Codex 88 0 / 0 147 0 / 0 154 0 149 / 0
    CodeRabbit 393 1 / 28 13 0 / 0 189 225 211 / 50
    SonarCloud 151 0 / 0 0 0 / 0 0 0 0 / 6
    Qodo Merge 0 0 / 0 7 0 / 0 0 0 0 / 0
    CodeAnt 168 0 / 0 0 0 / 0 498 165 112 / 31
    Graphite 36 0 / 0 3 0 / 0 0 38 49 / 0
    cubic 234 0 / 0 0 0 / 0 0 191 505 / 0
    • Reviews β€” formal reviews the bot submitted, each occurrence (a PR re-reviewed on 5 commits = 5), leaving out refusals, skips and status notices. A bot that posts no formal review but delivers its verdict as a comment (e.g. SonarCloud's quality-gate comment) has its finding / clean comments counted instead.
    • βœ… / πŸ”„ β€” of those reviews, how many were APPROVED / CHANGES_REQUESTED (comment-only responses carry no verdict).
    • Refusal events β€” every refused message, each occurrence.
    • πŸ‘ / πŸ‘Ž β€” reactions on the bot's inline comments.
    • Status / Unclassified messages β€” message counts; neither ever makes a PR count as reviewed.
    • Findings: threads / bodies β€” the two parts of Findings raised: review threads the bot opened, and findings counted from its review, comment or check-run bodies.

    Quota timing

    Refusals (quota, rate limit, trial or plan) per bot per UTC day, and the time of the first one that day. Dependabot PRs are left out.

    Reviewer Day (UTC) Refusals First refusal (UTC)
    Gemini Code Assist 2026-11-13 9 05:54
    Gemini Code Assist 2026-11-14 3 01:28
    Codex 2026-11-13 26 02:28
    Codex 2026-11-14 6 03:06
    CodeRabbit 2026-11-14 2 00:22
    Graphite 2026-11-12 1 19:03
    Graphite 2026-11-18 1 08:15

    Unclassified messages

    Bot messages that no pattern recognized. They count toward no column above. Add a row to scripts/lib/reviewer-message-classes.tsv for each new shape (see docs/reviewer-report.md).

    • Gemini Code Assist β€” 102 unclassified messages
      • NoteGemini is unable to generate a review for this pull request due to the file types involved not b…
      • Code Review This pull request introduces a new check to verify that the pr-auto-review caller job sk…
      • Code Review This pull request updates the Node.js engine requirements to >=20.19.4 and introduces pa…
    • CodeRabbit β€” 225 unclassified messages
      • ``
      • ``
      • ``
    • CodeAnt β€” 165 unclassified messages
      • ``
      • ``
      • Skipping PR review because a bot author is detected. If you want to trigger CodeAnt AI, comment @cod…
    • Graphite β€” 38 unclassified messages
      • ``
      • ``
      • ``
    • cubic β€” 191 unclassified messages
      • 1 issue found across 9 files Prompt for AI agents (unresolved issues) Check if these issues are vali…
      • 8 issues found across 4 files (changes from recent commits). Prompt for AI agents (unresolved issues…
      • All reported issues were addressed Reply with feedback, questions, or to request a fix.View guided d…

    Coverage matrix

    PRs each bot reviewed (a finding or clean message) out of the non-draft PRs opened in each repo in the window. Dependabot PRs are left out. off (by design) marks a bot that is meant not to run on that repo (declared in scripts/lib/reviewer-message-classes.tsv).

    Repo GitHub Copilot Gemini Code Assist Codex CodeRabbit SonarCloud Qodo Merge CodeAnt Graphite cubic
    combsmith/.github 0 / 3 3 / 3 2 / 3 3 / 3 3 / 3 0 / 3 3 / 3 2 / 3 3 / 3
    combsmith/.github-private 0 / 16 13 / 16 6 / 16 16 / 16 13 / 16 0 / 16 11 / 16 9 / 16 16 / 16
    combsmith/broodly 0 / 3 2 / 3 1 / 3 2 / 3 3 / 3 0 / 3 2 / 3 2 / 3 3 / 3
    combsmith/google-app-scripts 0 / 1 1 / 1 1 / 1 1 / 1 1 / 1 0 / 1 1 / 1 0 / 1 1 / 1
    combsmith/software-factory 0 / 84 65 / 84 11 / 84 off (by design) 84 / 84 0 / 84 39 / 84 81 / 84 83 / 84
    combsmith/software-factory-canary 0 / 10 4 / 10 0 / 10 7 / 10 10 / 10 0 / 10 1 / 10 0 / 10 10 / 10

    Coverage overlap

    • PRs reviewed by β‰₯2 bots: 173 of 194 eligible (89%)
    • Higher overlap = more redundant coverage; lower = reviewers are specializing or missing PRs.

    Agent comment noise

    • Agent comments: 0

    • No-action comments: 0 (n/a of agent comments)

    • Active PRs (window): 198

    • PRs with agent comments: 0

    • Affected PRs (β‰₯1 no-action comment): 0

    • No-action comments per PR: 0.00

    • Noise = agent comments that ask nothing of a human β€” a reported no-change, a no-op engine run, or a clean/repeat approval. Actionable agent comments (applied fixes, escalated findings, human-attention flags) are excluded. Classification is deterministic (scripts/lib/comment-noise.sh), computed from the automation markers each comment already carries β€” no LLM.

    pr-review miss rate

    The false-NEGATIVE signal (#1596): accepted defects a trusted advisory bot found on a PR pr-review had already approved. Deterministic β€” computed with jq from review data, no LLM. Only findings resolved as accepted count; refuted false positives and ambiguous dispositions do not.

    • Miss rate: 0% β€” 0 missed vs 0 caught (accepted findings; miss = found first by a bot after approval).
    • Approvals on partial evidence: 0 β€” pr-review approved via the advisory-gate timeout fallback (before all bots reported).
    • Approved PRs in window: 0.

    Known gaps

    • Cost is not measured here. These reviewers are external SaaS GitHub Apps and emit no token-usage artifacts, so per-review $ cost is not observable from our side. For our own Claude reviewer's spend, see the Token Cost Observatory.
    • Comment usefulness / false-positive rate for the third-party reviewers requires judgment and is intentionally out of this deterministic report (a separate, human-approved LLM add-on). Note the Agent comment noise section above is a different, deterministic metric β€” it scores OUR automation's no-action share from the markers each comment already carries.
    • Latency baseline is PR-creation time (v1). A refinement to last-human-push is tracked for v2.

    Source: scripts/reviewer_report.sh Β· .github/workflows/reviewer-report.yml β€” no LLM in this pipeline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    reviewer-reportWeekly Third-Party Reviewer Scorecard report

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions