Repository navigation
Description
Problem
After #2504 (Reviewer scorecard: classify each bot message so refusals, skips and status notices stop counting as reviews), Findings raised counts a bot's review-body findings only on a PR where the bot opened no inline thread. If the bot opened even one thread, every finding written only in its review body is dropped, on the assumption that the body summarizes the threads.
That assumption doesn't hold for the two bots that write the most body-only findings:
- coderabbitai puts findings in collapsible body sections that never become threads: "Outside diff range comments", "Nitpick comments" and, on some reviews, security or architecture sections.
- chatgpt-codex-connector sometimes lists suggestions in its review body that have no matching inline comment.
So Findings raised undercounts exactly the bots whose contribution the scorecard is meant to measure. The maintainer accepted the current rule for #2504 on 2026-11-13 (recorded in #2504's body) so the refusal fix could land, and asked for this follow-up.
The forced review of PR #2505 raised a second, smaller issue. Per-bot rows with kind: any apply to inline comments for every class except refused. Broad status and skipped rows can therefore turn a real inline finding into status or skipped, which drops it from threads and findings. Examples are the codeant-ai Review Status row, its reviews? … (skipped|disabled) row, and the coderabbitai review in progress row.
Depends on
#2504 (its classifier and scripts/lib/reviewer-message-classes.tsv sidecar).
Scope
Scorecard only (scripts/reviewer_report.sh, the sidecar, tests, docs). Don't change gate behaviour.
Acceptance
- Count independent body findings by structure. For coderabbitai, every item under "Outside diff range comments" and "Nitpick comments" counts toward Findings raised whether or not the bot opened threads on that PR. Use the counts CodeRabbit prints in those section headers, or the items in them. For chatgpt-codex-connector, a body suggestion counts unless an inline comment on the same file and line exists from the same review. Other bots keep the current rule.
- Separate counts. The Details section shows, per bot, findings from threads and findings from review bodies separately. The headline Findings raised is their sum.
- Inline-safe rows. Sidecar rows of class
statusorskippedno longer apply to inline comments. Either the loader excludes them forkind: inline, as it already does forrefused, or the rows are narrowed tokind: issue/review. A fixture shows an inline finding containing "Review Status" or "review in progress" staying afinding. - Tests and docs. Fixtures in
tests/reviewer_report.batscover a CodeRabbit review with threads plus outside-diff and nitpick sections, and a Codex review with one suggestion matched by an inline comment and one not.docs/reviewer-report.mddescribes the body-finding rule for each bot.
Activity
don-petry commentedon Nov 13, 2026
Dev-Lead: implementation deferred — automation PR queue is full
No pull request was opened for issue #2527. The org-wide
open-automation-PR queue is currently at its signed-off cap (see
standards/pr-limits.jsonincombsmith/.github).This issue stays labeled
dev-leadand will be re-attempted automatically oncethe queue drains — no action is required.
don-petry commentedon Nov 13, 2026
dev-lead is withholding action on this item.
It is labeled
dev-lead:hands-off(deliberately kept off-limits to dev-lead (e.g. changes to the dev-lead automation itself)), so dev-lead will not pick it up while that label is present. This notice is posted once so the withhold is visible rather than looking like a stalled run.To re-enable automated pickup: remove the
dev-lead:hands-offlabel.fix(reviewer-report): compare Codex location paths exactly (#2527)
fix(reviewer-report): URL-decode Codex permalink paths before matching (
fix(reviewer-report): count a Codex location written both ways once (#…
fix(reviewer-report): keep spaces in rendered Codex snippet paths (#2527
fix(reviewer-report): accept a Codex snippet path on the same line as…