Skip to content

Reviewer scorecard: count summary-only findings for CodeRabbit and Codex, and keep status/skip rows off inline comments #2527

Closed

Description

@don-petry

Problem

After #2504 (Reviewer scorecard: classify each bot message so refusals, skips and status notices stop counting as reviews), Findings raised counts a bot's review-body findings only on a PR where the bot opened no inline thread. If the bot opened even one thread, every finding written only in its review body is dropped, on the assumption that the body summarizes the threads.

That assumption doesn't hold for the two bots that write the most body-only findings:

  • coderabbitai puts findings in collapsible body sections that never become threads: "Outside diff range comments", "Nitpick comments" and, on some reviews, security or architecture sections.
  • chatgpt-codex-connector sometimes lists suggestions in its review body that have no matching inline comment.

So Findings raised undercounts exactly the bots whose contribution the scorecard is meant to measure. The maintainer accepted the current rule for #2504 on 2026-11-13 (recorded in #2504's body) so the refusal fix could land, and asked for this follow-up.

The forced review of PR #2505 raised a second, smaller issue. Per-bot rows with kind: any apply to inline comments for every class except refused. Broad status and skipped rows can therefore turn a real inline finding into status or skipped, which drops it from threads and findings. Examples are the codeant-ai Review Status row, its reviews? … (skipped|disabled) row, and the coderabbitai review in progress row.

Depends on

#2504 (its classifier and scripts/lib/reviewer-message-classes.tsv sidecar).

Scope

Scorecard only (scripts/reviewer_report.sh, the sidecar, tests, docs). Don't change gate behaviour.

Acceptance

  1. Count independent body findings by structure. For coderabbitai, every item under "Outside diff range comments" and "Nitpick comments" counts toward Findings raised whether or not the bot opened threads on that PR. Use the counts CodeRabbit prints in those section headers, or the items in them. For chatgpt-codex-connector, a body suggestion counts unless an inline comment on the same file and line exists from the same review. Other bots keep the current rule.
  2. Separate counts. The Details section shows, per bot, findings from threads and findings from review bodies separately. The headline Findings raised is their sum.
  3. Inline-safe rows. Sidecar rows of class status or skipped no longer apply to inline comments. Either the loader excludes them for kind: inline, as it already does for refused, or the rows are narrowed to kind: issue / review. A fixture shows an inline finding containing "Review Status" or "review in progress" staying a finding.
  4. Tests and docs. Fixtures in tests/reviewer_report.bats cover a CodeRabbit review with threads plus outside-diff and nitpick sections, and a Codex review with one suggestion matched by an inline comment and one not. docs/reviewer-report.md describes the body-finding rule for each bot.

Activity

  1. don-petry commented on Nov 13, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: implementation deferred — automation PR queue is full

    No pull request was opened for issue #2527. The org-wide
    open-automation-PR queue is currently at its signed-off cap (see
    standards/pr-limits.json in combsmith/.github).

    This issue stays labeled dev-lead and will be re-attempted automatically once
    the queue drains — no action is required.

  2. don-petry commented on Nov 13, 2026

    @don-petry
    CollaboratorAuthor

    dev-lead is withholding action on this item.

    It is labeled dev-lead:hands-off (deliberately kept off-limits to dev-lead (e.g. changes to the dev-lead automation itself)), so dev-lead will not pick it up while that label is present. This notice is posted once so the withhold is visible rather than looking like a stalled run.

    To re-enable automated pickup: remove the dev-lead:hands-off label.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupdev-lead:hands-offExclude from dev-lead agent automationenhancementFeature requestsreviewer-reportWeekly Third-Party Reviewer Scorecard report

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions