Repository navigation
feat(pr-limits): add the no-LLM success-metric report (#510) - #573
Conversation
Phase 4 of the PR-limits initiative (epic #505): measure the success metric — open, non-draft, automation-authored PRs org-wide vs. the signed-off cap. - scripts/pr-limits-report.sh: pure gh + jq (NO LLM). Reads the cap + exempt actors/labels from standards/pr-limits.json (single source of truth, nothing hardcoded) and applies the SAME counting semantics as the admission gate (scripts/lib/pr-limit-gate.sh) so the report and the gate always agree. Prints a Markdown summary (total / counted / exempt / cap / headroom + a by-source breakdown + an over/under-cap status line) and appends it to $GITHUB_STEP_SUMMARY. Robust to an empty/failed enumeration (reports 0). - .github/workflows/pr-limits-report.yml: schedule (once daily) + manual dispatch; permissions: {} with a contents:read job; github.token for gh search prs; pinned checkout with persist-credentials:false; timeout 10. Modeled on daily-org-status.yml. - test/scripts/pr-limits/pr-limits-report.bats: 8 gh-stubbed cases (under/at/ over cap, exempt exclusion, by-source, empty queue, read-only gh, cap read from config). - .github/workflows/pr-limits-report-tests.yml: shellcheck + bats gate for the report, modeled on auto-rebase-tests.yml. shellcheck clean; 8/8 new bats (28/28 in test/scripts/pr-limits/); actionlint clean. Hand-authored after the dev-lead engine hit its 600s timeout on #510. Closes #510 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HUhVsBbihbTjVvbcVf1WuU
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
📝 WalkthroughWalkthroughAdds a deterministic PR-limits reporting script ( ChangesPR-limits report
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Dev-Lead — review-changes (no-changes)No changes were needed for this PR. |
There was a problem hiding this comment.
Code Review
This pull request introduces scripts/pr-limits-report.sh, a bash script that generates a Markdown report comparing open, non-draft, automation-authored PRs against a configured cap, alongside a comprehensive BATS test suite in test/scripts/pr-limits/pr-limits-report.bats. The review feedback highlights critical improvements for robustness, specifically recommending the use of the optional/try operator (?) in jq to prevent fatal errors when a PR has a null or missing author field, adding a warning when the gh command fails completely, and adding a corresponding test case to verify graceful handling of null authors.
There was a problem hiding this comment.
Pull request overview
Introduces the Phase 4 PR-limits “success metric” report: a no-LLM gh + jq script and scheduled workflow that measures open non-draft automation PR volume against the configured cap, with a dedicated ShellCheck+Bats quality gate.
Changes:
- Added
scripts/pr-limits-report.shto compute and render a Markdown cap/headroom summary plus a by-author breakdown fromstandards/pr-limits.json. - Added a daily + manual GitHub Actions workflow to publish the report into
$GITHUB_STEP_SUMMARYusinggithub.token. - Added Bats tests and a dedicated workflow gate (
shellcheck+bats) for the report script.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
test/scripts/pr-limits/pr-limits-report.bats |
Adds gh-stubbed Bats coverage for report status/headroom math, exemptions, and read-only behavior. |
scripts/pr-limits-report.sh |
Implements the cap/exempt-aware org-wide PR enumeration and Markdown report rendering. |
.github/workflows/pr-limits-report.yml |
Schedules and dispatches the report job, writing output to the job summary. |
.github/workflows/pr-limits-report-tests.yml |
Adds a dedicated ShellCheck + Bats gate for PR-limits report changes. |
Superseded by automated re-review at
|
Address advisory review on #573: - gh-failure vs empty (Gemini/Copilot): capture the gh exit code and, on a real failure, print '⚠️ Metric unavailable' + exit 1 instead of silently reporting a misleading '0/cap Under cap' table. A scheduled report must fail loudly when it can't measure (the opposite of the admission gate's fail-open, which is correct for PR creation). rc=0 with an empty result is still a real '0 open PRs'. - Tests (+3, now 11): null/missing author counted as 'unknown' with no jq error (confirms the existing // guards are null-safe); interleaved authors group correctly regardless of input order (locks in jq group_by's internal sort); gh-failure prints Metric unavailable and exits non-zero with no fake table. Null-author .author.login accesses left unchanged — jq returns null when indexing null with a field key, caught by // — verified and now tested. shellcheck clean; 11/11 bats (31/31 in test/scripts/pr-limits/). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HUhVsBbihbTjVvbcVf1WuU
|
@donpetry-bot review |
|
@don-petry I'm on it — starting a fresh review now. Results will appear in a few minutes. |
donpetry-bot
left a comment
There was a problem hiding this comment.
Automated review — APPROVED ✓
Risk: MEDIUM
Reviewed commit: 403156480483a7c6c17f1f172eeaa0aeaafeac57
Cascade: triage → deep (triage: haiku 4.5 → deep: opus 4.8 + duck: o4-mini → audit: fable 5)
Summary
New no-LLM PR-limits success-metric report (#510): a pure gh+jq script plus a scheduled report workflow, a test workflow, and 12 bats tests. The triage escalation (inconsistent null-safety at pr-limits-report.sh:93) is resolved at head 4031564 — both jq author-login accessors now use the // "" / // "unknown" guard (verified empirically: null and missing authors collapse to "unknown", exit 0, no jq error). Gemini's medium finding (silent gh failure) is fixed via an explicit gh_rc!=0 -> exit 1 'Metric unavailable' branch, and its low finding is covered by a dedicated null/missing-author bats test. Downstream impact: (none).
Findings
- INFO: Advisory HIGH finding resolved: scripts/pr-limits-report.sh:93
is_exempt_actornow uses.author.login // "", consistent with line 101's.author.login // "unknown". jq indexing of a null/missing author returns null (not an error) and the//guard supplies a defined fallback; verified null+missing author payload yields exit 0 and groups as 'unknown'. - INFO: Advisory MEDIUM finding resolved: a
gh search prsfailure (gh_rc != 0) now fails loudly with a 'Metric unavailable' summary and exit 1, distinguished from a genuinely empty queue (rc 0, empty/[] -> reported as 0). Prevents a transient API error masquerading as a false-positive 'Under cap 0/cap'. - INFO: Advisory LOW finding resolved: bats suite adds 'null or missing author is counted as unknown without a jq error', asserting no 'Cannot index' leakage and correct 'unknown' grouping. Suite also covers under/at/over cap, exempt actor/label exclusion, empty queue, read-only gh usage, and config-driven cap.
- INFO: Both new workflows follow least-privilege: pr-limits-report.yml uses
permissions: {}at top with job-levelcontents: read, pr-limits-report-tests.yml usescontents: read; both SHA-pin actions/checkout@v7.0.0 with persist-credentials:false, no pull_request_target, no untrusted github.event interpolation, token limited to github.token. No Actions security smells. - INFO: run_secret_scanning MCP tool not available in this environment; skipped per instructions (never block on MCP unavailability). No credentials introduced by the diff (pure gh+jq report) and the gitleaks CI check passed.
Reviewed by the PR-review cascade (triage: haiku 4.5 → deep: opus 4.8 + duck: o4-mini → audit: fable 5). Reply if you need a human review.
|
Dev-Lead — waiting on PR blockers (intent: review-changes)PR: #573 |
|
Note @don-petry I reviewed this PR and no code changes were needed, but it still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews), so I cannot mark it done yet. I'll re-check automatically. |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/pr-limits-report.sh`:
- Around line 90-107: The METRICS jq filter in pr-limits-report.sh is doing a
redundant second pass to compute exempt by reselecting the full PR array and
re-evaluating counts_toward_cap. Update the jq object in the METRICS assignment
so counted is computed once from the filtered array, and derive exempt
arithmetically as total minus counted. Keep the existing counts_toward_cap
helper and by_source logic intact.
- Around line 60-75: The PR-limits report logic in the gh search prs flow is
treating a truncated 1,000-item result set as the full queue. Update the script
around the PRS retrieval and TOTAL/COUNTED calculation to detect when the search
hits GitHub’s ceiling, and emit a clear capped-result warning instead of
reporting “Under cap” from incomplete data. If needed, split the query or add
pagination/partitioning in the reporting path so the summary logic cannot rely
on a truncated array from gh search prs.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: f84d46c5-958d-41e6-8aa4-82ed322be85c
📒 Files selected for processing (4)
.github/workflows/pr-limits-report-tests.yml.github/workflows/pr-limits-report.ymlscripts/pr-limits-report.shtest/scripts/pr-limits/pr-limits-report.bats
| PRS="$(gh search prs \ | ||
| --owner "$ORG" \ | ||
| --state open \ | ||
| --draft=false \ | ||
| --limit 1000 \ | ||
| --json author,labels,repository,title,url \ | ||
| 2>/dev/null)" || gh_rc=$? | ||
|
|
||
| if [ "$gh_rc" -ne 0 ]; then | ||
| UNAVAILABLE="$(printf '## PR-limits report — %s\n\n> ⚠️ **Metric unavailable** — `gh search prs` failed (rc=%s); the open-PR queue could not be enumerated. No cap comparison was produced.\n' "$ORG" "$gh_rc")" | ||
| printf '%s\n' "$UNAVAILABLE" | ||
| if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then | ||
| printf '%s\n' "$UNAVAILABLE" >>"$GITHUB_STEP_SUMMARY" | ||
| fi | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the target script around the cited lines.
git ls-files scripts/pr-limits-report.sh
wc -l scripts/pr-limits-report.sh
cat -n scripts/pr-limits-report.sh | sed -n '45,130p'Repository: petry-projects/.github
Length of output: 4545
🌐 Web query:
GitHub Search API 1000 result cap total_count no pagination documentation gh search prs limit 1000
💡 Result:
The GitHub Search API (both REST and GraphQL) imposes a hard limit of 1,000 results for any given search query [1][2]. While the API response includes a total_count field representing the total number of items matching your query, you cannot retrieve more than 1,000 records regardless of the value in that field [1][3]. This limitation applies to all search endpoints, including issues, pull requests, code, and repositories [1][2]. Although the API supports pagination (via the page and per_page parameters in REST, or standard GraphQL connections), pagination will only allow you to traverse up to the first 1,000 items [1][2][4]. Requests attempting to access results beyond this threshold will typically fail or return an empty set once the 1,000-result limit is reached [5][6]. To work around this constraint when you need to access more than 1,000 results, you must decompose your search into smaller, more specific queries [5]. Common strategies include: 1. Refining your search criteria: Use more granular qualifiers (e.g., date ranges, repository filters, language constraints, or label exclusions) to narrow the total number of matches to under 1,000 [5][6]. 2. Batching by date or time intervals: If you are searching for issues or pull requests, you can split your query by creation or update dates (e.g., searching for items created in specific monthly or weekly blocks) to ensure each batch returns fewer than 1,000 results [5]. Regarding the GitHub CLI (gh), commands such as gh search prs automatically handle pagination behind the scenes but remain subject to this same underlying API restriction; they cannot bypass the 1,000-item limit [6].
Citations:
- 1: https://docs.github.com/en/rest/search/search?apiVersion=2022-1128
- 2: https://docs.github.com/en/enterprise-cloud@latest/graphql/reference/search
- 3: Search results limited to 1000 per search query PyGithub/PyGithub#1072
- 4: https://github.com/octokit/plugin-rest-endpoint-methods.js/blob/main/docs/search/issuesAndPullRequests.md
- 5: https://stackoverflow.com/questions/61810553/how-to-get-more-than-1000-search-results-with-api-github
- 6: Support paginating
gh search codecli/cli#9734
Warn when the search result set hits GitHub’s 1,000-item ceiling.
gh search prs is capped at 1,000 matches, and this script only measures the returned array length. If the org grows past that threshold, TOTAL/COUNTED will be truncated and the summary can still show “Under cap” from an incomplete sample. Flag the cap hit or split the query so the report doesn’t present a capped result set as the full queue.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/pr-limits-report.sh` around lines 60 - 75, The PR-limits report logic
in the gh search prs flow is treating a truncated 1,000-item result set as the
full queue. Update the script around the PRS retrieval and TOTAL/COUNTED
calculation to detect when the search hits GitHub’s ceiling, and emit a clear
capped-result warning instead of reporting “Under cap” from incomplete data. If
needed, split the query or add pagination/partitioning in the reporting path so
the summary logic cannot rely on a truncated array from gh search prs.
| METRICS="$(jq -c \ | ||
| --argjson exempt_actors "$EXEMPT_ACTORS" \ | ||
| --argjson exempt_labels "$EXEMPT_LABELS" ' | ||
| def is_exempt_actor: ((.author.login // "") as $a | ($exempt_actors | index($a)) != null); | ||
| def has_exempt_label: ([.labels[]?.name] | map(. as $n | $exempt_labels | index($n)) | any); | ||
| def counts_toward_cap: ((is_exempt_actor | not) and (has_exempt_label | not)); | ||
| { | ||
| total: length, | ||
| counted: ([ .[] | select(counts_toward_cap) ] | length), | ||
| exempt: ([ .[] | select(counts_toward_cap | not) ] | length), | ||
| by_source: ( | ||
| [ .[] | select(counts_toward_cap) | (.author.login // "unknown") ] | ||
| | group_by(.) | ||
| | map({ source: .[0], count: length }) | ||
| | sort_by(-.count) | ||
| ) | ||
| } | ||
| ' <<<"$PRS")" |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
Redundant double pass over the PR array for counted/exempt.
counted and exempt each re-select the full array and re-evaluate counts_toward_cap per element. Since exempt is simply total - counted, this can be derived arithmetically instead of a second full select/length pass.
♻️ Proposed simplification
{
total: length,
counted: ([ .[] | select(counts_toward_cap) ] | length),
- exempt: ([ .[] | select(counts_toward_cap | not) ] | length),
+ exempt: (length - ([ .[] | select(counts_toward_cap) ] | length)),
by_source: (📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| METRICS="$(jq -c \ | |
| --argjson exempt_actors "$EXEMPT_ACTORS" \ | |
| --argjson exempt_labels "$EXEMPT_LABELS" ' | |
| def is_exempt_actor: ((.author.login // "") as $a | ($exempt_actors | index($a)) != null); | |
| def has_exempt_label: ([.labels[]?.name] | map(. as $n | $exempt_labels | index($n)) | any); | |
| def counts_toward_cap: ((is_exempt_actor | not) and (has_exempt_label | not)); | |
| { | |
| total: length, | |
| counted: ([ .[] | select(counts_toward_cap) ] | length), | |
| exempt: ([ .[] | select(counts_toward_cap | not) ] | length), | |
| by_source: ( | |
| [ .[] | select(counts_toward_cap) | (.author.login // "unknown") ] | |
| | group_by(.) | |
| | map({ source: .[0], count: length }) | |
| | sort_by(-.count) | |
| ) | |
| } | |
| ' <<<"$PRS")" | |
| METRICS="$(jq -c \ | |
| --argjson exempt_actors "$EXEMPT_ACTORS" \ | |
| --argjson exempt_labels "$EXEMPT_LABELS" ' | |
| def is_exempt_actor: ((.author.login // "") as $a | ($exempt_actors | index($a)) != null); | |
| def has_exempt_label: ([.labels[]?.name] | map(. as $n | $exempt_labels | index($n)) | any); | |
| def counts_toward_cap: ((is_exempt_actor | not) and (has_exempt_label | not)); | |
| { | |
| total: length, | |
| counted: ([ .[] | select(counts_toward_cap) ] | length), | |
| exempt: (length - ([ .[] | select(counts_toward_cap) ] | length)), | |
| by_source: ( | |
| [ .[] | select(counts_toward_cap) | (.author.login // "unknown") ] | |
| | group_by(.) | |
| | map({ source: .[0], count: length }) | |
| | sort_by(-.count) | |
| ) | |
| } | |
| ' <<<"$PRS")" |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/pr-limits-report.sh` around lines 90 - 107, The METRICS jq filter in
pr-limits-report.sh is doing a redundant second pass to compute exempt by
reselecting the full PR array and re-evaluating counts_toward_cap. Update the jq
object in the METRICS assignment so counted is computed once from the filtered
array, and derive exempt arithmetically as total minus counted. Keep the
existing counts_toward_cap helper and by_source logic intact.



Phase 4 of the PR-limits initiative (epic #505): the success-metric report.
What
A pure gh + jq (no LLM) report measuring open non-draft automation PRs org-wide vs. the signed-off cap.
scripts/pr-limits-report.sh— reads cap + exempt actors/labels fromstandards/pr-limits.json(single source of truth) and applies the same counting semantics as the admission gate (scripts/lib/pr-limit-gate.sh) so report and gate agree. Emits a Markdown table (total / counted / exempt / cap / headroom) + by-source breakdown + status line, and appends to$GITHUB_STEP_SUMMARY. Robust to empty/failed enumeration..github/workflows/pr-limits-report.yml—schedule(once daily) +workflow_dispatch;permissions: {}+contents: readjob;github.token; pinned checkout,persist-credentials: false;timeout-minutes: 10. Modeled ondaily-org-status.yml.test/scripts/pr-limits/pr-limits-report.bats(8 gh-stubbed cases) + apr-limits-report-tests.ymlshellcheck+bats gate.Validation
shellcheck clean; 8/8 new bats (28/28 in
test/scripts/pr-limits/); actionlint clean. Current real numbers: 41/50 counted, headroom 9.Note
Hand-authored after the dev-lead engine hit its 600s timeout on #510 (3 attempts). The story also referenced a phantom
auto_rebase_health.sh; this uses the realorg_status.sh/daily-org-status.ymlmodels. Closes #510.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes