Repository navigation
feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report - #931
Conversation
…render the speed/quality/cost report
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
More reviews will be available in 49 minutes and 9 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughThis PR adds an LSP pilot report generator, synthetic candidate run captures, report documentation, and Bats/CI coverage for the new report workflow. ChangesLSP pilot report generation
Sequence Diagram(s)sequenceDiagram
participant render_pilot_report
participant "scripts/lsp_pilot_compare.sh" as lsp_pilot_compare_sh
participant lpr_candidate_verdict
render_pilot_report->>lsp_pilot_compare_sh: render per-candidate comparison output
render_pilot_report->>lpr_candidate_verdict: compute GO/NO-GO verdict
lpr_candidate_verdict-->>render_pilot_report: verdict text and exit status
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Code Review
This pull request implements the LSP pilot comparative report renderer (scripts/lsp_pilot_report.sh), adding documentation, candidate run datasets, and BATS unit tests. The review feedback focuses on improving script robustness under set -u by validating positional parameters across multiple functions, explicitly checking for CLI dependencies (jq and awk) at the entry point, and adding error handling for mktemp calls in the test suite.
|
Caution Failed to replace (edit) comment. This is likely due to insufficient permissions or the comment being deleted. Error details |
Dev-Lead — review-changes (applied)Changes committed and pushed. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/lsp_pilot_report.sh`:
- Around line 303-343: The coverage check in lpr_pilot_report.sh currently only
prints a warning, so a candidate with missing corpus PRs can still reach GO.
Update the verdict path around lpr_coverage, lpr_candidate_verdict, and the
verdicts/go_names handling so cov_rc != 0 forces NO-GO for that candidate. If
the reduction ratio is used for the final decision, also adjust the nav-total
logic to include uncovered corpus PRs instead of excluding them, so the
token-reduction result stays consistent with the full corpus.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 8ba83a14-54ea-4ddb-ad61-3cfdfe7815d7
📒 Files selected for processing (8)
.github/workflows/lint.ymldocs/lsp-pilot-report.mddocs/lsp-pilot.mdevals/lsp-pilot/README.mdevals/lsp-pilot/runs/agent-lsp.jsonlevals/lsp-pilot/runs/serena.jsonlscripts/lsp_pilot_report.shtests/lsp_pilot_report.bats
|
@coderabbitai resolve |
Dev-Lead — review-changes (applied)Changes committed and pushed. |
✅ Action performedComments resolved and changes approved. |
|
Dev-Lead — waiting on PR blockers (intent: fix-reviews)PR: #931 |
donpetry-bot
left a comment
There was a problem hiding this comment.
Automated review — APPROVED ✓
Risk: LOW
Reviewed commit: 49ace714746e827465364cedd316e45a10bf2b04
Review mode: triage-approved (single reviewer)
Summary
Additive LSP-pilot tooling: a new pure-bash comparative report renderer (scripts/lsp_pilot_report.sh) that consumes the existing story-2 harness (lsp_pilot_compare.sh) plus a 234-line Bats suite, a generated seed report, synthetic de-identified candidate run captures, a lint-workflow wiring line, and an MD024 markdownlint relaxation. No review-engine behavior change and no production code path touched.
Linked issue analysis
Closes #844 (story: run comparison for the LSP pilot, epic #839). The PR substantively delivers it: per-candidate GO/NO-GO success-metric verdict (>=2x nav-token reduction AND no false-positive/precision regression AND cold-start P95 within the 30s SLA), cold-start P50/P95 vs SLA, honest SLA-skip/MCP-degraded annotations, and a corpus-coverage smoke check. The report is explicitly flagged as seed/structural data until the real corpus is repinned, which matches the epic's contract.
Findings
No blocking findings.
- Security: pure bash, no network, no secrets/credentials, no eval or injection vectors; reads JSONL via jq/awk. agent-shield, CodeQL, gitleaks, and SonarCloud all green. Committed run data is synthetic/de-identified with zeroed SHAs.
- Advisory bots: gemini-code-assist raised 11 MEDIUM suggestions (arg-count guards under set -u; command -v dependency check). All are already implemented in the reviewed head — the functions carry the requested guards and main() checks jq/awk. Nothing outstanding.
- SonarCloud quality gate passed (0 new issues). Codex review hit a usage limit (no substantive feedback); CodeRabbit was rate-limited.
- Secret-scanning MCP tool not available in this environment; relied on the gitleaks CI check, which passed.
- Coverage: helpers and the full renderer are unit-tested, including SLA breach, MCP-degraded, MISSING-corpus, ratio, quality-regression, and the data-favored tie-break path.
CI status
All required checks green: Lint, shellcheck/ShellCheck, bats/unit-tests, CodeQL, Analyze (actions/python), agent-shield, SonarCloud, gitleaks, holdout-guard, validate-agent-profiles, gh-aw-compile. SKIPPED checks are inapplicable conditionals (dependency-audit ecosystems, dependabot). CANCELLED entries are the dev-lead agent's own dispatch/ci-relay relay jobs, not failures. No human review threads or unanswered questions; downstream-impact: none.
Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
…render the speed/quality/cost report (#931) * feat: implement issue #844 — [Phase 3] Run the comparative pilot and render the speed/quality/cost report * chore: apply manual instructions [skip ci-relay] * chore: apply manual instructions [skip ci-relay] --------- Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>



Closes #844
Implemented by dev-lead agent. Please review.
Summary by CodeRabbit
New Features
Documentation
Tests