Repository navigation
Closed
Labels
dev-leadFor dev-lead agent pickupFor dev-lead agent pickupinitiativeEpic / initiative tracking issueEpic / initiative tracking issue
Description
Story
As a deep-review agent,
I want see a few-shot examples of past review→merge outcomes for this repo injected into my prompt,
so that I calibrate to what this repo's maintainers actually accept and flag, improving precision cheaply.
Acceptance Criteria
- A context-injection step attaches few-shot examples drawn from this repo's past review→merge histories into the deep-tier prompt input.
- Few-shot examples are sourced ONLY from a proposer-visible / dev split (never from evals/deep-review/holdout), so the reviewer is never shown a case the held-out gate later scores against.
- The injection is opt-in and inert by default, and bounded in size so it stays within the ET cost cap.
- Scored against evals/deep-review/holdout only: aggregate score does not regress versus the Phase-1 frozen baseline, and median ET stays within the 1.5× cost cap.
Tasks / Subtasks
- Build a script that extracts de-identified few-shot examples from merged-PR review histories and stores them in a dev-visible split. (AC: test issue from agent #1, Go-live improvements for PR review agent #2)
- Inject the examples into the deep-review prompt via the optional-file contract ($FEWSHOT_FILE, analogous to $DOWNSTREAM_IMPACT_FILE), size-bounded. (AC: test issue from agent #1, Add @claude delegation, auto-merge, and rebase handling #3)
- Add a guard/assertion that the few-shot source path can never be evals/**/holdout (mirror the holdout-guard rule). (AC: Go-live improvements for PR review agent #2)
- Run scripts/evals/run-eval.sh deep-review on the holdout split; compare score + median ET to the frozen baseline. (AC: Optimize review: small-PR and incremental fast paths #4)
Dev Notes
- The held-out discipline is non-negotiable here: evals/README.md documents the dev/holdout split and the no-overlap rule; the proposer (few-shot source) must be built from the dev split ONLY, and holdout-guard.yml + scripts/lib/holdout-guard.sh + the CODEOWNERS rule over evals/ enforce that a proposer identity cannot touch a held-out path. Store few-shot data under a dev-visible path, and add a hard guard that the source is never evals/**/holdout.
- De-identification is required (evals/README.md decision A3): strip secrets, tokens, internal hostnames, and PII from any extracted PR content before committing.
- Injection uses the same optional-file prompt contract as downstream-impact / safety-checks in prompts/deep-review.md ($FEWSHOT_FILE) — keep it backward-compatible and size-bounded so it does not blow the ET cap.
- Eval-tuning story: score ONLY against evals/deep-review/holdout, compare to the Phase-1 frozen baseline, never mutate the baseline. This is exactly the overfitting risk the held-out split exists to prevent — the guard in task 3 is the safeguard.
- Testing: bats unit test asserting the few-shot source refuses a holdout path + de-identification scrub; shellcheck.
Project Structure Notes
Few-shot extractor under scripts/lib/ or scripts/evals/; dev-split data under a proposer-visible path (NOT evals/**/holdout). Reuses the optional-file prompt contract in prompts/deep-review.md.
References
- evals/README.md
- scripts/lib/holdout-guard.sh
- prompts/deep-review.md
- scripts/lib/downstream-impact.sh
- scripts/evals/run-eval.sh
- evals/deep-review/holdout/cases.jsonl
- AGENTS.md#cost-reporting
Likely target surface
scripts/evals/scripts/lib/prompts/deep-review.mdtests/dev-lead/unit/
Story prepared by the BMAD Scrum Master (Bob) for epic #1088. Status: ready-for-dev.
Metadata
Metadata
Assignees
Labels
dev-leadFor dev-lead agent pickupFor dev-lead agent pickupinitiativeEpic / initiative tracking issueEpic / initiative tracking issue
Activity
don-petry commentedon Jul 14, 2026
Dev-Lead: issue #1093 implementation failed — will retry
All engines were rate-limited. This was attempt 1 of 3; dev-lead will retry automatically. You can also re-apply the
dev-leadlabel to retry now.rate-limitedLast 40 lines of engine output (redacted)
Attempt 9 failed with status 429. Retrying with backoff... _ApiError: {"error":{"message":"{\n \"error\": {\n \"code\": 429,\n \"message\": \"Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. \",\n \"status\": \"RESOURCE_EXHAUSTED\"\n }\n}\n","code":429,"status":"Too Many Requests"}} at throwErrorIfNotOK (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:258424:24) at process.processTicksAndRejections (node:internal/process/task_queues:103:5) at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:258187:7 at async Models.generateContentStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:259283:16) at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:301279:19 at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:278341:23 at async retryWithBackoff (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:298287:23) at async GeminiChat.makeApiCallAndProcessStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:321564:28) at async GeminiChat.streamWithRetries (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:321382:29) at async Turn.run (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:322128:24) { status: 429 } Attempt 10 failed: Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. . Max attempts reached Error when talking to Gemini API Full report available at: /tmp/gemini-client-error-Turn.run-sendMessageStream-2026-07-14T11-52-23-387Z.json RetryableQuotaError: Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. at classifyGoogleError (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:297599:14) at retryWithBackoff (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:298310:31) at process.processTicksAndRejections (node:internal/process/task_queues:103:5) at async GeminiChat.makeApiCallAndProcessStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:321564:28) at async GeminiChat.streamWithRetries (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:321382:29) at async Turn.run (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:322128:24) at async GeminiClient.processTurn (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:335620:22) at async GeminiClient.sendMessageStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:335717:14) at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/gemini-APOZRZEF.js:23725:26 at async main (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/gemini-APOZRZEF.js:29133:5) { cause: { code: 429, message: 'Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. ', details: [] }, retryDelayMs: undefined } { "session_id": "0525cce9-3771-497b-81c3-d2a34615f177", "error": { "type": "Error", "message": "Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. ", "code": 1 } }don-petry commentedon Jul 14, 2026
Dev-Lead: issue #1093 implementation failed — will retry
All engines were rate-limited. This was attempt 2 of 3; dev-lead will retry automatically. You can also re-apply the
dev-leadlabel to retry now.rate-limitedLast 40 lines of engine output (redacted)
Attempt 9 failed with status 429. Retrying with backoff... _ApiError: {"error":{"message":"{\n \"error\": {\n \"code\": 429,\n \"message\": \"Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. \",\n \"status\": \"RESOURCE_EXHAUSTED\"\n }\n}\n","code":429,"status":"Too Many Requests"}} at throwErrorIfNotOK (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:258424:24) at process.processTicksAndRejections (node:internal/process/task_queues:103:5) at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:258187:7 at async Models.generateContentStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:259283:16) at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:301279:19 at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:278341:23 at async retryWithBackoff (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:298287:23) at async GeminiChat.makeApiCallAndProcessStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:321564:28) at async GeminiChat.streamWithRetries (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:321382:29) at async Turn.run (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:322128:24) { status: 429 } Attempt 10 failed: Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. . Max attempts reached Error when talking to Gemini API Full report available at: /tmp/gemini-client-error-Turn.run-sendMessageStream-2026-07-14T14-16-55-785Z.json RetryableQuotaError: Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. at classifyGoogleError (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:297599:14) at retryWithBackoff (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:298310:31) at process.processTicksAndRejections (node:internal/process/task_queues:103:5) at async GeminiChat.makeApiCallAndProcessStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:321564:28) at async GeminiChat.streamWithRetries (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:321382:29) at async Turn.run (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:322128:24) at async GeminiClient.processTurn (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:335620:22) at async GeminiClient.sendMessageStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:335717:14) at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/gemini-APOZRZEF.js:23725:26 at async main (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/gemini-APOZRZEF.js:29133:5) { cause: { code: 429, message: 'Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. ', details: [] }, retryDelayMs: undefined } { "session_id": "1d30fdc2-e45a-4b9c-a340-07c1364f1c4c", "error": { "type": "Error", "message": "Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. ", "code": 1 } }don-petry commentedon Jul 14, 2026
Dev-Lead: cannot implement issue #1093 — needs human attention
Automatic retries are exhausted (attempt 3 of 3). The most recent failure was
rate-limited.Please review the failures above. To retry anyway, remove the
dev-lead:needs-humanlabel and re-apply thedev-leadlabel.Last 40 lines of engine output (redacted)
Attempt 9 failed with status 429. Retrying with backoff... _ApiError: {"error":{"message":"{\n \"error\": {\n \"code\": 429,\n \"message\": \"Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. \",\n \"status\": \"RESOURCE_EXHAUSTED\"\n }\n}\n","code":429,"status":"Too Many Requests"}} at throwErrorIfNotOK (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:258424:24) at process.processTicksAndRejections (node:internal/process/task_queues:103:5) at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:258187:7 at async Models.generateContentStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:259283:16) at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:301279:19 at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:278341:23 at async retryWithBackoff (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:298287:23) at async GeminiChat.makeApiCallAndProcessStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:321564:28) at async GeminiChat.streamWithRetries (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:321382:29) at async Turn.run (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:322128:24) { status: 429 } Attempt 10 failed: Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. . Max attempts reached Error when talking to Gemini API Full report available at: /tmp/gemini-client-error-Turn.run-sendMessageStream-2026-07-14T15-50-23-484Z.json RetryableQuotaError: Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. at classifyGoogleError (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:297599:14) at retryWithBackoff (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:298310:31) at process.processTicksAndRejections (node:internal/process/task_queues:103:5) at async GeminiChat.makeApiCallAndProcessStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:321564:28) at async GeminiChat.streamWithRetries (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:321382:29) at async Turn.run (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:322128:24) at async GeminiClient.processTurn (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:335620:22) at async GeminiClient.sendMessageStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-W4YMYD53.js:335717:14) at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/gemini-APOZRZEF.js:23725:26 at async main (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/gemini-APOZRZEF.js:29133:5) { cause: { code: 429, message: 'Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. ', details: [] }, retryDelayMs: undefined } { "session_id": "abad247f-5e2f-4c27-91d9-21e50b882b83", "error": { "type": "Error", "message": "Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. Learn more at https://ai.google.dev/gemini-api/docs/billing#prepay. ", "code": 1 } }don-petry commentedon Sep 1, 2026
Dev-Lead Implementation Plan
Issue: #1093 — [Phase 2] Inject merged-PR few-shot examples into the deep tier (dev-split only)
Scope
Add an opt-in, inert-by-default context-injection pass that attaches size-bounded, de-identified few-shot review→merge examples (sourced ONLY from a proposer-visible dev split, never from
evals/**/holdout) into the deep-review tier via the$FEWSHOT_FILEoptional-file contract, mirroring$DOWNSTREAM_IMPACT_FILE/$SYMBOL_CONTEXT_FILE.Implementation Checklist
scripts/lib/fewshot.sh— new lib:fewshot_scrub(de-identify: secrets/tokens/hostnames/emails/PII),fewshot_source_is_holdout(hard guard predicate rejecting anyevals/**/holdout/**source),assemble_fewshot(read dev-split JSONL, refuse holdout source, de-identify, bound count + bytes, write block, exportFEWSHOT_FILE; always inert-safe).scripts/evals/extract-fewshot.sh— extractor that de-identifies merged-PR review→merge histories into the dev-visible split (refuses a holdout output path; reusesfewshot_scrub+ the holdout guard).evals/deep-review/dev/fewshot.jsonl— seed de-identified dev-split few-shot data (proposer-visible; NOT holdout).scripts/review-one-pr.sh— sourcefewshot.sh; gatedFEWSHOT_ENABLED(default off → byte-identical prompt, zero cost); assemble + exportFEWSHOT_FILEfor the deep tier.prompts/deep-review.md— add$FEWSHOT_FILEto Inputs + a step to weigh it (backward-compatible, informational, not an auto-escalation trigger)..github/workflows/lint.yml— register the new bats test in thebatsjob list.Tests to Write
tests/dev-lead/unit/test_fewshot_assemble.bats— asserts: (a)assemble_fewshotREFUSES aevals/**/holdoutsource path (hard guard), (b) de-identification scrub strips secrets/tokens/hostnames/emails from injected content, (c) block is bounded by max-examples + max-bytes, (d)FEWSHOT_FILEis exported and points at the written file, (e) a missing/empty source → literal(none), zero injection (inert).Test Command
bats tests/dev-lead/unit/test_fewshot_assemble.bats(full: thebatsjob list +shellcheck --severity=warning -x scripts/**/*.sh+bash .dev-lead/scripts/dev-lead-lint.sh)Note on AC #4
The holdout-eval scoring step (
scripts/evals/run-eval.sh deep-reviewvs. the frozen Phase-1 baseline) requires a credentialed model run and is executed in CI, not in this pre-push workspace — it will be reported there, not fabricated here.don-petry commentedon Sep 1, 2026
Dev-Lead: Progress (in progress — not yet durable)
Plan Execution (intended)
scripts/lib/fewshot.sh:fewshot_source_is_holdout(hard guard mirroring the holdout-guard rule),fewshot_scrub(de-identify secrets/tokens/hostnames/emails/IPs →[REDACTED]),assemble_fewshot(read dev-split JSONL → bounded, de-identifiedFEWSHOTblock; exportsFEWSHOT_FILE; refuses a holdout source).scripts/evals/extract-fewshot.sh: de-identifies merged-PR review→merge histories into the dev split; refuses a holdout--out.evals/deep-review/dev/fewshot.jsonl(de-identified, proposer-visible; NOT holdout).scripts/review-one-pr.shbehindFEWSHOT_ENABLED(default off → byte-identical prompt/cost); exportsFEWSHOT_FILEfor the deep tier only.$FEWSHOT_FILEtoprompts/deep-review.mdInputs + a step framing it as informational calibration context (never an auto-approve/escalate trigger, never overrides the taxonomy or a hard-stop).tests/dev-lead/unit/test_fewshot_assemble.batsin thelint.ymlbats job.Local check output (provisional — not the pushed run)
Files touched (this workspace, not yet pushed)
scripts/lib/fewshot.sh(new): guard + scrub + assembler.scripts/evals/extract-fewshot.sh(new): dev-split extractor.evals/deep-review/dev/fewshot.jsonl(new): seed dev-split data.scripts/review-one-pr.sh: source the lib + gated tier-2 assembly.prompts/deep-review.md:$FEWSHOT_FILEinput + step..github/workflows/lint.yml: register the new bats test.tests/dev-lead/unit/test_fewshot_assemble.bats(new): guard/scrub/bounds/inert.Notes
run-eval.sh deep-reviewon the holdout vs. the frozen Phase-1 baseline, median ET within the 1.5× cap) requires a credentialed model run and runs in CI, not this pre-push workspace — reported there, not fabricated here. By construction the feature is default-OFF (FEWSHOT_ENABLEDunset), so the holdout eval is byte-identical to the frozen baseline unless explicitly enabled; the frozen baseline fixture is untouched (test_deep_review_baseline.batsstill green).holdout-guard.ymloverevals/**/holdout/.don-petry commentedon Sep 1, 2026
Dev-Lead: Implementation Complete — PR #1622
The implementation for issue #1093 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.
b37877721cd419b963b86d970c26cc3383761e33Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.
feat: implement issue #1093 — [Phase 2] Inject merged-PR few-shot exa…