Repository navigation
Description
Epic: #839 · Story: #844 · Builds on: #961 (capture/emit) + #952 (extractor/harness)
Context
PR #961 added the pilot capture/emit logic to scripts/engine.sh + scripts/review-one-pr.sh (gated on the env LSP_PILOT_ENABLED), but only touched the scripts — not .github/workflows/pr-review.yml. Two gaps leave the feature inert / incomplete in real CI. This issue closes both so a production pr-review can emit a comparable lsp-off (A) and lsp-on (B) lsp_pilot_run record pair.
Gap 1 — the capture env var is never plumbed
_lsp_pilot_active() (engine.sh) and the trap in review-one-pr.sh gate on the env LSP_PILOT_ENABLED=true, but pr-review.yml only references the repo variable vars.LSP_PILOT_ENABLED (to gate the setup-lsp-pilot step, lines 378/387). vars.* are not automatically env vars, and neither the job env: block nor setup-lsp-pilot.sh exports LSP_PILOT_ENABLED to GITHUB_ENV. Result: with the repo var on, the LSP gets wired but no transcript is captured and no record is emitted (the unit tests pass only because they set the env directly).
Gap 2 — capture and LSP wiring can't be decoupled
Both the capture and the setup-lsp-pilot step key off the same vars.LSP_PILOT_ENABLED, and lpe_variant() returns lsp-on whenever REVIEW_MCP_CONFIG is wired. So a production run can only ever produce an lsp-on (B) record — there is no way to produce the lsp-off (A) control leg (capture on, LSP wiring off). The comparison needs both legs from the same pipeline.
Acceptance criteria
- Plumb the capture gate. The review job in
pr-review.ymlmust exportLSP_PILOT_ENABLEDas an env var to the step(s) that run the engine /review-one-pr.sh, so_lsp_pilot_active()actually sees it. When capture is enabled and a review runs, akind:"lsp_pilot_run"record is emitted toTOKEN_LOG_FILE(already uploaded as thetoken-usage-<run_id>artifact). - Decouple capture from LSP wiring. Add an independent control so a run can enable capture/emit WITHOUT wiring the LSP MCP (→
lsp-offA leg) and another that enables capture WITH LSP wiring (→lsp-onB leg). Suggested shape: aworkflow_dispatchinput (with a matching repo-var default), e.g.lsp_pilot_variant: off | on | none—onrunssetup-lsp-pilot(wiresREVIEW_MCP_CONFIG) + capture;offruns capture only (skipssetup-lsp-pilot, noREVIEW_MCP_CONFIG);none/unset = today's behaviour. Preserve the existingvars.LSP_PILOT_ENABLEDmeaning for anyone already using it. - Both legs join. Dispatching the same PR twice (variant
off, thenon) must yield twolsp_pilot_runrecords with the sameprkey and variantslsp-off/lsp-on, thatscripts/lsp_pilot_compare.shrenders as an off-vs-on table. - Off-pilot unchanged. With neither the variable nor the input set,
pr-review.ymland the engine behave byte-for-byte as before — no stream capture, no records, no extra flags. The 5 consumer repos must be unaffected. - Thin-caller forwarding. If a new dispatch input is added, forward it through
pr-review-trigger.yml(the ring-0 trigger stub) within its allowed-input contract so the A/B can be driven on this repo's own PRs. Respect the thin-caller rules in AGENTS.md. - Tests + hygiene. Coverage for the new plumbing/decoupling;
shellcheck --severity=warning -x scripts/*.shclean;pr-review.ymlvalidates as YAML; existingtest_engine_lsp_pilot/test_lsp_pilot_emit/test_lsp_pilot_measuresuites stay green. Preserve all action/SHA pinning.
Out of scope
- Enabling the pilot in production / setting repo variables — owner-gated (that's the run, not this change).
- Any change to the
lsp_pilot_measure.sh/lsp_pilot_compare.shrecord schema. - The go/no-go decision ([Phase 3] Record the human go/no-go decision and fleet-wide recommendation #845).
Done when
Dispatching the ring-0 pr-review on one PR with the pilot variant off then on drops two joinable lsp_pilot_run records (lsp-off + lsp-on) into their token-usage artifacts, and scripts/lsp_pilot_compare.sh renders a real off-vs-on table — no synthetic data and no manual env hacks.
Activity
github-actions commentedon Jul 3, 2026
Thanks for the thorough write-up and clear acceptance criteria — this is a well-scoped issue! Before we dig in: for Gap 2's proposed
lsp_pilot_variantdispatch input, do you have a preference on whether theoffleg should still respectvars.LSP_PILOT_ENABLEDas a fallback default, or should the repo var exclusively gate the existing behaviour so the new input is fully independent?don-petry commentedon Jul 3, 2026
Dev-Lead Implementation Plan
Issue: #1031 — fix(#844): plumb LSP_PILOT_ENABLED env + decouple capture from LSP wiring so the pilot A/B runs in CI
Scope
Export the
LSP_PILOT_ENABLEDcapture gate as a real env var inpr-review.yml, and add anlsp_pilot_variant(on|off|none) control that decouples capture from LSP-MCP wiring so a production review can emit both anlsp-off(A) andlsp-on(B)lsp_pilot_runrecord from the same pipeline — with off-pilot behaviour byte-for-byte unchanged.Root cause notes
LSP_PILOT_ENABLED, but the workflow only referenced the repo varvars.LSP_PILOT_ENABLED(never exported to the job env) → capture never ran in CI.lpe_variant()returnslsp-onwheneverREVIEW_MCP_CONFIGis a readable file. Becauseengine.shauto-defaultsREVIEW_MCP_CONFIGto the committed.github/review-mcp.json(present in this repo) andreview-one-pr.shsourcesengine.sh, theoff(A) leg would be mislabelledlsp-on. Fix: an explicitLSP_PILOT_VARIANT(on|off) that is authoritative for the leg label, falling back toREVIEW_MCP_CONFIGdetection when unset (preserves legacyvars.LSP_PILOT_ENABLEDusers).Implementation Checklist
scripts/lsp_pilot_emit.sh—lpe_variant()honours explicitLSP_PILOT_VARIANT=on|off(authoritative), else falls back toREVIEW_MCP_CONFIGdetection;lpe_candidate()follows..github/workflows/pr-review.yml— export job-envLSP_PILOT_ENABLED(on whenvars.LSP_PILOT_ENABLED == 'true'OR variant is on/off) andLSP_PILOT_VARIANT; gate the two LSP-wiring steps (Cache LSP index, Set up LSP pilot servers) onvars.LSP_PILOT_ENABLED == 'true' || variant == 'on'; addlsp_pilot_variantworkflow_call input..github/workflows/pr-review-trigger.yml— addlsp_pilot_variantworkflow_dispatch input and forward it (thin-caller contract)..github/workflows/test-dev-lead.yml— run the new plumbing test + trigger onpr-review*.ymlpath changes.Tests to Write
tests/dev-lead/unit/test_lsp_pilot_emit.bats—LSP_PILOT_VARIANT=offforceslsp-offeven withREVIEW_MCP_CONFIGset;onforceslsp-onwith it unset;none/unset falls back; candidate + emit-record A-leg decoupling.tests/dev-lead/integration/test_pr_review_lsp_pilot_plumbing.py—pr-review.ymlexports the capture gate + variant env and gates wiring onon;pr-review-trigger.ymldeclares and forwardslsp_pilot_variant.Test Command
bats tests/dev-lead/unit/ && python3 tests/dev-lead/integration/test_pr_review_lsp_pilot_plumbing.py && bash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Jul 3, 2026
Dev-Lead: Implementation Complete
Plan Execution
pr-review.ymlnow exports the ENVLSP_PILOT_ENABLEDto the review job (the engine + emit trap read the env, not the repo var), so a production review captures the stream-json transcript and appends akind:"lsp_pilot_run"record toTOKEN_LOG_FILE(already uploaded astoken-usage-<run_id>).lsp_pilot_variant: on|off|nonecontrol (with avars.LSP_PILOT_VARIANTrepo-var default).onwires the LSP MCP + captures (lsp-on/B);offcaptures only, skips wiring (lsp-off/A);none/unset = today's behaviour.lpe_variant()honour explicitLSP_PILOT_VARIANTas authoritative — this is required becauseengine.shauto-defaultsREVIEW_MCP_CONFIGto the committed.github/review-mcp.json(present here) andreview-one-pr.shsourcesengine.sh, which would otherwise mislabel the A leg aslsp-on. Falls back toREVIEW_MCP_CONFIGdetection when unset (preserves the legacyvars.LSP_PILOT_ENABLEDpath).lsp_pilot_variantthrough the ring-0 trigger stubpr-review-trigger.yml.Resolved leg truth table (verified)
''''vars.LSP_PILOT_ENABLED=truetrue''offtrueoffontrueonnone''noneTest Results
Files Changed
.github/workflows/pr-review.yml: exportLSP_PILOT_ENABLED(capture gate) +LSP_PILOT_VARIANTjob env; addlsp_pilot_variantworkflow_call input; gate the two LSP-wiring steps (Cache LSP index, Set up LSP pilot servers) onvars.LSP_PILOT_ENABLED == 'true' || variant == 'on'..github/workflows/pr-review-trigger.yml: addlsp_pilot_variantworkflow_dispatch input (choice: none/off/on) and forward it to the reusable.scripts/lsp_pilot_emit.sh:lpe_variant()treats explicitLSP_PILOT_VARIANT=on|offas authoritative, else falls back toREVIEW_MCP_CONFIGdetection.tests/dev-lead/unit/test_lsp_pilot_emit.bats: 7 new tests for the variant authority + A/B emit decoupling; hermetic-unset ofLSP_PILOT_VARIANT.tests/dev-lead/integration/test_pr_review_lsp_pilot_plumbing.py: new guard for the capture-gate export, variant decoupling, wiring-step gates, and trigger-stub forwarding..github/workflows/test-dev-lead.yml: run the new plumbing test as a job; addpr-review*.ymlto the trigger paths.Notes
lsp_pilot_measure.sh/lsp_pilot_compare.sh) is untouched — out of scope. The A/B pair still joins on the sameprkey with variantslsp-off/lsp-on, whichlsp_pilot_compare.shrenders unchanged..github/mcp/lsp.jsonand.github/review-mcp.jsonexist in this repo, so a realonrun wires the LSP config (overriding the auto-default) and a realoffrun keeps the baseline yet is correctly labelledlsp-off.feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …
feat: implement issue #1031 — fix(#844): plumb LSP_PILOT_ENABLED env …