Repository navigation
Description
Why this exists
Epic #839 piloted LSP (language-server) code intelligence in the pr-review harness for exactly one language/server: Shell via bash-language-server behind agent-lsp. The measured result was a NO-GO (decision: docs/lsp-pilot.md §7, recorded on #845):
| Run | Model | Agg nav-token ratio | High-nav stratum |
|---|---|---|---|
| 28718240665 | sonnet-4-6 | 1.26× | 1.01× |
| 28940391169 | opus-4-8 (prod) | 1.05× | 1.04× |
The ≥2× navigation-token target was missed decisively; on shell diffs already cheap to grep, agent-lsp's MCP tool-result payloads cost about as much context as the reads they replace. That result does not generalize — it says nothing about languages with deeper cross-file type resolution, which is exactly where find-references is expected to beat grep.
This is a holding issue (inert: initiative:hold, no initiative:auto) to keep the open door tracked. Promote a child to a real pilot by removing the hold and planning it; do not auto-release.
The reusable harness is already in place (dormant, cost-free)
A per-language re-pilot is a restart, not a rebuild — reuse:
scripts/lsp_pilot_run.sh— live A/B runner (LSP-off vs LSP-on on the same PRs)scripts/lsp_pilot_measure.sh/scripts/lsp_pilot_compare.sh— nav-token / tool-call / cold-start extraction + rendering.github/workflows/lsp-pilot-run.yml— on-demandworkflow_dispatchdriverevals/lsp-pilot/— the frozen-corpus pattern (repin per language)
Per-language deltas needed each time: (a) pick + version-pin the LSP server, (b) point .github/mcp/lsp.json at it (or a per-lang variant) and update setup-lsp-pilot.sh install, (c) repin evals/lsp-pilot/holdout/cases.jsonl to real cross-file PRs in that language's repos, (d) dispatch the A/B.
Method fixes carried over from the Shell pilot (do these before re-running)
- Gate on tokens first. The success metric is a conjunction (≥2× nav-token reduction AND no precision regression). Precision cannot rescue a GO once tokens miss — so run the token A/B first and only build a precision harness if tokens clear ~2×.
- [Phase 2] Add the LSP finding-verification step to the deep/audit pr-review tiers #843 verification can't measure precision symmetrically.
scripts/lib/lsp-verification.shonly emitsfinding_verificationrecords on the LSP-on leg (it needsmcp__lsp__*in the allowlist), so it structurally cannot produce a false-positive delta vs LSP-off. Measuring the precision half needs a neutral judge (same scorer over both legs) — reuse the evalscore_llm_judgepath, not [Phase 2] Add the LSP finding-verification step to the deep/audit pr-review tiers #843. - N≥3 median per PR to beat the run-to-run navigation-strategy variance seen in the Shell controls.
Candidate children (the languages we actually use)
- [LSP re-pilot placeholder] Python (pyright) — broodminder-export / TalkTerm #1125 — Python re-pilot (
pyright) —broodminder-export(96% Py),TalkTerm(34% Py). Highest a-priori odds: real cross-file imports + type resolution. - [LSP re-pilot placeholder] TypeScript / JavaScript (typescript-language-server / tsgo) — broodly / google-app-scripts / markets #1126 — TypeScript / JavaScript re-pilot (
typescript-language-server, ortsgowhen GA) —broodly(TS),google-app-scripts(Apps Script JS), embedded JS inmarkets. The vendor-cited strong case.
Lower-priority / not currently scoped: HTML, Groovy, HCL/Terraform (small footprint; weak LSP-vs-grep case for review navigation).
Refs: discussion #578 · epic #839 · decision #845 · docs/lsp-pilot.md.
Activity
don-petry commentedon Jul 9, 2026
📌 Rebuild pointer — how to restart this without keeping the harness in-tree
The Shell pilot (NO-GO, #845) is being removed from the repo (retire PR — the ~3,800 LOC harness + Shell corpus + docs + the production-integration hooks). Everything below is pinned here so a future per-language re-pilot is a cherry-pick + edit 3 files, not a rebuild.
Stable git ref
The complete, working harness lives at
petry-projects/.github-private@f882266373a1c16ba18760c1ef57753652f9cf7d(origin/main immediately before the retire PR). Built by PRs #848 · #853 · #852 · #861 · #922 · #931 · #952 · #961 · #1034 (stories #840–846). Restore withgit checkout f882266 -- <path>.What to cherry-pick (the self-contained token-A/B harness — language-agnostic)
Per-language changes needed (this is the entire delta)
.github/mcp/lsp.json— one line:bash:bash-language-server,start→python:pyright([LSP re-pilot placeholder] Python (pyright) — broodminder-export / TalkTerm #1125) ortypescript:typescript-language-server([LSP re-pilot placeholder] TypeScript / JavaScript (typescript-language-server / tsgo) — broodly / google-app-scripts / markets #1126).scripts/setup-lsp-pilot.sh— install that server instead of bash-language-server (agent-lsp resolvesmcp__lsp__*generically, so the allowed-tools set is unchanged)..github/workflows/lsp-pilot-run.yml— swap theBASH_LANGUAGE_SERVER_VERSIONenv for the per-language server-version pin.evals/lsp-pilot/holdout/cases.jsonlto real cross-file PRs in that language's repos (the actual research each time).What is NOT needed to test another language (deliberately removed, do NOT resurrect for a test)
The production-integration arm —
scripts/lsp_pilot_emit.sh,scripts/lib/lsp-verification.sh, and the hooks inpr-review.yml(LSP_PILOT_ENABLED gate),review-one-pr.sh(apply_lsp_verification),engine.sh,token-metrics.sh, and thelsp_verificationannotation blocks inprompts/deep-review.md+security-audit.md. The standalone A/B runner does not touch any of it. (It's also the asymmetric-verification path that can't measure a symmetric precision delta — see method note above; use a neutral judge instead.)The Shell go/no-go corpus (for reproducibility — repin per language, don't reuse)
The 20 real
.github-privatePRs the Shell pilot scored, pinned by head SHA:5bcfcb49379a6def6d0ed9f9e76120d25aa215b82baf1d13a1ac99f3c0943d374b3bc08456a68897cab661ce4a49fc7d15adf90cadd261c5ae9a60ce6255bea2a327e1b425b28993a77aa42f0f80382b898eb14aa699132666e7aeb0582dd86290009abebd56f48671a753ba4d772cfbcc915dd92e6b81f4chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845)
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
chore(lsp-pilot): retire the LSP pilot after Shell NO-GO (#839, #845) (…
don-petry commentedon Sep 1, 2026
Closing — placeholder for a re-pilot that is not happening
Maintainer decision 2026-09-01: the LSP re-pilot is not being pursued.
The parent initiative #839 ("Pilot LSP code intelligence in the agent") closed on 2026-07-09 with a NO-GO verdict for Shell. These per-language placeholders were carried forward as
initiative:hold+dev-lead:hands-offspeculative follow-ups in case a re-pilot was scoped for another language. None was, and they have sat untouched for nearly two months.Closing rather than holding indefinitely: a placeholder that nothing is planned against is backlog noise, not a plan. The rationale in #839 stands as the record, and nothing here is lost — if a language-server pilot is ever revisited, it should be re-scoped against the toolchain of that day rather than resumed from a two-month-old stub.