Skip to content

Initiative (placeholder): Per-language LSP re-pilot — follow-up to #839 (Shell = NO-GO) #1124

Closed

Description

@don-petry

Why this exists

Epic #839 piloted LSP (language-server) code intelligence in the pr-review harness for exactly one language/server: Shell via bash-language-server behind agent-lsp. The measured result was a NO-GO (decision: docs/lsp-pilot.md §7, recorded on #845):

Run Model Agg nav-token ratio High-nav stratum
28718240665 sonnet-4-6 1.26× 1.01×
28940391169 opus-4-8 (prod) 1.05× 1.04×

The ≥2× navigation-token target was missed decisively; on shell diffs already cheap to grep, agent-lsp's MCP tool-result payloads cost about as much context as the reads they replace. That result does not generalize — it says nothing about languages with deeper cross-file type resolution, which is exactly where find-references is expected to beat grep.

This is a holding issue (inert: initiative:hold, no initiative:auto) to keep the open door tracked. Promote a child to a real pilot by removing the hold and planning it; do not auto-release.

The reusable harness is already in place (dormant, cost-free)

A per-language re-pilot is a restart, not a rebuild — reuse:

  • scripts/lsp_pilot_run.sh — live A/B runner (LSP-off vs LSP-on on the same PRs)
  • scripts/lsp_pilot_measure.sh / scripts/lsp_pilot_compare.sh — nav-token / tool-call / cold-start extraction + rendering
  • .github/workflows/lsp-pilot-run.yml — on-demand workflow_dispatch driver
  • evals/lsp-pilot/ — the frozen-corpus pattern (repin per language)

Per-language deltas needed each time: (a) pick + version-pin the LSP server, (b) point .github/mcp/lsp.json at it (or a per-lang variant) and update setup-lsp-pilot.sh install, (c) repin evals/lsp-pilot/holdout/cases.jsonl to real cross-file PRs in that language's repos, (d) dispatch the A/B.

Method fixes carried over from the Shell pilot (do these before re-running)

  1. Gate on tokens first. The success metric is a conjunction (≥2× nav-token reduction AND no precision regression). Precision cannot rescue a GO once tokens miss — so run the token A/B first and only build a precision harness if tokens clear ~2×.
  2. [Phase 2] Add the LSP finding-verification step to the deep/audit pr-review tiers #843 verification can't measure precision symmetrically. scripts/lib/lsp-verification.sh only emits finding_verification records on the LSP-on leg (it needs mcp__lsp__* in the allowlist), so it structurally cannot produce a false-positive delta vs LSP-off. Measuring the precision half needs a neutral judge (same scorer over both legs) — reuse the eval score_llm_judge path, not [Phase 2] Add the LSP finding-verification step to the deep/audit pr-review tiers #843.
  3. N≥3 median per PR to beat the run-to-run navigation-strategy variance seen in the Shell controls.

Candidate children (the languages we actually use)

Lower-priority / not currently scoped: HTML, Groovy, HCL/Terraform (small footprint; weak LSP-vs-grep case for review navigation).

Refs: discussion #578 · epic #839 · decision #845 · docs/lsp-pilot.md.

Activity

  1. added
    initiativeEpic / initiative tracking issue
    initiative:holdInitiative driver must not release this issue (human hold)
    dev-lead:hands-offExclude from dev-lead agent automation
    on Jul 8, 2026
  2. don-petry commented on Jul 9, 2026

    @don-petry
    CollaboratorAuthor

    📌 Rebuild pointer — how to restart this without keeping the harness in-tree

    The Shell pilot (NO-GO, #845) is being removed from the repo (retire PR — the ~3,800 LOC harness + Shell corpus + docs + the production-integration hooks). Everything below is pinned here so a future per-language re-pilot is a cherry-pick + edit 3 files, not a rebuild.

    Stable git ref

    The complete, working harness lives at petry-projects/.github-private@f882266373a1c16ba18760c1ef57753652f9cf7d (origin/main immediately before the retire PR). Built by PRs #848 · #853 · #852 · #861 · #922 · #931 · #952 · #961 · #1034 (stories #840–846). Restore with git checkout f882266 -- <path>.

    What to cherry-pick (the self-contained token-A/B harness — language-agnostic)

    scripts/lsp_pilot_run.sh        # the A/B runner (LSP-off vs LSP-on on the same PRs)
    scripts/lsp_pilot_measure.sh    # nav-token / tool-call extraction from the stream-json transcript
    scripts/lsp_pilot_compare.sh    # renders baseline-vs-candidate verdict
    scripts/lsp_pilot_report.sh     # optional richer report
    scripts/setup-lsp-pilot.sh      # installs the LSP server + prints the allowed-tools set
    .github/workflows/lsp-pilot-run.yml   # on-demand workflow_dispatch driver
    .github/mcp/lsp.json                  # the agent-lsp MCP server config
    evals/lsp-pilot/README.md + dir scaffold  # frozen-corpus conventions + validator
    tests/lsp_pilot_compare.bats, tests/lsp_pilot_report.bats,
    tests/dev-lead/unit/test_lsp_pilot_run.bats, .../test_lsp_pilot_measure.bats
    

    Per-language changes needed (this is the entire delta)

    1. .github/mcp/lsp.json — one line: bash:bash-language-server,start → python:pyright ([LSP re-pilot placeholder] Python (pyright) — broodminder-export / TalkTerm #1125) or typescript:typescript-language-server ([LSP re-pilot placeholder] TypeScript / JavaScript (typescript-language-server / tsgo) — broodly / google-app-scripts / markets #1126).
    2. scripts/setup-lsp-pilot.sh — install that server instead of bash-language-server (agent-lsp resolves mcp__lsp__* generically, so the allowed-tools set is unchanged).
    3. .github/workflows/lsp-pilot-run.yml — swap the BASH_LANGUAGE_SERVER_VERSION env for the per-language server-version pin.
    4. Repin evals/lsp-pilot/holdout/cases.jsonl to real cross-file PRs in that language's repos (the actual research each time).

    What is NOT needed to test another language (deliberately removed, do NOT resurrect for a test)

    The production-integration arm — scripts/lsp_pilot_emit.sh, scripts/lib/lsp-verification.sh, and the hooks in pr-review.yml (LSP_PILOT_ENABLED gate), review-one-pr.sh (apply_lsp_verification), engine.sh, token-metrics.sh, and the lsp_verification annotation blocks in prompts/deep-review.md + security-audit.md. The standalone A/B runner does not touch any of it. (It's also the asymmetric-verification path that can't measure a symmetric precision delta — see method note above; use a neutral judge instead.)

    The Shell go/no-go corpus (for reproducibility — repin per language, don't reuse)

    The 20 real .github-private PRs the Shell pilot scored, pinned by head SHA:

    PR head SHA stratum
    #528 5bcfcb49379a high-nav
    #928 6def6d0ed9f9 high-nav
    #453 e76120d25aa2 high-nav
    #379 15b82baf1d13 high-nav
    #468 a1ac99f3c094 high-nav
    #649 3d374b3bc084 high-nav
    #720 56a68897cab6 high-nav
    #784 61ce4a49fc7d high-nav
    #592 15adf90cadd2 high-nav
    #658 61c5ae9a60ce high-nav
    #609 6255bea2a327 high-nav
    #1012 e1b425b28993 high-nav
    #460 a77aa42f0f80 medium-nav
    #1011 382b898eb14a medium-nav
    #494 a699132666e7 medium-nav
    #196 aeb0582dd862 diagnostics
    #1065 90009abebd56 control
    #953 f48671a753ba control
    #867 4d772cfbcc91 control
    #847 5dd92e6b81f4 control
  3. added 18 commits that reference this issue on Aug 2, 2026
  4. don-petry commented on Sep 1, 2026

    @don-petry
    CollaboratorAuthor

    Closing — placeholder for a re-pilot that is not happening

    Maintainer decision 2026-09-01: the LSP re-pilot is not being pursued.

    The parent initiative #839 ("Pilot LSP code intelligence in the agent") closed on 2026-07-09 with a NO-GO verdict for Shell. These per-language placeholders were carried forward as initiative:hold + dev-lead:hands-off speculative follow-ups in case a re-pilot was scoped for another language. None was, and they have sat untouched for nearly two months.

    Closing rather than holding indefinitely: a placeholder that nothing is planned against is backlog noise, not a plan. The rationale in #839 stands as the record, and nothing here is lost — if a language-server pilot is ever revisited, it should be re-scoped against the toolchain of that day rather than resumed from a two-month-old stub.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-lead:hands-offExclude from dev-lead agent automationinitiativeEpic / initiative tracking issueinitiative:holdInitiative driver must not release this issue (human hold)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions