Benchmark version-correct edit replay through the ry server process

[sims1253]

Motivation

docs/performance.md measures incremental core operations (Criterion, in-process Project::check_incremental) and VS Code activation-to-first-diagnostic latency, and explicitly states that the incremental core benchmarks "model part of the LSP workload but do not exercise an editor or the JSON-RPC transport". There is currently no measurement of edit-to-diagnostics latency through a real ry server process: the functional tests that drive a real server with didOpen/didChange assert correctness only, and the extension performance test (editors/code/test/performance.cjs) measures activation and the first diagnostic, not the edit loop. This proposal adds that measurement while preserving the existing correctness contract. It is a measurement extension, not a newly reproduced LSP defect.

Foundation to extend

• crates/ry-testkit already provides the needed transport: JsonRpcProcess::spawn drives a real stdio server child (already used by crates/ry-cli/tests/lsp_testkit.rs and crates/ry-lsp/tests/protocol.rs), alongside ObservedDiagnostic normalization and the independent CliProcess comparison.
• Session state-machine tests, protocol-contract tests, and incremental-index-budget tests already pin server behavior.
• scripts/performance/collect.py already merges measurement sources into the CI performance report.

Do not build a second correctness oracle and do not describe lifecycle testing as absent.

Proposed first increment

Reuse the existing protocol client/normalization abstractions with a child-process transport for the release server. Add a small versioned replay corpus and a measurement adapter feeding the existing performance report.

Start with three scenarios in a fixed multi-file project: a local edit, a cross-file caller-visible signature change, and an unrelated-file edit. Measure initial startup separately from warm edits. Expand later to rapid edit bursts, syntax break/repair, file add/remove/rename, configuration/import changes, and unsaved buffers, reusing existing lifecycle fixtures.

Define precisely when timing begins and ends: for example, from sending a complete didChange frame to receiving the expected latest-version diagnostics for the scenario. Include server queuing/debounce work; exclude fixture preparation and the independent correctness oracle. Track publication scope and version/snapshot identity so an old, duplicate, or empty lifecycle notification cannot count as fast completion.

At replay checkpoints, compare the final observable state against fresh analysis using equivalent source bytes, package/config context, and publication rules. Materialize unsaved buffers in an isolated comparison tree where needed. Do not assume arbitrary file reordering preserves R shadowing semantics.

Acceptance criteria

• The first three scenarios run against an actual release server process, reuse existing normalization, and produce explicit readiness/completion evidence instead of sleeping for a guessed interval.
• Latest-version results match independent fresh analysis under the documented publication contract; missing diagnostics, stale publications, and stalled completion fail correctness.
• A fixture that ends cleanly proves the diagnostic clear belongs to the final analysis, not merely a document-close notification.
• Record replay/workload hashes, binary and toolchain identity, server/thread settings, sample counts, and raw durations. Report median and tail latency from enough samples with an explicit calculation, not a percentile from a handful of startup runs.
• Warm repetitions reset to a defined initial snapshot; setup/reset effects and potentially warm OS caches are separated from actual cold-start claims.
• Deterministic correctness checks stay enforced; hosted-runner timing deltas start advisory, and suspected regressions require repeated same-machine comparison, as docs/performance.md already requires for the existing metrics.

Boundaries

Do not promise editor-rendering latency from a server-protocol benchmark. Keep the existing real-extension startup metric. No new public dashboard, universal latency threshold, whole-LSP rewrite, or mandatory phase-tracing dependency is required.

[pullfrog]

  ^{｜ Build this ➔ ｜ Make a plan ➔}