Repository navigation
mcpx testing: deterministic execution as a long-term aspiration (retrofit vs Rust rewrite) #1963
Description
Relocated from dezren39/darwin-nix#126 on 2026-10-28 (placement audit): mcpx's single source lives in developing-today/code (pkgs/mcpx); re-home tracked at #309. Original body below; issue references inside it are historical (dezren39/nix numbering).
Parent: #93
Summary
The simple, explicitly requested issue: consider deterministic execution as a long-term design aspiration, two ways -- retrofit mcpx (inject clock, randomness, filesystem, process spawn and network behind interfaces) or a Rust rewrite, which the notes call "not a plan at all". Not a plan to build either; a short record of what each would take, what blocks it, and the cheap steps that pay off regardless.
Original notes
- think a hack job of antithesis/tigerbeetle. this is the primary testing method is throw insane amount of variations. if something is found outside this system and a test that could have had missed it, make a different generator that uses the same seed and makes an infinite amount of things matching the offensive shape. - we aren't shooting for deterministic execution to start, maybe we should but no. consider it a longterm design aspiration to retrofit or for rewrite in rust (not a plan at all, but could see it). separate simple issue for considering both those things.
Current state
- Execution is nondeterministic by design today: Go goroutine scheduling; 78
time.Now()calls in non-testinternal/code (05c78b2); ids fromcrypto/randinpool,tasks,elicit,artifacts,mcpserver,logging/trace; 11exec.Commandsites; unix sockets (Deno/Bun) and TCP (Node) back to the daemon. The daemon's idle exit runs on wall-clock time. It is a setting,autostart.idleExit(default4h,internal/defaults/defaults.json:138; a setting since chore(deps): bump rustix from 0.37.19 to 0.37.25 in /sources/pavex/examples/realworld #56, not hardcoded), which Phase 5: tag event bodies must travel with the event #182 sets to 60 s for e2e. - Some seams already exist:
pool.ReapIdle(now time.Time)takes the clock as a parameter (internal/pool/pool.go:697); e2e isolates state/cache/config/registry/idle-exit by env (internal/e2e/e2e_test.go:52-68). - The seeded-input design (Update Rust crate sqlx to 0.8.0 [SECURITY] #121, Update Rust crate tracing-subscriber to v0.3.20 [SECURITY] - autoclosed #124) reproduces inputs and test order, not interleavings. This issue is the other half.
- Go 1.25+ ships
testing/synctest: inside a "bubble",timeuses a fake clock that advances only when every goroutine in the bubble is durably blocked. Its own docs say to avoid the real network and external processes inside a bubble (go doc testing/synctest).
What blocks determinism in mcpx
| Source | Retrofit difficulty | Why |
|---|---|---|
| Wall clock / timers | Low-medium | inject a Clock; or run in-process components under synctest |
crypto/rand ids |
Low | inject an id source; seeded in tests |
| Filesystem | Medium | an fs seam for state/cache/artifacts; t.TempDir already isolates |
| Goroutine scheduling | High | Go has no deterministic scheduler; synctest makes time deterministic, not interleavings |
| Unix sockets / TCP / HTTP | High | real kernel I/O; in-process net.Pipe transports only for components that can run in one process |
| Runtime subprocesses (deno, bun, node) | Not feasible | separate processes with their own event loops, JITs and clocks; the core of mcpx exec |
| Real MCP servers (#97 real-software subset) | Not feasible | third-party processes |
Approaches
A -- Rust rewrite for determinism (least recommended; "not a plan at all", agreed)
What it would buy: Rust async simulation frameworks (e.g. madsim, turmoil for tokio) simulate time and network in one process, TigerBeetle/FoundationDB-style. What it would not buy: determinism for the deno/bun/node subprocess that runs every script, or for real MCP servers -- the part of mcpx most likely to misbehave. A full rewrite of a working Go codebase for a property it still would not achieve end to end is the wrong trade. Worth revisiting only if a rewrite happens for other reasons.
B -- full retrofit: every nondeterminism source behind an interface, a simulated daemon
Clock, ids, fs, spawn and network all injected; a test harness runs the daemon, pool, events and a fake MCP server in one process on simulated transports, seeded from the run seed, VOPR-style. Achievable for the daemon's internal logic (pool admission/reaping, task lifecycle, elicitation broker, event store #6, state/cache #78), and it would find real concurrency bugs. Costs: interfaces threaded through most packages, a simulated transport layer, and it still stops at the runtime subprocess boundary. L-XL effort for partial coverage.
C -- incremental seams where they pay today; Antithesis-style tooling as the end-to-end answer (recommended)
- Add seams only when a test needs them: a
Clockfor idle-exit (already configurable throughautostart.idleExit, but a test of it still waits in real time), seeded id sources,ReapIdle-stylenowparameters. - Use
testing/synctestfor in-process concurrent components (pool, tasks, elicit, events): deterministic time and "wait until everything is blocked" instead oftime.Sleeppolling -- which also removes the flake-prone sleeps found in chore(deps): update dependency happy-dom to v20 [security] - autoclosed #125. - Derive every injected randomness source from the run seed (Update Rust crate sqlx to 0.8.0 [SECURITY] #121) so partial determinism composes with seeded inputs.
- For true end-to-end determinism including the runtime subprocesses, the realistic path is a deterministic hypervisor (Antithesis runs unmodified containers), not a rewrite. That needs a container image. One now exists: Phase 5: tag event bodies must travel with the event #182 added
pkgs/mcpx/Containerfile(debian bookworm-slim runtime, bun, 413 MB;docs/container.md). It has no CI job and no tests yet (Update Rust crate rand to v0.9.3 [SECURITY] #134). This is a prerequisite that is now met, not a plan. - Why C: each step is small, useful on its own (fewer flakes, faster tests), and moves toward B without committing to it.
Open questions
- Is idle-exit the first
Clockseam? Phase 5: tag event bodies must travel with the event #182 fixed chore(deps): bump build/buck/preludes/buck2-prelude fromd956689toe7457f6#52 without one (cleanup plus a 60 s idle exit), so a bug no longer forces it. It is still the seam whose tests otherwise wait in real time. - Which in-process components would
synctestcover without refactoring (does anything inside them touch real sockets or processes)? - The container image exists (Phase 5: tag event bodies must travel with the event #182). Is running it under a deterministic hypervisor worth one experiment once Update Rust crate rand to v0.9.3 [SECURITY] #134 gives it a CI job?
Ease · Impact · Related
Ease S (this issue: record + first seam list) · C incremental: S per seam · B: L-XL · A: XL, not planned · Impact low now, med long-term (flake reduction is the near-term win)
Depends on / related: #121, #124, #125; #52 (closed, fixed in #182), #78 (state/cache), #6 (event store), #97 and #134 (the container image and its CI job, real-software subset), #99 (flake policy).
Security/auth deferred; see tracker.
Acceptance criteria
- A short record exists (on this issue or in the docs tree and linked here) comparing the two routes to deterministic execution — retrofit (clock, randomness, filesystem, process spawn, network behind interfaces) vs a Rust rewrite — with what each would take and what blocks it.
- The cheap steps that pay off regardless of route are listed, and each is either filed as its own issue or explicitly dropped with a reason.
- No implementation is started under this issue (it is explicitly not a plan to build either route).
- Evidence (the record's location) is recorded on this issue before it is closed.
Activity
developing-today commentedon Oct 28, 2026
CODE-888
dezren39 commentedon Oct 28, 2026
History carried over from dezren39/darwin-nix#126 (verbatim, oldest first):
@dezren39 · 2026-09-30T07:20:55Z
Integration pass. Placed in the DAG rooted at #181.
@dezren39 · 2026-09-30T12:03:29Z
Issues pass (2026-09-30, at 05c78b2).
d956689toe7457f6#52)". It is a setting,autostart.idleExit(default4h,defaults.json:138), and Phase 5: tag event bodies must travel with the event #182 sets it to 60 s for e2e. chore(deps): bump build/buck/preludes/buck2-prelude fromd956689toe7457f6#52 is closed. I edited the body because the old sentence would send someone to build aClockseam to fix a bug that no longer exists.pkgs/mcpx/Containerfile(bookworm-slim + bun, 413 MB), which has no CI job yet (Update Rust crate rand to v0.9.3 [SECURITY] #134). The two open questions that depended on chore(deps): bump build/buck/preludes/buck2-prelude fromd956689toe7457f6#52 and on the missing container are reworded.time.Now()sites. Re-citede2e_test.go:52-68.dezren39 commentedon Oct 29, 2026
Intake review
🤖 Agent note · state: filed
1 remaining item
dezren39 commentedon Oct 29, 2026
Triage verdict — task 2864 (2026-10-29)
pkgs/mcpxmain (GitHub API, 2026-10-29):pkgs/mcpxpresent;go.moddeclares go 1.27.2, sotesting/synctest(stable since Go 1.25) is available;pkgs/mcpx/Containerfilepresent; code search:synctest= 0 hits (the aspiration is not yet implemented anywhere),ReapIdleseam = 6 hits,time.Nowinpkgs/mcpx= 98 hits — same order as the body's count of 78 non-testinternal/sites at the older commit it cites. The body's A/B/C record and blockers table still match the code as it stands.go test ./...via CI, named in any filed step issue's acceptance criteria.🤖 Agent note · state: triaged
dezren39 commentedon Oct 30, 2026
Picked up — queue task 3259
Close-out of this issue per the triage verdict (task 2864): disposition each approach-C cheap step (file as its own issue or drop with a reason), record the evidence here, and close. No determinism implementation under this task.
Revalidation (2026-10-30): issue OPEN, no hard-stop labels (
held/user/owned/do-not-merge/do-not-auto-editabsent). Dedupe searches run fresh today for synctest / antithesis / hypervisor / clock seam / seed-derived / deterministic — no existing issue covers any approach-C step; #1949 (run-seed format/delivery) and #1956 (generator engine) cover seeded inputs, not execution determinism or injectable id sources.🤖 Agent note · state: picked-up
dezren39 commentedon Oct 30, 2026
Close-out — evidence record and approach-C dispositions (queue task 3259, 2026-10-30)
Record location: the retrofit-vs-Rust-rewrite record this issue's acceptance criteria require is this issue's own body — the blockers table, approaches A (Rust rewrite, not a plan), B (full retrofit), and C (incremental seams, recommended), with Ease/Impact and open questions. Triage task 2864 verified the body still matches current
pkgs/mcpxmain (2026-10-29), and this close-out re-ran the dedupe searches fresh today (synctest / antithesis / hypervisor / clock seam / seed-derived / deterministic: only #1963 itself, plus the distinct seed-programme issues #1949/#1956).Approach-C cheap steps — each filed or dropped:
go test ./...via CI).testing/synctestpilot for one in-process component (pool/tasks/elicit/events) → FILED as mcpx testing: testing/synctest pilot for one in-process component (pool, tasks, elicit, or events) #2639.pool,tasks,elicit,artifacts,mcpserver,logging/trace) that a seed would drive. Linked as depends-on, not duplicated.pkgs/mcpxcontainer image → FILED as mcpx testing: one deterministic-hypervisor (Antithesis) experiment on the pkgs/mcpx container image #2641, with its prerequisites named inside it (container CI job mcpx testing: environments -- nix flake, container, local, remote -- as CI matrix entries #1983; Antithesis is a commercial service, so account access/cost is Drew's decision before any run). Nothing about that decision blocks this close-out — the experiment issue carries its own go/no-go gate. Dropped: nothing. No step needed dropping; each was small, clearly scoped, and uncovered elsewhere.No implementation was started under this issue, per its acceptance criteria — all determinism work lives in the four step issues above.
Closing this issue as completed: record exists (body), every cheap step is filed, evidence recorded here.
🤖 Agent note · state: done