diff --git a/doc/fleet/2026-10-19-reliability-review.md b/doc/fleet/2026-10-19-reliability-review.md new file mode 100644 index 000000000..0047e0a35 --- /dev/null +++ b/doc/fleet/2026-10-19-reliability-review.md @@ -0,0 +1,265 @@ +# Reliability review, 2026-10-19 + +A dated record of one long working session: what broke, why, what was changed, +what was found and left open, and what turned out to be wrong along the way. +**It describes the state on 2026-10-19 at commit `e509884`, before the next +`just switch` applied it (generation 304 was active).** It is a snapshot, not a +status page. The durable mechanisms are in [`services.md`](services.md) and +[`mcp.md`](mcp.md). Each open problem below became an issue on 2026-10-22, +tracked in dezren39/nix#672. + +Facts marked *(audit)* come from three read-only investigations run that day. +They covered OpenChamber, OpenCode and mcpx reliability; the MCP servers and their +credentials; and how developing-today/code replaces services under systemd. +Everything else was checked directly while writing this page. + +## Changes made + +| Commit | What | +|---|---| +| `365ac99` | The desktop app bundles the real OpenCode, not the deploy link. Its dangling link made nix-darwin's rsync into `/Applications/Nix Apps` fail ("symlink has no referent", exit 23) | +| developing-today/openchamber `94b7e56f9`, pinned here in `3402586` | The desktop app's bundle links `packages/` next to `node_modules/`, so `@openchamber/web` resolves. The app crashed at launch with `ERR_MODULE_NOT_FOUND` | +| `a0c0cdf`, `8f05b36`, `c90cd1b`, `e2d8f39` | Temporary darwin skips for mcpx tests that only failed in the nix sandbox, each removed once fixed upstream (#428, #434, #435, #445) | +| `50d22f1` | `simple-rebuild.sh`: fail the build phase when `nix build` exits 0 but produces no `./result` | +| `5a856f6` | `openchamber.nix`: link the start script, not the deploy key, and fail activation if the link is not executable | +| `f940ba9` | `just switch` ends with a health check of the long-running agents; the nix-config skill documents activation testing and launchd debugging | +| `d65df61` | Removed lootbox, the second MCP gateway. `mcpx.config.nix` became the only server list; `mcp-remote` and `chrome-devtools-mcp` are now built from a lockfile | +| `e220322` | Documented the stray `%` when reading the sops UI password | +| `ecca0a1` | mcpx: config as a real file at a fixed path, `daemon.autostart = false`, an agent wrapper that clears strays, a CLI-based health check, `ExitTimeOut` on mcpx and OpenChamber | +| `e509884` | Docs: the live OpenCode config is not the repo's `opencode-global.jsonc` | + +Upstream issues filed: developing-today/code#368 (mcpx socket name follows the +symlink-resolved config path) and developing-today/opencode#134 (OpenCode's mcpx +plugin starts its own daemon regardless of `daemon.autostart`). + +Outside the repo, the same day: +- lootbox was unloaded and its binary, UI, logs and leftover folders deleted. + `~/.lootbox` (14 browser-automation scripts written by hand) was moved to the + Trash. +- The `lootbox.md` line was removed from the live OpenCode config. +- Six Renovate PRs that only touched lootbox's UI were closed. +- `nix store gc` freed 22.9 GiB (10,438 store paths). + +## Incidents and their causes + +### mcpx restarted 841 times (launchd: 842 runs) + +**Symptom.** From 11:47 to 14:09, launchd's mcpx daemon exited 1 about every +10 s: "a daemon is already listening on ~/.local/state/mcpx/daemon-541cdd605dd1.sock". + +**Cause.** mcpx names its socket from the symlink-resolved path of its config, and +the config was a home-manager link into the nix store. + +- Socket name per generation *(audit)*: generations 290-295 gave `3489ca4d2ea0`, + 296 gave `539ba2d32a06`, 297-304 gave `541cdd605dd1`. +- nix-darwin restarts launchd agents before home-manager re-links files (activate + script lines ~862 and ~1100). + +Timeline, from `/tmp/mcpx.err.log` and `~/.local/state/mcpx/logs/`: + +| Time | Event | +|---|---| +| 2026-10-18 20:58 | Generation 298's activation restarts the agent. It reads the still-old link and listens on `daemon-539ba2d32a06.sock` | +| 04:48:10 | A command computes the current name, finds no daemon and autostarts a detached one (`daemon --detached --idle-exit 4h`, mcpx build `hnqq7r7w…`) on `daemon-541cdd605dd1.sock`. Being used, it never idles out | +| 11:47 | Generation 302 brings a new mcpx (`p8wcs9ml…`); launchd's daemon restarts on the current name and finds it held | +| 14:09 | `mcpx stop` ends the stray and launchd's daemon (pid 73031) takes the socket | + +Meanwhile, clients ran on the stray's older build. + +**Fix** (`ecca0a1`): +- the config is a real file at a fixed path; +- commands no longer start daemons; +- the agent's wrapper stops anything on its socket before starting; +- the health check fails when the CLI cannot reach the daemon. + +Each part was reproduced and tested with real mcpx binaries on scratch state +directories. A symlinked config re-pointed to another file left the CLI unable to +reach the running daemon (exit 1). A real file rewritten in place kept the socket +and was hot-reloaded. + +### OpenChamber unreachable + +**Symptom.** The desktop app showed the local instance as Unreachable, and every +request to `127.0.0.1:3000` was refused. + +**Cause.** `493be7d` added the UI password's hash to the deploy key +(`