Skip to content

serve handover: verify under a real systemd user session and close the teardown race #81

New issue
New issue
Closed
#777
Closed
serve handover: verify under a real systemd user session and close the teardown race#81
#777
Labels
effort/mMedium: expected under 1 day of workkind/testTest coverage, test infrastructure, or test-only fixesmeta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedpriority/normalOrdinary priority (the default)queue/queuedHas a queued task in the Supabase agent queuesource/humanFiled directly by a humanstatus:done

Description

@dezren39
dezren39
opened on Oct 18, 2026

Problem

TODO.md ("Still open" under the supervisor and handoff section) lists two gaps in the opencode serve handover that nothing tracks as an issue.

  1. Systemd is untested. Type=notify needs NotifyAccess=all, because when the wrapper starts the server the notifying process is the server child, not the unit's main pid. The handover tests stand in for systemd with a socket the test process owns and a fake notify socket. Nobody has run it under a real systemd user session.
  2. A teardown race remains. A connection the predecessor accepted in the last instant before it closes can get a 503 (service_stopping) or a reset when its request arrives after teardown begins. In packages/cli/test/serve-handover.test.ts, about a thousand requests across handovers showed none, and the control run (no supervisor socket) showed one per run. So the race is rare, but it is not shown to be gone.

What to do

  1. Write a unit file with Type=notify and NotifyAccess=all, run two generations through a real handover on a systemd host, and record the result here.
  2. Decide whether the teardown race needs a fix. If it does, the predecessor should keep answering what it has accepted until its drain ends, and the test should run enough requests to show a difference.

Acceptance criteria

  • A real systemd handover is run and its output is attached.
  • The teardown race is either fixed with a test that fails before the fix, or the measured rate is written in TODO.md with the reason it is accepted.

Activity

  1. dezren39
    added
    kind/testTest coverage, test infrastructure, or test-only fixes
    source/humanFiled directly by a human
    priority/normalOrdinary priority (the default)
    effort/mMedium: expected under 1 day of work
    status/needs-triageNot yet triaged; needs a first look
    meta/enrichedIntake review done: acceptance criteria, links, and normalized labels added
    on Oct 26, 2026

  2. dezren39 commented on Oct 26, 2026

    @dezren39
    dezren39
    on Oct 26, 2026
    Author

    Intake review

    🤖 Agent note · filed

    • Agent: fleet monitor
    • Session / ID: repo-fleet-monitor run 2026-10-26 18:19 CDT
    • Found via: monitor reconciliation
    • Task: Intake enrichment (§5) for this issue
    • Expectation: m — done = labels normalized, acceptance criteria confirmed present, still-live check recorded

    Still live (verified this run): no PR in this repo runs a handover under a real systemd user session, and no later work closes the teardown race — a search for systemd/handover PRs returns nothing beyond the original handover work. Both gaps from TODO.md remain untracked elsewhere.

    Related: developing-today-forks/openchamber#81 (the supervisor proposal that consumes this handover contract — companion item, filed the same minute; its systemd check is this issue), packages/cli/test/serve-handover.test.ts (the fake-systemd test named in the body).

    Notes / risks: the body already carries a complete ## Acceptance criteria section — no body edit needed. The systemd half needs a host with a real systemd user session; it cannot be verified from a non-systemd sandbox, so the run should be planned for such a host and its output attached here, as the first acceptance item requires. The teardown race is rare (~1 per control run), so a fix decision needs the high-request-count test the body describes, not a single green run.

    Suggested approach: write the Type=notify + NotifyAccess=all unit, drive two generations through a real handover on a systemd host, attach the journal/output here; then run the teardown probe at high request volume with and without the supervisor socket to decide the race fix on measurements.

  3. dezren39 commented on Oct 28, 2026

    @dezren39
    dezren39
    on Oct 28, 2026
    Author

    Triage verdict: CONFIRMED — both gaps still open at current main

    Verified against origin/main at cd21eab1dec5 (2026-10-28), not the issue text alone:

    1. Systemd is still untested. TODO.md ("Still open") still reads verbatim: Systemd units: Type=notify needs NotifyAccess=all, because the notifying process is the server child, not the unit's main pid, when the wrapper starts it. Not verified under a real systemd user session. The tests stand in for systemd with a socket the test process owns and a fake notify socket. The stand-in is packages/cli/test/fixture/systemd-socket.ts, whose header says: "The systemd roles a test plays without systemd."
    2. The teardown race is still undecided. The same "Still open" section still lists the residual race: a connection the predecessor accepted in the last instant before it closes can get a 503 (service_stopping) or a reset; the handover runs showed none, the control showed one per run. No measured-rate/acceptance note and no fix have landed.
    3. Tracking agrees: ROADMAP.md:43 — "Verify the handover under real systemd | Now | proposed | Type=notify with NotifyAccess=all is only tested with stand-ins. Also decide on the rare teardown race. | serve handover: verify under a real systemd user session and close the teardown race #81".
    4. A real host exists for the verification: laptop amd probed this run via ~/tools/laptop-ssh.sh — systemctl --user is-system-running returns degraded (a real user session; failed units should be identified before blaming the handover) and loginctl shows Linger=yes.

    Route

    Scoped follow-up packet enqueued to repo-health-opencode-pilot as task 2687 (kind small_fix, dedupe repo-health:v1:fix:developing-today/opencode:issue:81): write a Type=notify + NotifyAccess=all user unit, run two generations through a real handover on the laptop's systemd user session, attach the output here, then either fix the teardown race with a fails-before test or record the measured rate in TODO.md with the acceptance reason — updating TODO.md "Still open" and ROADMAP.md:43 to match.

    Labels synced: status/needs-triage removed; queue/queued set for task 2687.

  4. dezren39
    added
    queue/queuedHas a queued task in the Supabase agent queue
    and removed
    status/needs-triageNot yet triaged; needs a first look
    on Oct 28, 2026

  5. dezren39 commented on Oct 28, 2026

    @dezren39
    dezren39
    on Oct 28, 2026
    Author

    🤖 Dispatcher binding · openchamber-dispatcher run 2026-10-28 16:00 CDT

  6. developing-today
    added
    status:in-review
    on Oct 28, 2026

  7. dezren39 commented on Oct 28, 2026

    @dezren39
    dezren39
    on Oct 28, 2026
    Author

    Checkpoint: systemd verification and teardown-race fix

    Both acceptance criteria are met. Work is in #777 (not merged).

    1. Real systemd user session

    Host: systemd 261 with Linger=yes, driven with systemctl --user. The unit is a verification unit (the repo does not ship one). A socket unit owns the listener; a Type=notify service runs serve through the launcher, so the process that sends READY=1 is the server child, not the unit's main pid.

    $ systemctl --user show opencode-handover-test.socket -p Listen -p ActiveState -p SubState
Listen=127.0.0.1:4319 (Stream)
ActiveState=active
SubState=listening

$ systemctl --user show opencode-handover-test.service -p Type -p NotifyAccess -p MainPID -p ActiveState -p SubState
Type=notify
NotifyAccess=all
MainPID=2065052
ActiveState=active
SubState=running

$ ps -o pid,ppid,cmd -p 2065052
2065052 1520 /run/current-system/sw/bin/node .../packages/cli/bin/opencode.cjs serve --hostname 127.0.0.1 --port 4319
$ ps -o pid,ppid,cmd --ppid 2065052
2065076 2065052 /run/current-system/sw/bin/opencode serve --hostname 127.0.0.1 --port 4319

$ ls -l /proc/2065052/fd/3 /proc/2065076/fd/3
... /proc/2065052/fd/3 -> socket:[22002968]
... /proc/2065076/fd/3 -> socket:[22002968]   # same listener in the wrapper and the server child

$ journalctl --user -u opencode-handover-test.service -o cat
Starting opencode serve handover verification service ...
server listening on http://127.0.0.1:4319
opencode: ready
Started opencode serve handover verification service ...

    Started appears only because systemd accepted READY=1 from the server child; that is what NotifyAccess=all permits.

    Two generations through systemctl --user restart (three generations total), with a client opening a fresh connection every 10 ms:

    --- generation 1 (current) ---
MainPID=2065052
2065076 2065052 /run/current-system/sw/bin/opencode serve --hostname 127.0.0.1 --port 4319
--- restart 1 -> generation 2 ---
MainPID=2075996
2076013 2075996 /run/current-system/sw/bin/opencode serve --hostname 127.0.0.1 --port 4319
--- restart 2 -> generation 3 ---
MainPID=2089489
2089506 2089489 /run/current-system/sw/bin/opencode serve --hostname 127.0.0.1 --port 4319
LOAD {"requests":6572,"ok":6572,"refused":0,"refusedSample":[],"failed":{}}

    So 6572 requests, 0 refused, 0 failed across the handovers. The socket unit kept the listener while the predecessor stopped and the successor started, so nothing was refused.

    Observation (pre-existing, out of scope): the stop path exits 130 (Effect's interruption-only teardown code), so systemd logs Main process exited, code=exited, status=130 and Failed with result 'exit-code' before starting the next generation. systemctl restart still succeeds and the next generation serves. A production unit should add SuccessExitStatus=130 (or Restart=always) if it treats a stop as failure. I have not changed this.

    2. Teardown race: fixed with a fails-before test

    Measured on this host before the fix, same real systemd restarts but a 5 ms loop over ten handovers:

    • 4 ECONNRESET in 1851 requests (0.22%) under systemctl --user restart.
    • The handover order alone (successor ready before SIGTERM), 8 handovers: 1 in 7376.

    Root cause: drain waited only on requests whose headers had been parsed (inFlight), and the Bun inherited-socket adapter called closeIdleConnections() the moment it closed. A connection accepted just before the drain was reset before its request arrived.

    Fix in packages/server/src/process.ts and packages/server/src/inherited-socket.ts: track sockets the listener accepts; after the existing session/request wait, keep serving accepted connections for a bounded 500 ms linger, then finish a request that arrived during it. The adapter stops accepting without dropping idle connections. A second SIGTERM still cuts the wait.

    After the fix, the same real systemd restarts, running the branch source on port 4320:

    handover 1: gen-main 2287224 -> 2288921 state=active
...
handover 10: gen-main 2297670 -> 2298339 state=active
LOAD {"requests":2065,"ok":2065,"refused":0,"refusedSample":[],"failed":{}}

    0 resets in 2065 requests.

    Fails-before test: packages/cli/test/serve-handover.test.ts → "a drain answers a request on a connection the server accepted before it began". Without the fix it fails (no response within 5s); with it, HTTP 200 on both runtimes. Full file: 9 pass, 0 fail. Root bun run typecheck: 36/36.

    3. Docs

    TODO.md "Still open" no longer lists the systemd or teardown-race gaps; it records the verified setup and the measured rates. ROADMAP.md drops the shipped row (the roadmap is forward-looking and GitHub is the record), and the area focus moves to the remaining session-state item.

    PR: #777

    Unit files used 
    # ~/.config/systemd/user/opencode-handover-test.socket
[Unit]
Description=opencode serve handover verification socket (issue developing-today/opencode#81)

[Socket]
ListenStream=127.0.0.1:4319

[Install]
WantedBy=sockets.target
    # ~/.config/systemd/user/opencode-handover-test.service
[Unit]
Description=opencode serve handover verification service (issue developing-today/opencode#81)
Requires=opencode-handover-test.socket
After=opencode-handover-test.socket

[Service]
Type=notify
NotifyAccess=all
ExecStart=/run/current-system/sw/bin/node /home/user/git/fleet/2687-81-systemd-handover/packages/cli/bin/opencode.cjs serve --hostname 127.0.0.1 --port 4319
Environment=OPENCODE_BIN_PATH=/run/current-system/sw/bin/opencode
Environment=OPENCODE_SERVER_PASSWORD=handover-test-password
Environment=OPENCODE_SHUTDOWN_GRACE_MS=30000
Environment=OPENCODE_DISABLE_MODELS_FETCH=1
Environment=OPENCODE_DISABLE_EMBEDDING=1
Environment=OPENCODE_DISABLE_PROJECT_CONFIG=1
Environment=OPENCODE_MODELS_PATH=/tmp/opencode/handover/models.json
Environment=HOME=/tmp/opencode/handover/home
Environment=XDG_DATA_HOME=/tmp/opencode/handover/home/.local/share
Environment=XDG_CACHE_HOME=/tmp/opencode/handover/home/.cache
Environment=XDG_CONFIG_HOME=/tmp/opencode/handover/home/.config
Environment=XDG_STATE_HOME=/tmp/opencode/handover/home/.local/state
Environment=OPENCODE_TMPDIR=/tmp/opencode/handover/tmp
TimeoutStopSec=90

[Install]
WantedBy=default.target

    The unfixed run used the installed Nix build (2.0.30+56590c5, same serve sources) through the launcher, which is the child-notifies path NotifyAccess=all exists for. The fixed run used the branch source under bun on port 4320 with the same socket-activation shape.

  8. dezren39 commented on Oct 28, 2026

    @dezren39
    dezren39
    on Oct 28, 2026
    Author

    🤖 Agent note · bound

  9. dezren39 commented on Oct 28, 2026

    @dezren39
    dezren39
    on Oct 28, 2026
    Author

    Outcome

    Both acceptance criteria are met; the change is in #777.

    • Real systemd user session: two generations through systemctl --user restart with a socket unit and a Type=notify service using NotifyAccess=all (the server child sends READY=1). 0 refused, 0 failed requests; transcript above.
    • Teardown race: fixed, with a fails-before test — packages/cli/test/serve-handover.test.ts → "a drain answers a request on a connection the server accepted before it began" (fails without the change, passes with it, both runtimes). Full file: 9 pass / 0 fail; root bun run typecheck: 36/36.
    • TODO.md "Still open" and ROADMAP.md updated.

    CI note: the workflows for head 4fc206a are still queued on a runner ~40 minutes after the push (unit (linux), typecheck, test, check, nix-eval, build), so the test evidence so far is the local run on the laptop (the task allows laptop or CI). The PR is open and will re-run when a runner is free.

    Closing as completed; #777 lands the change.

  10. dezren39
    closed this as completedon Oct 28, 2026
  11. developing-today
    added
    status:done
    and removed
    status:in-review
    on Oct 28, 2026

  12. dezren39 commented on Oct 28, 2026

    @dezren39
    dezren39
    on Oct 28, 2026
    Author

    Blocker: CI jobs are not being picked up

    Parking here. The change is complete and verified locally, but the PR's checks cannot run.

    State: head 4fc206a — check, nix-eval, test, pr-standards are all still queued ~1 hour after the push, with no runner assigned (unit (linux), typecheck, test, check, nix-eval, build (linux/windows) all pending). This includes the lightest workflows, so it looks like a runner/queue problem on the repo rather than anything in this branch.

    What is verified on the laptop instead (the task allows laptop or CI):

    • packages/cli/test/serve-handover.test.ts — 9 pass / 0 fail, including the new "a drain answers a request on a connection the server accepted before it began", which fails without the drain change and passes with it on both runtimes.
    • root bun run typecheck — 36/36.
    • real systemd user session — two generations, 0 refused, 0 failed (transcript above).
    • teardown race — 4/1851 resets before, 0/2065 after.

    PR: #777. When a runner frees up, the checks should re-run on 4fc206a; nothing further is needed from me unless they fail.

  13. dezren39
    added a commit that references this issue on Oct 30, 2026

    docs(roadmap): drop landed rows (#78/#81/#82/#8/#19) (#890)

    05dcca9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    effort/mMedium: expected under 1 day of workkind/testTest coverage, test infrastructure, or test-only fixesmeta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedpriority/normalOrdinary priority (the default)queue/queuedHas a queued task in the Supabase agent queuesource/humanFiled directly by a humanstatus:done

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions