Fleet tunnel: banner-phase stalls hang the ssh -L forward forever — end-to-end watchdog needed in the tunnel lifecycle #1284

Incident (2026-09-19, repeated through the day)

The fleet's launchd ssh -L forward can hang in a state where the process is alive, the local listener accepts TCP, but no forwarded request ever completes — the UI renders chrome and then nothing lands. ssh has no banner-exchange timeout: TCP connects, the remote banner never arrives (observed repeatedly on relayed Tailscale/DERP paths during today's flapping), and the connection sits in that state indefinitely. ssh's own ServerAliveInterval/CountMax does not fire pre-establishment, and ConnectTimeout only bounds TCP, not the banner.

Observed variants today:

• banner-phase stalls (above) — the forward hangs silently until manually killed
• in-place binary swaps under a live mapping wedge the vnode: subsequent execs of the same path hang in U-state (uninterruptible wait) — relevant to any flow that cps over the vendored binary while an engine is running
• path-quality inversion: direct tailscale and via-mini hop each degraded at different times of day; a forward pinned at connect-time to the bad path stalls while the other path is healthy

Shipped locally (pattern to adopt)

~/.amico/ops/amico-tunnel-watchdog.sh + launchd co.harmoniqs.amico-tunnel-watchdog: every 30s, an end-to-end HTTP probe through the forward (a real request — the only test that catches banner hangs; port checks and process liveness do not); 3 consecutive failures → kill the ssh process → launchd KeepAlive reconnects, re-evaluating the route on each connect. Worst-case blind window per stall: ~90s, versus forever. It also no-ops when the tunnel agent is unloaded (standalone) so it never spams logs.

Ask

Fold an equivalent end-to-end health loop into the fleet tunnel lifecycle in the extension (or the install that manages the launchd agent): probe through the forward, bounded-failure kill, reconnect, honest log lines. This composes with #1260's pluggable transport (every provider needs the same end-to-end liveness definition) and with #777's attach budget (a probe timeout is only meaningful against a forward that actually forwards).