Skip to content

ops hardening: swap must archive the outgoing binary; refuse non-release candidates; smoke needs a soak; finish fleet-status systemd port on erlich #776

Description

@aarontrowbridge

From the 2026-09-03 hub-wedge incident (see #775 + vault sessions/session-20260903-hub-wedge-forensics.md). Four gaps, each small:

1. hub-restart.sh swap must archive the outgoing binary

The 11:12 swap overwrote c99d6eab — the only binary that provably survived production (4 days) — with no copy kept. The Aug-30-era manual flow kept .bak/.new; the script path doesn't. The rollback path is only as good as the last archive.

  • swap saves the outgoing binary to bin/archive/<sha>-<date> (or the opencode.prev convention from the upgrade receipts) before the rename.

2. swap should refuse non-release candidates

This morning's wedge-era binary was a local dev build (0.0.0--202609031509 — no version injection), staged and swapped through the otherwise-safe path. Release-only is policy (amicode-release.yml); enforce it at the seam.

  • swap rejects candidates whose --version matches 0.0.0-* unless --force (and document the force escape hatch).

3. hub-upgrade-smoke.sh needs a post-listening soak

Both wedge classes from #775 pass today's smoke ("listening within 25 s"): the dev build served for ~12 min before wedging; the .18 release wedged ~90 s after listening. The harness now exists offline (incident dir); wire it in.

  • After "listening", poll /session + CPU for a configurable soak (default ≥ 15 min), fail on timeout or RSS growth anomaly.

4. fleet-status systemd port on erlich is unfinished

No timer ran today (the 11:23 collection was manual); the guard still has launchctl remnants (reported "server not running" while it served 200); DEVICES order/aliases are from the mini-canonical era.

  • systemd --user timer (5 min) + systemctl pid lookup + verify the guard's bad-state notification path works on erlich.

Activity

  1. aarontrowbridge commented on Sep 3, 2026

    @aarontrowbridge
    MemberAuthor

    Client-side hit of the same #270 class (2026-09-03, plane wifi): Go Standalone failed on a fleet client — the vendored amicode.18 binary (pre-#272-guards) crashed on the local DB with duplicate column name: directories because the local journal lacked 20260828201050_normal_stryfe. Heal applied manually (journal row inserted; backup kept). This strengthens the case for shipping a guarded release (the .18 binary is the vendored CLIENT binary too — every fleet client's local DB can be drifted): any pre-guard binary blocks Go Standalone, which is exactly the mode you want when the hub is unreachable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions