Skip to content

fleet: laptop SSH unreachable since 2026-10-28 ~11:16Z — all laptop-worktree queue work blocked (MaxStartups pattern or laptop offline) #1667

Closed

Description

@dezren39

Problem

Laptop SSH has been unreachable from the sandbox since ~2026-10-28 11:16Z and remains down as of 13:1xZ. Every attempt via ~/tools/laptop-ssh.sh fails with:

Connection closed by UNKNOWN port 65535 (rc 255)

That byte pattern matches the sshd MaxStartups early-drop signature root-caused on 2026-10-25 (see ~/TOOLS.d/laptop.md), but it is equally consistent with the laptop being asleep/offline — it cannot be distinguished from the sandbox side, and that ambiguity is part of the problem.

Impact

Fleet worktrees, edits, builds and tests live on the laptop by hard rule (worker instruction §3; sandbox builds banned 2026-10-27), so this single outage blocks all implementation work in the code queue. Queue tasks parked solely on this outage so far:

Known pending permanent fix

From the 2026-10-25 root-cause: the laptop's NixOS config still needs services.openssh.settings.MaxStartups = "100:30:300" + a switch timed when no interactive laptop SSH is in use. The connection-multiplexing interim fix in laptop-ssh.sh does not help when no master connection can be established at all.

Acceptance

  • Laptop SSH reachable again from the sandbox via ~/tools/laptop-ssh.sh (a plain echo round-trip succeeds).
  • The parked tasks above can be claimed and worked.
  • The MaxStartups permanent fix is either applied or explicitly scheduled, so a burst of fleet connections cannot reproduce the drop pattern.

Activity

  1. added
    kind/bugSomething isn't working (prefixed form of the classic bug label)
    priority/highImportant and time-sensitive
    on Oct 28, 2026
  2. dezren39 commented on Oct 28, 2026

    @dezren39
    MemberAuthor

    Intake review

    • Still live: Yes — confirmed this run: queue tasks are still parking on this outage (e.g. tasks 1691 and 1821 parked at ~13:22Z naming the laptop SSH outage; opencode task 1861's checkpoint at 13:02Z records SSH still unreachable). Filed by repo-queue-worker-A at 2026-10-28T12:58Z.
    • Related: laptop sshd unreachable from sandbox (Connection closed by UNKNOWN port 65535) — apply the pending MaxStartups fix dezren39/darwin-nix#1020 (the same outage seen from the nix side — sandbox cannot reach laptop sshd), code#1637 / PR feat(mcpx): re-initialize on a legacy session 404 and retry once (#1637) #1642 (CI-green, blocked on a laptop-side conflict resolution), and the fleet of tasks parked on this: 1616, 1675, 1682, 1777, 1824, 1852, 1853 among them.
    • Notes: The body already carries an ## Acceptance section; treating it as the acceptance-criteria section (no duplicate appended). Labels normalized this pass: added source/queue-worker, status/needs-triage, effort/s, area/infra (already kind/bug, priority/high).
    • Risks: The failure signature is ambiguous from the sandbox (sshd MaxStartups early-drop vs laptop asleep/offline); any "fix" attempted purely from the sandbox risks treating the wrong cause. The permanent MaxStartups config change must be timed when no interactive laptop SSH is in use.
    • Suggested approach: First restore reachability (laptop-side check: machine awake, sshd up), verify with a ~/tools/laptop-ssh.sh echo round-trip, then schedule the services.openssh.settings.MaxStartups = "100:30:300" switch on the laptop and record it here so the parked tasks can resume.

    — fleet monitor intake pass, 2026-10-28T13:45Z

  3. added
    source/queue-workerFiled or worked by the queue/worker pilot
    effort/sSmall: expected under 2 hours of work
    area/infraInfrastructure, deployment, and operations
    meta/enrichedIntake review done: acceptance criteria, links, and normalized labels added
    on Oct 28, 2026
  4. dezren39 commented on Oct 28, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · evidence update

  5. dezren39 commented on Oct 28, 2026

    @dezren39
    MemberAuthor

    Closing — resolved. The 2026-10-28 laptop SSH outage (06:17–~08:53 CDT) was sshd MaxStartups early-drops under fleet connection bursts. The permanent fix is live: the laptop switched to NixOS generation 511 at 12:17 CDT today, and the running sshd config carries MaxStartups 100:30:250 + MaxSessions 50. Operator-verified with 5 parallel cold SSH connections, all successful. Full evidence on dezren39/darwin-nix#1020.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/infraInfrastructure, deployment, and operationseffort/sSmall: expected under 2 hours of workkind/bugSomething isn't working (prefixed form of the classic bug label)meta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedpriority/highImportant and time-sensitivesource/queue-workerFiled or worked by the queue/worker pilotstatus/needs-triageNot yet triaged; needs a first look

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions