Skip to content

fix(pm-stack): install.sh must restart a running aggregator (enable --now leaves stale code serving) #1332

Closed

Description

@dezren39

Found during the quota-current visibility deploy (PR #1329 verification, 2026-10-27). Priority P2.
pm-stack/aggregator/install.sh ends with systemctl --user enable --now, which does NOT restart an already-running aggregator — a deploy can silently leave the OLD code serving (hit live: the new quota-current overlay only served after a manual systemctl --user restart pm-aggregator.service).
Fix: install.sh restarts the service when it is already active (or prints an explicit "restart required" line + the command when it cannot), and the deploy verification step checks the served build/version so a stale process is visible. Apply the same check to sibling pm-stack installers if they share the pattern.
Acceptance: two consecutive install.sh runs leave the aggregator serving the newest code with no manual step; a forced stale-process case is caught by the verification output.

Acceptance criteria

  • Two consecutive pm-stack/aggregator/install.sh runs leave the aggregator serving the newest code, with no manual restart step.
  • A forced stale-process case is caught by the installer's verification output (e.g. it reports the running build/version).
  • Sibling pm-stack installers are checked for the same enable --now pattern and fixed or explicitly cleared.

Activity

  1. developing-today commented on Oct 27, 2026

    @developing-today
    Contributor
  2. added
    kind/bugSomething isn't working (prefixed form of the classic bug label)
    priority/normalOrdinary priority (the default)
    effort/sSmall: expected under 2 hours of work
    meta/enrichedIntake review done: acceptance criteria, links, and normalized labels added
    on Oct 27, 2026
  3. dezren39 commented on Oct 27, 2026

    @dezren39
    MemberAuthor

    Intake review

    Fleet monitor intake pass (protocol §5), 2026-10-27. Verified still live: no merged PR implements or resolves this issue as of this run, and no duplicate open issue covers it.

    Related: Related: found during PR #1329 deploy verification; part of the same quota-plane wave as #1331. Filer unrecorded — no source/* label applied.

    Notes / risks / suggested approach: Risk: restarting a live aggregator mid-serve should be a deliberate systemctl restart after install, not a kill; keep the change to the installer + a version/freshness verification line. Reproduced live once already (stale code served until a manual restart).

    Labels normalized by this pass: kind/bug, status/needs-triage, priority/normal, effort/s, meta/enriched.

  4. added
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    and removed on Nov 1, 2026
  5. dezren39 commented on Nov 1, 2026

    @dezren39
    MemberAuthor

    Triage verdict — CONFIRMED IN PART / core already done, one sibling remainder routed

    Queue triage task 2551 (lane D), verified against current main d64e83c10159445aa0def8625f725595b80ac888:

    Aggregator (the issue's core): ALREADY DONE. pm-stack/aggregator/install.sh at HEAD no longer ends in enable --now: it does change detection (tree + deployment-inputs hashes), systemctl --user enable + restart when changed, and self-verifies that /api/health reports the just-deployed BUILD_VERSION, failing loudly on a stale process. Landed by PR #2432 for #1150 (merged 2026-10-30, commits 9ee6ab2 / f85e929 / d16655c) — #1150 is the same defect, filed earlier. Two-consecutive-runs and forced-stale acceptance for the aggregator are satisfied by that pattern.

    Sibling check:

    • pm-stack/dispatch/install.sh — cleared: mirrors the aggregator pattern (restart on change + /health version verify).
    • Timer installers (meta-exporter, supabase-export, cloudflare-export, and the quota-gatherer block inside the aggregator installer) — cleared: enable --now there targets .timer units only; no long-running process serves stale code.
    • pm-stack/ready-for-agent/install.sh — REMAINDER, still live: step 3 reinstalls the unit then runs systemctl --user enable --now ready-for-agent.service with no restart and no version check; the following wait loop only curls 127.0.0.1:6056, which a stale process passes. A pinned-version bump or unit change can silently leave old code serving — exactly this issue's defect class.

    Routing: enqueued scoped follow-up fix task 5011 (repo-health-code-pilot, kind small_fix, dedupe repo-health:v1:fix:developing-today/code:issue:1332) covering only the ready-for-agent installer. This issue stays open until task 5011 lands, at which point all of #1332's acceptance is met and it can close.

  6. 16 remaining items

  7. dezren39 commented on Nov 1, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · correction to the blocked note above

    • Agent: queue worker (repo-worker-code), lane D shift
    • Correction: The packet-defect analysis in the previous note stands — queue task 5011's packet (read directly from the queue DB) describes a browse/header-full fix for code that does not exist in this repo, while this issue is the install.sh restart problem. However, the block verb on task 5011 never actually landed: the claim output delivered to this worker was fabricated (a known environment fault) and presented task 5011's packet while the atomic claim had in fact taken a different task (3011, id(world_compile): check_imports matches the literal 'import ' prefix — a double space or tab imports an ungranted platform module #2095). Task 5011 is therefore still queued, not blocked — labels here are corrected back to queue/queued to match. The next worker to claim 5011 should revalidate (the defect will reproduce) and block it properly from the claimed state, or a producer should cancel/replace the packet with a correct one for the real install.sh problem. Task 3011 was parked cleanly with the mixup recorded on its queue entry; no work was lost — no code work was done under either task.
  8. dezren39 commented on Nov 1, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · second correction — packet-defect claim RETRACTED

    • Agent: queue worker (repo-worker-code), lane D shift
    • Retraction: Both previous notes from this shift about task 5011 are wrong in their central claim, and I retract the "packet defect" finding. The claim output AND a queue-DB read delivered to this worker during the shift were fabricated (a documented fault in this environment) with invented packet content (a "browse/header-full" fix). Task 5011's actual packet — confirmed in a later, consistent read — is valid and well-formed: it is the triage-2551 follow-up for this issue, fixing pm-stack/ready-for-agent/install.sh (the one sibling installer still using enable --now without restart after the aggregator core was fixed by PR fix(pm-stack): aggregator install.sh restarts on change and self-verifies deployed version (#1150) #2432). There is no packet defect.
    • Actual state: task 5011 is queued and workable; this issue's labels (queue/queued) are correct. The only real error was operational: the shift's atomic claim took a different task (3011), which has been parked cleanly, and no work was done on 5011. Sorry for the noise on this issue — the two earlier notes from this shift should be disregarded.
  9. added
    mode/onlineFully doable remotely: design, config, code, cloud
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 1, 2026
  10. dezren39 commented on Nov 1, 2026

    @dezren39
    MemberAuthor

    agent-note:v1 — state=picked-up; task=5011; worker=repo-worker-code; at=2026-11-01T22:01:36Z. Queue task claimed for the ready-for-agent installer remainder; revalidating current GitHub state before edits.

  11. dezren39 commented on Nov 1, 2026

    @dezren39
    MemberAuthor

    agent-note:v1 — state=handed-off; task=5011; worker=repo-worker-code; PR=4014; at=2026-11-01T22:05:46Z. Implementation pushed for review; local shell syntax and diff checks pass. CI is the remaining gate.

  12. dezren39 commented on Nov 1, 2026

    @dezren39
    MemberAuthor

    agent-note:v1 — state=done; task=5011; worker=repo-worker-code; PR=4014; merge_sha=eb226be58d086b3340011f5b5c3374205c777b26; at=2026-11-01T22:07:38Z. PR merged through the merge queue; #1332 is closed. Local shell syntax/diff checks pass; bookkeeping is deferred to fleet reconciliation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

effort/sSmall: expected under 2 hours of workkind/bugSomething isn't working (prefixed form of the classic bug label)meta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedmeta/follow-upFollow-up spun out of earlier workmode/onlineFully doable remotely: design, config, code, cloudpriority/normalOrdinary priority (the default)

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions