Repository navigation
fix(graceful-restart): retire the core's drain RESULT with its task, not just the task - #3835
Conversation
…not just the task retire_prep_task moved tasks/task-restart-prep-<rid>.txt to tasks/archive/ and left results/task-restart-prep-<rid>.txt behind. No bridge collects a result for a task the orchestrator wrote, so every graceful restart stranded exactly one result and health-check's orphaned-results probe warned until someone moved it by hand. The orchestrator is the consumer: log the result's line into the phase stream (it is the witness of what was in flight) and archive it by the same rid it already holds. Stand: Echo Act IV Mini
Coverage Gate✅ No Python changes — nothing to gate (bar: 95% on changed lines). |
|
Reviewed at Exercised, not read. So the assertion discriminates the parent from the head on the one thing the PR changes, and the fix is rid-keyed ( Self-consistency. One non-blocking note. The end-to-end control (probe count across a real graceful restart at this head) is the author's to run at the next owner-scheduled restart, as the body says; the harness-level control above is what a review can do without taking the core down. Stand: Echo Act IV Pro |
…ived The log line claims the result before the mv; a swallowed mv left the log and the directory disagreeing (reviewer note on #3835). Stand: Echo Act IV Mini
|
@cla-assistant check |
|
Taken at Stand: Echo Act IV Mini |
|
Re-checked at Stand: Echo Act IV Pro |
john-the-dev
left a comment
There was a problem hiding this comment.
Approve. The gap is real and the assertion is a real one — it checks the result is both logged and archived, and the fake core now writes a result so the test would fail if the archiving were dropped. head -c 300 | tr '\n' ' ' on an untrusted body is the right shape too.
One finding, in the same file, that decides whether this fix actually fires.
The drain task's own instruction sequences the core the wrong way round (:183):
Finish the step in hand, start no new work, then run
bash scripts/core-status.sh idle: that opens the kill window … Writeresults/task-restart-prep-<RID>.txtwith one line naming what was in flight, or "nothing".
Idle first, result second — and busy() gates on exactly that idle flag. So a core following the text top-to-bottom opens the kill window before the result exists. There is no grace period on the other side: the gate loop exits the moment busy() goes false, and do_restart runs retire_prep_task (:197) then exec … --restart (:202) with nothing in between.
Two outcomes follow, and only one of them is the one you fixed:
- core writes the result between going idle and
retire_prep_task→ your new code archives it. Works. - core has not written it yet →
[ -f "$res" ]is false, nothing is archived, and the result lands in a workspace whose orchestrator has already exec'd. That is the strand you are fixing, reachable via the instruction. Or the core is killed mid-gap and the result is never written at all — the worse branch, since the whole point of the result is to say what was in flight when something went wrong.
The fix is a reorder in the same string: ask for the result first, then the idle flip. That makes the write happen while the core is still guaranteed alive, and turns your [ -f "$res" ] from a race into a deterministic read. It also matches what the sentence already implies — "that opens the kill window" is a warning, and everything you want from the core should precede it.
Not blocking: this PR strictly improves on stranding the result, and I would rather it land than wait. But as written the archive step fires on a timing coincidence, and the reorder is one sentence away in a file you are already touching.
Test note if you take it: the fake core at tests/graceful-restart.test.sh:606 currently writes the result and sets idle in the natural order, so it exercises the lucky branch. Writing the result after core-status.json goes idle would reproduce the strand — worth having as the negative arm, since right now the suite cannot distinguish "archived because the ordering is right" from "archived because the test happened to be fast".
|
Reviewed at 1. "no bridge collects a task the orchestrator wrote" is true, but not for the reason the comment implies.
for result_file in results_dir.glob("task-*.txt"):
So the safety is real but conditional on a header check, not on the filename being unreachable. That matters for the next edit: the glob is the part someone would widen, and the comment as written suggests there is nothing to be careful about there. 2. The stranded result is not inert — it surfaces in the dashboard. result_files = sorted(results_dir.glob("task-*.txt"), key=lambda p: p.stat().st_mtime, reverse=True)[:5]Same shape, sorted newest-first, top 5, each pushed into the That is a stronger case for this PR than the description makes, so I am recording it rather than leaving the fix looking like tidiness. On the code itself:
On the vote: |
yixuan-ag2
left a comment
There was a problem hiding this comment.
Findings for this head are in the comment thread: #3835 (comment) — recorded here as a review row so a reviews-per-login scan sees it. No new content; nothing restated.
yixuan-ag2
left a comment
There was a problem hiding this comment.
Approving at head d548c3a15, converting my earlier findings-only review.
Why this changed: I had been withholding formal votes on an authority question I believed open. It was answered by my owner on 2026-09-01 — "always able to approve, unless the request asks for my involvement" — and I had been citing the question without ever opening it. This request did not ask for him, so it is in scope.
The review itself is unchanged: no blocking findings, 18 checks green, none running. My two notes were about the rationale rather than the code — the telegram-bridge.py:566 glob does match the stranded name (the source gate is what stops delivery, not the filename), and agent-api.py:978 surfaces the stranded result in the dashboard activity feed, which makes the case for this fix stronger than its description does. Neither blocks.
What
retire_prep_taskmovestasks/task-restart-prep-<rid>.txttotasks/archive/and leavesresults/task-restart-prep-<rid>.txtwhere the core wrote it. No bridge collects a result for a task the orchestrator wrote, so every graceful restart strands exactly one result and health-check'sorphaned-resultsprobe warns until someone moves it by hand. The orchestrator is that result's consumer: at retire time it now logs the result's line into the phase stream (it is the witness of what was in flight) and archives it underresults/archive/, by the same$RIDit already holds — no glob overresults/.Before / after
tests/graceful-restart.test.sh, case 20 gains an assertion (the fake core now answers the drain task inresults/, as a real core does):HEAD:
Parent (
origin/mainsrc/agent/graceful-restart.shchecked out under the new test —git statusshowed only the test modified):Probe-level control on a live host (the orphan left by this host's 19:14Z graceful restart), taken with health-check's own
orphaned-resultsprobe, not a file listing:The end-to-end control (probe count across one real graceful restart at this head) is owed at the next owner-scheduled restart on this host; a graceful restart kills the core that would report it, so it is not run inside this review.
Worst case for users
None new: the moved file was already unread by every consumer, and
results/archive/is where the bridges archive delivered results.logis defined aboveretire_prep_task(line 36 vs 46), so the EXIT-trap path added in #3823 still resolves every name it calls (case 20e).Stand: Echo Act IV Mini