Skip to content

[runner] Mark logs finished only once the executor has finished - #4280

Merged
un-def merged 1 commit into
masterfrom
pr_runner_defer_logs_finished_state
Sep 9, 2026
Merged

un-def merged 1 commit into
masterfrom
pr_runner_defer_logs_finished_state

Conversation

@un-def

@un-def un-def commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Previously, Server.stop() set WaitLogsFinished as soon as /api/stop arrived. /api/pull treats the first request it serves in that state as the final one and closes pullDoneCh -- the barrier that keeps the runner alive until the server has fetched the job's last logs and state.

The job is still stopping at that point. The command has up to killDelay before it is killed, so the state being served is not final, and the barrier is satisfied before the state it is meant to protect exists:

08:38:39  /api/stop
08:38:41  /api/pull                     <- closes pullDoneCh here
08:38:51  Job state changed new=terminated
08:38:51  Job finished, shutting down
08:38:51  Logs streaming finished endpoint=/api/pull   <- 92us later

The runner exited 64ms after computing the terminal state, without ever serving a pull that carried it. The server's next poll found nothing listening, reported the instance unreachable, and terminated the job on the disconnect timeout two minutes later -- discarding the state the runner had worked out.

Now stop() only cancels the run. The executor sets WaitLogsFinished from its own defer when Run returns, after the terminal job state has been recorded, so the first pull to observe it carries that state and the runner exits only once the server has taken it.

HasMore is derived from the same state and was likewise reported as false for the whole stop window. Nothing observed it: the server's PullResponse has no such field.

Previously, `Server.stop()` set WaitLogsFinished as soon as `/api/stop`
arrived. `/api/pull` treats the first request it serves in that state as
the final one and closes `pullDoneCh` -- the barrier that keeps the
runner alive until the server has fetched the job's last logs and state.

The job is still stopping at that point. The command has up to
`killDelay` before it is killed, so the state being served is not final,
and the barrier is satisfied before the state it is meant to protect
exists:

    08:38:39  /api/stop
    08:38:41  /api/pull                     <- closes pullDoneCh here
    08:38:51  Job state changed new=terminated
    08:38:51  Job finished, shutting down
    08:38:51  Logs streaming finished endpoint=/api/pull   <- 92us later

The runner exited 64ms after computing the terminal state, without ever
serving a pull that carried it. The server's next poll found nothing
listening, reported the instance unreachable, and terminated the job on
the disconnect timeout two minutes later -- discarding the state the
runner had worked out.

Now `stop()` only cancels the run. The executor sets WaitLogsFinished
from its own defer when `Run` returns, after the terminal job state has
been recorded, so the first pull to observe it carries that state and
the runner exits only once the server has taken it.

`HasMore` is derived from the same state and was likewise reported as
false for the whole stop window. Nothing observed it: the server's
`PullResponse` has no such field.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@un-def
un-def merged commit 34d4022 into master Sep 9, 2026
26 checks passed
@un-def
un-def deleted the pr_runner_defer_logs_finished_state branch September 9, 2026 10:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant