Repository navigation
Settled hosted executions become indeterminate after a diagnostics disconnect #2662
Description
Problem
A hosted execution may already have settled as completed or failed, but runHostedExecution() performs another diagnostics request afterward.
If that request loses the Runtime Host connection, the settled outcome is replaced with indeterminate. Eval then skips verification and may retry an execution whose authoritative result was already known.
Post-settlement observation failure should not rewrite the execution outcome.
Expected behavior
Once Runtime Host reports a settled completed or failed projection, that outcome remains authoritative. Later connection or cleanup failures may be reported separately, but must not change the execution result.
Proposed direction
Simplify the handoff after a settled projection:
- close the client connection;
- release the owned ephemeral Runtime Host to the enclosing Trial environment;
- let
idleGraceMs=0stop a Host with no retained runtime resources; - let the Trial finalizer reclaim a Host whose resources must remain available to the verifier.
This should allow removal of the post-settlement diagnostics request, environmentResourcesRemain, and the client-side dependency on runtime-resource labels.
Cancelled and indeterminate executions must still settle or terminate before control returns to the verifier.
Done when
- A settled
completedorfailedprojection remains authoritative if the connection becomes unavailable afterward. - A Host with no retained resources exits automatically.
- Resources needed by a shared-environment verifier remain available until Trial finalization.
- Cancelled and indeterminate paths are fully settled or terminated before returning.
- The obsolete diagnostics and runtime-resource inspection path is removed.
中文对照
问题
Hosted execution 已经得到 completed 或 failed 后，runHostedExecution() 还会发起一次 diagnostics 请求。如果此时 Runtime Host 连接断开，已经结算的结果会被改写为 indeterminate，导致 Eval 跳过 verifier，并可能重复执行本来已经有权威结果的 attempt。
结算后的观测失败不应改写执行结果。
期望行为
Runtime Host 一旦返回已结算的 completed 或 failed，该结果就保持权威。后续连接或清理失败可以单独报告，但不能改变执行结果。
建议方向
删除结算后的额外 diagnostics 查询。关闭 client connection 后，把已结算的临时 Runtime Host 交还给 Trial environment。没有保留资源的 Host 通过 idleGraceMs=0 自动退出；需要为 verifier 保留资源的 Host 由 Trial finalizer 回收。
取消或无法确定的执行仍必须在返回 verifier 前完成结算或终止。
This issue was drafted with material assistance from OpenAI Codex and reviewed by a human before publication.
Activity
Astro-Han commentedon Aug 11, 2026
I’m taking this issue. I’ll keep the change scoped to hosted-execution terminal monotonicity: once Runtime Host has produced a canonical
completedorfailedresult, later abort, connection, or cleanup events must not rewrite it.The implementation will remove the client’s post-settlement diagnostics and runtime-resource label inspection, preserve full settlement for cancelled and indeterminate paths, and add focused regression coverage. I do not plan to broaden this into a hosted-execution runner rewrite.
AI disclosure: OpenAI Codex is assisting with the implementation and drafted this comment; I reviewed the scope and wording before posting.
fix(runtime-host): preserve settled hosted outcomes (#2671)