You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(headless): bind a persistent PTY session and stdin writes to the benchmark tool surface #2242
The headless benchmark surface exposes only a blocking, one-shot Bash (buildForegroundBashTool): the call returns when the command exits or the 120s timeout kills it. There is no session handle, no stdin, no mid-flight output. Any task needing an interactive process — a REPL, a debugger, an installer prompt, a booting VM — is unreachable, and any long command is all-or-nothing with no way to observe or intervene.
Codex's exec_command returns after a 10s yield with a session_id while the process keeps running, and write_stdin writes to or polls that session. Across the same 89-task Terminal-Bench 2.1 benchmark Codex issued 381 write_stdin calls; on qemu-alpine-ssh it was 22 exec_command paired 1:1 with 22 write_stdin.
Desired outcome
The headless surface can start a process, get a handle back before it finishes, read incremental output, and write to its stdin — so interactive tasks are representable and long commands are observable.
Alternatives or workarounds
packages/runtime/src/shell-tools.ts already has buildManagedBashTool with background and PTY support; the headless path simply binds the foreground variant instead (packages/headless/src/tools.ts:162). The gap is wiring, not a missing capability. The workaround the model currently discovers on its own is nohup … &, which leaves orphan processes alive into the grading phase.
Activity
fix(headless): keep managed Bash timeouts and session reaping whole
feat(headless): bind managed shell sessions to the in-container tool …
fix(headless): keep managed Bash timeouts and session reaping whole
feat(headless): bind managed shell sessions to the in-container tool …