Skip to content

feat(headless): bind a persistent PTY session and stdin writes to the benchmark tool surface #2242

Closed

Description

@Astro-Han

Problem

The headless benchmark surface exposes only a blocking, one-shot Bash (buildForegroundBashTool): the call returns when the command exits or the 120s timeout kills it. There is no session handle, no stdin, no mid-flight output. Any task needing an interactive process — a REPL, a debugger, an installer prompt, a booting VM — is unreachable, and any long command is all-or-nothing with no way to observe or intervene.

Codex's exec_command returns after a 10s yield with a session_id while the process keeps running, and write_stdin writes to or polls that session. Across the same 89-task Terminal-Bench 2.1 benchmark Codex issued 381 write_stdin calls; on qemu-alpine-ssh it was 22 exec_command paired 1:1 with 22 write_stdin.

Desired outcome

The headless surface can start a process, get a handle back before it finishes, read incremental output, and write to its stdin — so interactive tasks are representable and long commands are observable.

Alternatives or workarounds

packages/runtime/src/shell-tools.ts already has buildManagedBashTool with background and PTY support; the headless path simply binds the foreground variant instead (packages/headless/src/tools.ts:162). The gap is wiring, not a missing capability. The workaround the model currently discovers on its own is nohup … &, which leaves orphan processes alive into the grading phase.

Evidence

Fact Location
Codex exec_command: PTY, returns session_id, default yield_time_ms 10000 codex-rs/core/src/tools/handlers/shell_spec.rs:92
Codex write_stdin: empty write polls without writing codex-rs/core/src/tools/handlers/shell_spec.rs:142
Maka headless binds the foreground variant packages/headless/src/tools.ts:162
Managed variant already exists packages/runtime/src/shell-tools.ts

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions