bug(docker): execute_command_in_container ignores timeout — a hung exec wedges session cleanup and can drain the 4-thread Docker pool #2969

Problem

docker_service.execute_command_in_container(..., timeout=N) accepts timeout but never forwards it; the docstring says "ACCEPTED AND NOT FORWARDED". docker-py's Container.exec_run has no timeout parameter (docker SDK 7.1.0), and the call runs on docker_utils._docker_executor, which has 4 threads. About 30 call sites pass timeout as though it bounded the call. Only 4 wrap it in asyncio.wait_for, and only one of those also bounds the process inside the container.

Impact

• Session cleanup stalls fleet-wide. SessionCleanupService._sweep_agent runs its find/rm execs one agent at a time under a single lock, with no outer bound. One hung exec stops JSONL reaping for every agent until the backend restarts (later cycles skip on _lock.locked()), and it holds one Docker thread permanently.
• The Docker pool can drain. pre_check_service.run_pre_check runs the agent-authored ~/.trinity/pre-check hook with no bound. The scheduler's HTTP call times out and fails open, but the backend thread stays blocked. Repeated fires of a hanging hook can occupy all 4 threads, and every docker_utils operation (start/stop/stats/get) then queues behind them. This part was derived from the code and not reproduced.
• An exec runs as the agent's own uid, so the agent can kill -STOP it. This was verified.

Reproduction

Against a throwaway container:

execute_command_in_container("<container>", "sh -c 'sleep 1000'", timeout=3)
# still blocked after 12s; `sleep 1000` keeps running in the container

Fix direction

Enforce the timeout inside the primitive: prefix the command with an in-container timeout N so the process and the thread are freed, and wrap the call in asyncio.wait_for so the caller is freed. Then remove the "not forwarded" note and the per-caller workarounds.