Skip to content

fix(agent-server): make drain_reader_threads async to prevent event-loop deadlock (#657) - #666

Merged
vybe merged 2 commits into
devfrom
feature/657-drain-reader-asyncio-deadlock
May 5, 2026
Merged

vybe merged 2 commits into
devfrom
feature/657-drain-reader-asyncio-deadlock

Conversation

@vybe

@vybe vybe commented May 5, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Converts drain_reader_threads in subprocess_pgroup.py from sync to async def
  • Replaces all t.join(timeout=N) calls with await asyncio.wait_for(asyncio.to_thread(t.join, N), timeout=N+1) so deadlines are enforced at the event-loop level even under CPU starvation from orphan subprocesses
  • Replaces the orphan-killer _orphan_thread.join(timeout=10) with a daemon thread + threading.Event: event.wait(10) always returns within 10s, so asyncio.run()'s shutdown_default_executor() never blocks on a slow /proc scan
  • Updates 4 sync call sites in claude_code.py to use asyncio.run(drain_reader_threads(...))
  • Updates all 5 test call sites in test_subprocess_pgroup.py to use asyncio.run()

Changes

  • docker/base-image/agent_server/utils/subprocess_pgroup.py — drain_reader_threads → async def, asyncio-based joins
  • docker/base-image/agent_server/services/claude_code.py — 4 call sites wrapped in asyncio.run()
  • tests/unit/test_subprocess_pgroup.py — 5 call sites wrapped in asyncio.run()
  • docs/memory/feature-flows/parallel-headless-execution.md — revision history entry

Test Plan

Fixes #657

Generated with Claude Code

vybe and others added 2 commits May 5, 2026 15:26
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…oop deadlock

drain_reader_threads t.join(timeout=30) could block the asyncio event loop
permanently when a CPU-heavy orphan MCP subprocess (setsid'd, escaped
terminate_process_group) consumes 140-180% CPU and starves OS-level thread
timeouts. All three t.join() calls and the orphan-killer wait are now run
via asyncio.to_thread + asyncio.wait_for so deadlines are enforced at the
event-loop level regardless of CPU pressure.

The orphan-killer changes from _orphan_thread.join(timeout=10) to a daemon
thread + threading.Event: event.wait(10) always returns within 10s, so
asyncio.run()'s shutdown_default_executor() never blocks on a slow /proc
scan. Sync callers in claude_code.py wrap with asyncio.run().

Fixes #657

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant