Repository navigation
fix(agent-server): make drain_reader_threads async to prevent event-loop deadlock (#657) - #666
Merged
Merged
Conversation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…oop deadlock drain_reader_threads t.join(timeout=30) could block the asyncio event loop permanently when a CPU-heavy orphan MCP subprocess (setsid'd, escaped terminate_process_group) consumes 140-180% CPU and starves OS-level thread timeouts. All three t.join() calls and the orphan-killer wait are now run via asyncio.to_thread + asyncio.wait_for so deadlines are enforced at the event-loop level regardless of CPU pressure. The orphan-killer changes from _orphan_thread.join(timeout=10) to a daemon thread + threading.Event: event.wait(10) always returns within 10s, so asyncio.run()'s shutdown_default_executor() never blocks on a slow /proc scan. Sync callers in claude_code.py wrap with asyncio.run(). Fixes #657 Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
drain_reader_threadsinsubprocess_pgroup.pyfrom sync toasync deft.join(timeout=N)calls withawait asyncio.wait_for(asyncio.to_thread(t.join, N), timeout=N+1)so deadlines are enforced at the event-loop level even under CPU starvation from orphan subprocesses_orphan_thread.join(timeout=10)with a daemon thread +threading.Event:event.wait(10)always returns within 10s, soasyncio.run()'sshutdown_default_executor()never blocks on a slow/procscanclaude_code.pyto useasyncio.run(drain_reader_threads(...))test_subprocess_pgroup.pyto useasyncio.run()Changes
docker/base-image/agent_server/utils/subprocess_pgroup.py—drain_reader_threads→async def, asyncio-based joinsdocker/base-image/agent_server/services/claude_code.py— 4 call sites wrapped inasyncio.run()tests/unit/test_subprocess_pgroup.py— 5 call sites wrapped inasyncio.run()docs/memory/feature-flows/parallel-headless-execution.md— revision history entryTest Plan
test_subprocess_pgrouptests pass (3 skipped: Linux-only/proctests)TestDrainOrphanKillerTimeout.test_drain_bounded_when_orphan_killer_blockscompletes in ~14s instead of 100s — regression for deadlocktest_buffered_data_preserved_after_grandchild_killpasses — data integrity regression (bug: drain_reader_threads closes stdout pipe before reader can drain backlog — silently loses final result line on long agentic tasks #531)test_unwinds_reader_stuck_on_grandchild_pipepasses — end-to-end drain still worksFixes #657
Generated with Claude Code