Repository navigation
fix: add configurable llm_call_timeout to bound stalled LLM calls - #204
Merged
Merged
Conversation
Internal LLM calls (main loop and sub-agents) had only an httpx per-read timeout, or none on the streaming path. A backend that trickles bytes without finishing resets that per-read timer forever, so tasks hang indefinitely with no error. Add a wall-clock cap enforced with asyncio.wait_for on every provider call, configurable via agents.defaults.llm_call_timeout (default 600s) and carried on GenerationSettings: - litellm, custom, azure and codex providers all honor the timeout; the streaming paths use a per-chunk idle cap so long but progressing generations are not cut off. - classify_error now treats the builtin TimeoutError raised by wait_for as a retryable network error, so the retry and model-fallback ladder engages instead of failing as unknown. - the streaming loop converts a timeout into a structured error response and closes the stream, instead of crashing the turn. - PerModelProvider propagates generation settings to its per-model sub-providers, so knn-routed calls honor the configured timeout, temperature and max_tokens. Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
arelchan
approved these changes
Jul 23, 2026
This was referenced Jul 23, 2026
0xKT
added a commit
that referenced
this pull request
Jul 23, 2026
## Summary Bump the package version from 0.1.8 to 0.1.9 (patch release). All five PRs merged since v0.1.8 are fixes, no new features: - #204 fix: add configurable llm_call_timeout to bound stalled LLM calls - #199 fix(cli): make onboard readable on light-background terminals - #189 fix(cli): pick the first node >= 22 across all PATH entries - #196 fix(commitlint): unify commit/PR length to header-max-length - #190 fix(tui): surface the real turn-failure detail, not just the code ## Type - [ ] Fix - [ ] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [x] Other ## Verification - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed Bump is limited to `pyproject.toml` and `uv.lock` (`uv lock` sync). Commit message lint passes locally. ## Risk - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes Version-only change; no code or behavior change. ## Related Issues N/A Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
KevinHuSh
pushed a commit
to KevinHuSh/Raven
that referenced
this pull request
Sep 14, 2026
…erMind-AI#204) ## Summary Internal LLM calls (main loop and sub-agents) had only an httpx per-read timeout, or none on the streaming path. A backend that trickles bytes without finishing resets that per-read timer forever, so tasks hang indefinitely with no error (about 1 in 6 tasks in a 1000+ batch rollout). This adds a configurable wall-clock cap enforced with `asyncio.wait_for` on every provider call: - New `agents.defaults.llm_call_timeout` (default 600s), carried on `GenerationSettings` and threaded to every provider construction (eager, lazy, and knn per-model). - litellm, custom, azure and codex providers all honor it. Streaming paths use a per-chunk idle cap, so a long but steadily-progressing generation is not cut off while a mid-stream stall raises after `llm_call_timeout` seconds of silence. - `classify_error` now treats the builtin `TimeoutError` raised by `wait_for` as a retryable `network` error, so the existing retry and model-fallback ladder engages (previously it fell to `unknown` and never retried). - The streaming loop converts a timeout into a structured error response and closes the stream, instead of letting it crash the turn. - `PerModelProvider` now propagates generation settings to its per-model sub-providers, so knn-routed calls honor the configured timeout, temperature and max_tokens. ## Type - [x] Fix ## Verification - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally Commands run: - `uv run pytest` over the provider / config / agent-loop test files (litellm timeout, azure, codex, fallback chain, per-model, lazy, agent-loop stream, gateway, stream, config loader): 74 passed. - `bash .claude/scripts/preflight_ci.sh`: all checks passed (commitlint, commit-message lint, large-files, pre-commit, ruff lint, focused tests 58 passed). New tests cover: provider-layer wall-clock and per-chunk idle timeouts, the `classify_error(TimeoutError)` regression, the streaming structured-error path, and per-model generation propagation. ## Risk - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes The default of 600s matches the previous hardcoded litellm cap, so behavior does not regress; the cap is now a wall-clock bound (closing the trickle-stall hole) and is configurable. New config field with a default, no schema or API breaking change. Rollback is a single-commit revert. ## Related Issues Fixes EverMind-AI#150 Refs EverMind-AI#183 (this PR incidentally fixes the knn generation-propagation leak tracked there) and EverMind-AI#149 (the timeout surfaces as a classified error; the run-level structured termination reason is to be coordinated with that issue). Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
KevinHuSh
pushed a commit
to KevinHuSh/Raven
that referenced
this pull request
Sep 14, 2026
## Summary Bump the package version from 0.1.8 to 0.1.9 (patch release). All five PRs merged since v0.1.8 are fixes, no new features: - EverMind-AI#204 fix: add configurable llm_call_timeout to bound stalled LLM calls - EverMind-AI#199 fix(cli): make onboard readable on light-background terminals - EverMind-AI#189 fix(cli): pick the first node >= 22 across all PATH entries - EverMind-AI#196 fix(commitlint): unify commit/PR length to header-max-length - EverMind-AI#190 fix(tui): surface the real turn-failure detail, not just the code ## Type - [ ] Fix - [ ] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [x] Other ## Verification - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [ ] User-facing docs or screenshots are updated when needed Bump is limited to `pyproject.toml` and `uv.lock` (`uv lock` sync). Commit message lint passes locally. ## Risk - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes Version-only change; no code or behavior change. ## Related Issues N/A Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Internal LLM calls (main loop and sub-agents) had only an httpx per-read timeout, or none on the streaming path. A backend that trickles bytes without finishing resets that per-read timer forever, so tasks hang indefinitely with no error (about 1 in 6 tasks in a 1000+ batch rollout).
This adds a configurable wall-clock cap enforced with
asyncio.wait_foron every provider call:agents.defaults.llm_call_timeout(default 600s), carried onGenerationSettingsand threaded to every provider construction (eager, lazy, and knn per-model).llm_call_timeoutseconds of silence.classify_errornow treats the builtinTimeoutErrorraised bywait_foras a retryablenetworkerror, so the existing retry and model-fallback ladder engages (previously it fell tounknownand never retried).PerModelProvidernow propagates generation settings to its per-model sub-providers, so knn-routed calls honor the configured timeout, temperature and max_tokens.Type
Verification
Commands run:
uv run pytestover the provider / config / agent-loop test files (litellm timeout, azure, codex, fallback chain, per-model, lazy, agent-loop stream, gateway, stream, config loader): 74 passed.bash .claude/scripts/preflight_ci.sh: all checks passed (commitlint, commit-message lint, large-files, pre-commit, ruff lint, focused tests 58 passed).New tests cover: provider-layer wall-clock and per-chunk idle timeouts, the
classify_error(TimeoutError)regression, the streaming structured-error path, and per-model generation propagation.Risk
The default of 600s matches the previous hardcoded litellm cap, so behavior does not regress; the cap is now a wall-clock bound (closing the trickle-stall hole) and is configurable. New config field with a default, no schema or API breaking change. Rollback is a single-commit revert.
Related Issues
Fixes #150
Refs #183 (this PR incidentally fixes the knn generation-propagation leak tracked there) and #149 (the timeout surfaces as a classified error; the run-level structured termination reason is to be coordinated with that issue).