Repository navigation
fix(spawn): memoize delegate list_models() with single-flight TTL cache - #302
Conversation
…light TTL cache Every child/delegate spawn with a glob-pattern model hint re-queried the provider live via resolve_model_pattern(). Parallel delegate waves fired one identical GET /v1/models per spawned session (observed ~80 concurrent calls per wave; ~16k/day in multi-agent setups), adding full-RTT latency to every spawn and pressuring shared egress connection tables. Memoize the fetched list per provider instance for 60s (model catalogs change rarely) behind an async single-flight lock, so a concurrent wave collapses into one upstream call. Failures are never cached; non-pattern hints still skip the query entirely. Refs microsoft/amplifier#369
Hardens microsoft#302's per-provider single-flight TTL cache for provider.list_models() in amplifier_foundation/spawn_utils.py, on top of danshapiro's original fix (f74168a). Three verified defects fixed: 1. Non-answers were cached as truth. A provider returning `[]` on a transient outage (e.g. ollama on ConnectionError/OSError/TimeoutError) got that empty result memoized for the full TTL (default 60s), so model-pattern resolution stayed broken for up to 60s after the provider actually recovered. Fixed: only a genuinely non-empty `list_models()` result populates the cache entry; an empty result is left as "no answer yet" so the very next resolution refetches live. Note: a soft-failure provider that degrades to a *non-empty* fallback list on error (e.g. chat-completions returning `[configured_model]` on any exception) is NOT distinguishable from a legitimately single-model catalog at this generic, provider-agnostic layer -- `test_sequential_resolutions_reuse_within_ttl` (existing, kept green) requires a genuine single-model result to be cached, so this fix cannot special-case "list of length 1" without breaking that contract. For that residual case, the (now-configurable) TTL bounds the exposure instead of a fixed, hardcoded 60s; the complete fix is a provider-side change (raise instead of soft-fail), which microsoft#302's original "failures are never cached" behavior already handles correctly once made. 2. The single-flight lock held the network call itself, so a caller that arrived after the lock was released but before the fetch finished re-ran its own full fetch (and, with provider-level retries, its own full retry campaign) instead of joining the in-flight one. Restructured so the lock only guards entry/task bookkeeping: the fetch now runs as a shared asyncio.Task (`_fetch_and_store`), and every concurrent caller -- success or failure -- awaits that same task. A failed task clears itself from the entry before raising, so it never persists past its own delivery: the next new caller starts a genuinely fresh fetch. 3. A waiter could block forever on someone else's hung fetch. Waiting on the shared task is now wrapped in `asyncio.timeout` (default 20s) and shielded; on timeout the caller falls through to its own direct, uncached `provider.list_models()` call instead of stalling an entire delegate wave behind one slow/hung fetch. 4. TTL and wait-timeout are now configurable via AMPLIFIER_LIST_MODELS_CACHE_TTL_SECONDS and AMPLIFIER_LIST_MODELS_WAIT_TIMEOUT_SECONDS, falling back to the existing LIST_MODELS_CACHE_TTL_SECONDS / new LIST_MODELS_WAIT_TIMEOUT_SECONDS module globals (still directly monkey-patchable, preserving the existing test/override surface). Verified against repro_302.py / repro_302_hol.py (standalone repros from PR due diligence): CASE 1 (ollama-style empty-list soft failure) now recovers at t+1 instead of being stuck for the full TTL. The head-of-line repro still collapses an 80-spawn wave into 1 shared upstream fetch (3 HTTP attempts from the provider's own internal retry ladder, not 240) on the happy path. Tests: added TestListModelsCacheHardening (empty-list-not-cached / refetch / later-caches, degraded-single-model-list bounded-by-TTL, concurrent-failure-shared-once-then-fresh-fetch, waiter-timeout-falls- through) and TestListModelsConfigurableKnobs (env override + module- global fallback for both knobs). All of microsoft#302's original tests pass unchanged. Full suite: 1563 passed. Fixes: microsoft#302 Co-Authored-By: danshapiro <danshapiro@users.noreply.github.com> 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Brian Krabach (bkrabach)
left a comment
There was a problem hiding this comment.
Due-diligence review + hardening: (1) only non-empty list_models results are cached, so soft-failure providers (ollama's [] on outage) no longer poison resolution for the TTL — repro-verified recovery on the call after the blip; (2) the in-flight fetch is a shared task delivered once to all concurrent waiters (success AND failure), preventing serial per-waiter retry campaigns during outages; (3) waiter timeout with uncached fallthrough so one hung fetch cannot stall a delegate wave; (4) TTL/wait knobs env-configurable. Single-flight collapse verified intact (80-wave -> 1 upstream fetch). Full suite 1563 passed. Known residual documented in-code: a soft-failure provider returning a plausible non-empty degraded list (chat-completions single-model fallback) is indistinguishable at this layer — needs a provider-side fix, tracked separately.
…mary (#318) _apply_single_override() promotes the override target's provider instance to priority=0 in the CHILD mount plan but never demotes any other instance already sitting at (or below) that priority. When a user's primary provider is declared FIRST at priority=0 -- the natural "make this my default model" config -- every sub-agent provider preference (role-resolved prefs and agent-frontmatter provider_preferences alike) ties the primary at priority=0. Provider resolution breaks that tie with a stable sort, so the first-declared instance wins: the primary, silently, regardless of what the override selected. Live evidence: a spawn recorded provider_preferences=[{model: gpt-5.6-terra}] in its delegate:agent_spawned event yet the sub-session ran gpt-5.6-sol (the primary). In a 5-run eval, 100% of sub-agent LLM calls ran the primary regardless of role -- completely silent. Fix: after promoting the override target to priority=0 in the child mount plan, demote every OTHER provider in that same child mount plan whose priority ties or beats it, to strictly below the target. This guarantees the overridden instance always strictly wins its own child's provider resolution -- it can no longer merely tie for it. The demotion touches only the child mount plan being built for this spawn (apply_provider_preferences / apply_provider_preferences_with_resolution are the only two callers); root/parent session resolution is untouched, and no public data shape changes. A debug-level log fires whenever a tie-demotion actually occurs, so the substitution-class failure is observable going forward. Complements #302 (memoized list_models() in the same file) -- no overlap in the code paths touched. Fixes: openai_improvement-ejq 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-authored-by: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Summary
Fixes microsoft/amplifier#369
Every child/delegate spawn with a glob-pattern model hint resolved the pattern by calling
provider.list_models()live — andresolve_model_patternruns once per spawn. Parallel delegate waves therefore fired one identicalGET /v1/modelsper spawned session (observed: ~80 concurrent calls per wave, ~16k/day in a multi-agent setup), adding a full network round trip to every spawn and pressuring shared egress connection tables.Change
Memoize the fetched model list per provider instance in
spawn_utils:LIST_MODELS_CACHE_TTL_SECONDS) — model catalogs change rarely, so a short window is functionally exact while collapsing a whole wave into a single upstream call.Tests
6 new regression tests in
tests/test_spawn_utils.py(TestListModelsSingleFlightMemoization):Full suite: 1533 passed (
uv run pytest tests/ -q, excluding the gRPC integration file).Notes
list_models()caching, but that cannot coalesce a parallel wave the way this shared-resolution point does.