Skip to content

fix(spawn): memoize delegate list_models() with single-flight TTL cache - #302

Merged
Brian Krabach (bkrabach) merged 2 commits into
microsoft:mainfrom
danshapiro:fix/delegate-list-models-singleflight
Aug 27, 2026
Merged

Brian Krabach (bkrabach) merged 2 commits into
microsoft:mainfrom
danshapiro:fix/delegate-list-models-singleflight

Conversation

@danshapiro

Copy link
Copy Markdown
Contributor

Summary

Fixes microsoft/amplifier#369

Every child/delegate spawn with a glob-pattern model hint resolved the pattern by calling provider.list_models() live — and resolve_model_pattern runs once per spawn. Parallel delegate waves therefore fired one identical GET /v1/models per spawned session (observed: ~80 concurrent calls per wave, ~16k/day in a multi-agent setup), adding a full network round trip to every spawn and pressuring shared egress connection tables.

Change

Memoize the fetched model list per provider instance in spawn_utils:

  • TTL of 60s (LIST_MODELS_CACHE_TTL_SECONDS) — model catalogs change rarely, so a short window is functionally exact while collapsing a whole wave into a single upstream call.
  • Single-flight: concurrent resolutions against the same provider coalesce behind one async lock — the first task fetches, the rest await its result. N parallel spawns = 1 upstream call.
  • Cache keyed by provider instance via weak references, so entries die with the provider; lock is rotated if the provider is driven from a different event loop.
  • Failures are never cached — errors propagate exactly as an uncached call would.
  • Non-pattern hints still short-circuit before any query, unchanged.

Tests

6 new regression tests in tests/test_spawn_utils.py (TestListModelsSingleFlightMemoization):

  • 50 concurrent resolutions → exactly 1 upstream fetch
  • sequential resolutions reuse the cache within the TTL
  • refetch after TTL expiry
  • exact (non-glob) model names never fetch
  • independent provider instances cached independently
  • a failing fetch is not cached; the next resolution retries

Full suite: 1533 passed (uv run pytest tests/ -q, excluding the gRPC integration file).

Notes

  • No behavior change to resolution results, ordering, or failure semantics — only fetch frequency and concurrency.
  • Providers could separately add their own list_models() caching, but that cannot coalesce a parallel wave the way this shared-resolution point does.

…light TTL cache

Every child/delegate spawn with a glob-pattern model hint re-queried the
provider live via resolve_model_pattern(). Parallel delegate waves fired one
identical GET /v1/models per spawned session (observed ~80 concurrent calls
per wave; ~16k/day in multi-agent setups), adding full-RTT latency to every
spawn and pressuring shared egress connection tables.

Memoize the fetched list per provider instance for 60s (model catalogs change
rarely) behind an async single-flight lock, so a concurrent wave collapses
into one upstream call. Failures are never cached; non-pattern hints still
skip the query entirely.

Refs microsoft/amplifier#369
Hardens microsoft#302's per-provider single-flight TTL cache for
provider.list_models() in amplifier_foundation/spawn_utils.py, on top of
danshapiro's original fix (f74168a). Three verified defects fixed:

1. Non-answers were cached as truth. A provider returning `[]` on a
   transient outage (e.g. ollama on ConnectionError/OSError/TimeoutError)
   got that empty result memoized for the full TTL (default 60s), so
   model-pattern resolution stayed broken for up to 60s after the
   provider actually recovered. Fixed: only a genuinely non-empty
   `list_models()` result populates the cache entry; an empty result is
   left as "no answer yet" so the very next resolution refetches live.

   Note: a soft-failure provider that degrades to a *non-empty* fallback
   list on error (e.g. chat-completions returning `[configured_model]`
   on any exception) is NOT distinguishable from a legitimately
   single-model catalog at this generic, provider-agnostic layer --
   `test_sequential_resolutions_reuse_within_ttl` (existing, kept green)
   requires a genuine single-model result to be cached, so this fix
   cannot special-case "list of length 1" without breaking that
   contract. For that residual case, the (now-configurable) TTL bounds
   the exposure instead of a fixed, hardcoded 60s; the complete fix is a
   provider-side change (raise instead of soft-fail), which microsoft#302's
   original "failures are never cached" behavior already handles
   correctly once made.

2. The single-flight lock held the network call itself, so a caller
   that arrived after the lock was released but before the fetch
   finished re-ran its own full fetch (and, with provider-level
   retries, its own full retry campaign) instead of joining the
   in-flight one. Restructured so the lock only guards entry/task
   bookkeeping: the fetch now runs as a shared asyncio.Task
   (`_fetch_and_store`), and every concurrent caller -- success or
   failure -- awaits that same task. A failed task clears itself from
   the entry before raising, so it never persists past its own
   delivery: the next new caller starts a genuinely fresh fetch.

3. A waiter could block forever on someone else's hung fetch. Waiting
   on the shared task is now wrapped in `asyncio.timeout` (default 20s)
   and shielded; on timeout the caller falls through to its own direct,
   uncached `provider.list_models()` call instead of stalling an entire
   delegate wave behind one slow/hung fetch.

4. TTL and wait-timeout are now configurable via
   AMPLIFIER_LIST_MODELS_CACHE_TTL_SECONDS and
   AMPLIFIER_LIST_MODELS_WAIT_TIMEOUT_SECONDS, falling back to the
   existing LIST_MODELS_CACHE_TTL_SECONDS / new
   LIST_MODELS_WAIT_TIMEOUT_SECONDS module globals (still directly
   monkey-patchable, preserving the existing test/override surface).

Verified against repro_302.py / repro_302_hol.py (standalone repros from
PR due diligence): CASE 1 (ollama-style empty-list soft failure) now
recovers at t+1 instead of being stuck for the full TTL. The
head-of-line repro still collapses an 80-spawn wave into 1 shared
upstream fetch (3 HTTP attempts from the provider's own internal retry
ladder, not 240) on the happy path.

Tests: added TestListModelsCacheHardening (empty-list-not-cached /
refetch / later-caches, degraded-single-model-list bounded-by-TTL,
concurrent-failure-shared-once-then-fresh-fetch, waiter-timeout-falls-
through) and TestListModelsConfigurableKnobs (env override + module-
global fallback for both knobs). All of microsoft#302's original tests pass
unchanged. Full suite: 1563 passed.

Fixes: microsoft#302

Co-Authored-By: danshapiro <danshapiro@users.noreply.github.com>
🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

@bkrabach Brian Krabach (bkrabach) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due-diligence review + hardening: (1) only non-empty list_models results are cached, so soft-failure providers (ollama's [] on outage) no longer poison resolution for the TTL — repro-verified recovery on the call after the blip; (2) the in-flight fetch is a shared task delivered once to all concurrent waiters (success AND failure), preventing serial per-waiter retry campaigns during outages; (3) waiter timeout with uncached fallthrough so one hung fetch cannot stall a delegate wave; (4) TTL/wait knobs env-configurable. Single-flight collapse verified intact (80-wave -> 1 upstream fetch). Full suite 1563 passed. Known residual documented in-code: a soft-failure provider returning a plausible non-empty degraded list (chat-completions single-model fallback) is indistinguishable at this layer — needs a provider-side fix, tracked separately.

@bkrabach
Brian Krabach (bkrabach) merged commit 24a467c into microsoft:main Aug 27, 2026
7 checks passed
Brian Krabach (bkrabach) added a commit that referenced this pull request Aug 27, 2026
…mary (#318)

_apply_single_override() promotes the override target's provider
instance to priority=0 in the CHILD mount plan but never demotes any
other instance already sitting at (or below) that priority. When a
user's primary provider is declared FIRST at priority=0 -- the natural
"make this my default model" config -- every sub-agent provider
preference (role-resolved prefs and agent-frontmatter
provider_preferences alike) ties the primary at priority=0. Provider
resolution breaks that tie with a stable sort, so the first-declared
instance wins: the primary, silently, regardless of what the override
selected.

Live evidence: a spawn recorded provider_preferences=[{model:
gpt-5.6-terra}] in its delegate:agent_spawned event yet the sub-session
ran gpt-5.6-sol (the primary). In a 5-run eval, 100% of sub-agent LLM
calls ran the primary regardless of role -- completely silent.

Fix: after promoting the override target to priority=0 in the child
mount plan, demote every OTHER provider in that same child mount plan
whose priority ties or beats it, to strictly below the target. This
guarantees the overridden instance always strictly wins its own
child's provider resolution -- it can no longer merely tie for it. The
demotion touches only the child mount plan being built for this spawn
(apply_provider_preferences / apply_provider_preferences_with_resolution
are the only two callers); root/parent session resolution is
untouched, and no public data shape changes. A debug-level log fires
whenever a tie-demotion actually occurs, so the substitution-class
failure is observable going forward.

Complements #302 (memoized list_models() in the same file) -- no
overlap in the code paths touched.

Fixes: openai_improvement-ejq

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-authored-by: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants