You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[amplifier-foundation] Per-spawn delegate model-pattern resolution re-queries provider list_models() with no caching — parallel spawns storm GET /v1/models #369
Every child/delegate session spawned with provider_preferences performs a fresh, un-cached provider.list_models() network call. Parallel spawns therefore fan out into N simultaneous GET /v1/models requests against the provider API. With recipes that spawn delegates in parallel, this produces a request storm proportional to spawn parallelism, adding latency to every spawn and (behind a shared egress proxy) measurable connection-table pressure.
Call chain (all on current main)
amplifier_foundation/bundle/_prepared.py — spawn() applies provider preferences before creating the child session:
amplifier_foundation/spawn_utils.py — apply_provider_preferences_with_resolution → resolve_model_pattern, which, for any glob-pattern model hint, queries the provider live (~line 208):
amplifier-module-provider-anthropic — list_models() issues await self.client.models.list() (__init__.py ~line 935) on every invocation: no memoization, no TTL, no cross-call reuse.
No caching exists at any of the three layers (foundation resolution, provider instance, coordinator). A burst of K parallel delegate spawns with a pattern hint = K concurrent identical GET /v1/models calls.
Observed impact
In a multi-agent workflow that spawns delegates in parallel waves (~80 wide at peak):
~80 simultaneous GET /v1/models per wave — the list is identical for all of them.
~16,600 such requests observed in a single day of repeated multi-agent runs.
Each spawn pays the full network round trip before the child session is created, so spawn latency scales with upstream API latency.
Behind a shared egress/credential proxy, the storm kept hundreds of concurrent connections warm and contributed to exhausting the proxy's connection table.
(Impact figures are measurements from our environment; the per-spawn call itself is directly verifiable from the code above.)
Suggested fixes (any one materially helps; both together are best)
Foundation-side memoization (the layer that owns the storm): cache the resolved pattern result (or the fetched model list) on the coordinator, keyed by (provider_name, pattern) with a short TTL (e.g. 60s). Parallel spawns in one wave then collapse to a single upstream call.
Provider-side TTL cache: memoize list_models() on the provider instance for a short window. This fixes it for all callers but only within one provider instance's lifetime.
Option 1 also keeps the per-spawn latency benefit even when the provider instance is recreated each spawn.
Model lists change rarely (on the order of weeks), so even a multi-minute TTL is functionally exact while cutting request volume by orders of magnitude.
Activity
point foundation include at fork branch with list_models single-fligh…
fix(spawn): memoize delegate list_models() with single-flight TTL cac…