Repository navigation
ChatCompletionCache cache key ignores tool_choice, serves stale cached response when only tool_choice differs #7968
Description
What happened?
ChatCompletionCache's cache key doesn't include tool_choice. Two calls with identical messages/tools/json_output/extra_create_args but different tool_choice collide on the same cache key, so the second call silently gets served a stale, wrong-shaped cached response instead of hitting the underlying client.
tool_choice is a first-class parameter of ChatCompletionClient.create/create_stream (auto lets the model decide, required forces a tool call, none forbids tool calls, a specific Tool forces that exact tool), and it materially changes the shape of the response, in the real OpenAI client it's translated directly into the API's tool_choice field and changes whether the response is plain text or a list of FunctionCalls.
ChatCompletionCache._check_cache (python/packages/autogen-ext/src/autogen_ext/models/cache/_chat_completion_cache.py) builds its hash from messages, tools, json_output, and extra_create_args only:
data = {
"messages": [message.model_dump() for message in messages],
"tools": [(tool.schema if isinstance(tool, Tool) else tool) for tool in tools],
"json_output": json_output_data,
"extra_create_args": extra_create_args,
}
serialized_data = json.dumps(data, sort_keys=True)
cache_key = hashlib.sha256(serialized_data.encode()).hexdigest()create() calls self._check_cache(messages, tools, json_output, extra_create_args), no tool_choice, even though it's an explicit parameter of create() itself and gets correctly forwarded to the underlying client a few lines later on cache miss. create_stream() has the identical gap.
This is exactly the scenario the codebase itself exercises internally: AssistantAgent._reflect_on_tool_use_flow explicitly passes tool_choice="none" to force a plain-text reflection response after tool execution, while the normal tool-call path uses the default tool_choice="auto". Any application/agent design that reuses one ChatCompletionCache-wrapped client across calls that vary only tool_choice for the same conversation prefix will silently get the wrong kind of answer.
What did you expect to happen?
A call that only differs from a previous one by tool_choice should be a cache miss and actually invoke the underlying client, not replay a response computed for a different tool_choice.
How can we reproduce it (as minimally and precisely as possible)?
import asyncio
from autogen_core import InMemoryStore
from autogen_core.models import UserMessage
from autogen_ext.models.cache import ChatCompletionCache
from autogen_ext.models.replay import ReplayChatCompletionClient
async def main():
responses = ["This is dummy message number 0", "This is dummy message number 1"]
replay_client = ReplayChatCompletionClient(responses)
replay_client.set_cached_bool_value(False)
cached_client = ChatCompletionCache(replay_client, InMemoryStore())
message = [UserMessage(content="hi", source="user")]
r0 = await cached_client.create(message, tool_choice="auto")
print("call 1 (tool_choice=auto):", r0.cached, repr(r0.content))
r1 = await cached_client.create(message, tool_choice="none")
print("call 2 (tool_choice=none):", r1.cached, repr(r1.content))
asyncio.run(main())Output:
call 1 (tool_choice=auto): False 'This is dummy message number 0'
call 2 (tool_choice=none): True 'This is dummy message number 0'
Call 2 reports cached=True and returns call 1's response, even though it was made with a different tool_choice and, with a real tool_choice-aware client, should produce a genuinely different (plain-text) response. I also reproduced this with a fake client whose response actually depends on tool_choice (mirroring a real provider), confirming the underlying client is never invoked for the second call at all (only tool_choice='auto' shows up in the fake client's call log).
AutoGen version
Reproduced against the current main branch source of _chat_completion_cache.py, using the installed autogen-ext/autogen-core packages from an editable install of the same source. Python 3.12.10, Windows.
Which package was this bug in?
Python autogen-ext
Model used
N/A, reproduced with ReplayChatCompletionClient and a fake client, no real model API needed.
Python version
3.12
Operating system
Windows
Additional Information
Opening a fix, will link the PR here.
Activity
0xbrainkid commentedon Jul 18, 2026
Good catch. I would treat
tool_choiceas part of the agent's effective capability state, not just a model-call formatting option.The risky case is any workflow that alternates between:
If those two calls share a cache key, the cache can replay a response produced under a different authority boundary. That is more serious than a stale text response because the caller may interpret the returned shape under the current step's policy, while it was generated under a previous step's policy.
A regression test I would want in the fix:
tool_choice="required"or a specific tooltool_choice="none"tool_choiceIt may also be worth normalizing explicit
tool_choice="auto"and omitted/defaulttool_choicedeliberately, rather than letting those become accidental cache behavior. The important invariant is that any parameter changing whether an agent can call tools, which tool it can call, or what response shape is valid must be represented in the cache key.This is the same class of issue we keep running into with multi-agent systems: identity and authorization are checked at the boundary, but the execution record needs to preserve the exact capability context that produced a result.
ErenAta16 commentedon Jul 18, 2026
That framing is better than mine, "effective capability state" names the actual invariant instead of just the symptom. My original report treated this as a cache-key completeness bug (tool_choice missing from the key), but you're right that the alternating tool-enabled/reflection-step pattern is the case that actually matters in practice, not just a theoretical hash-collision.
Agreed on normalizing
"auto"vs. omittedtool_choicedeliberately rather than letting that fall out of whatever the cache key construction happens to do withNonevs the string"auto". If those two hash differently by accident, we'd have swapped one silent bug for a different, more confusing one (unnecessary cache misses that look like a partial fix).I'll fold your regression test shape into the fix rather than the narrower one I had (same messages/tools/args,
tool_choice="required"thentool_choice="none", assert no cache hit and the second call reaches the client with the right value), plus a case for"auto"vs. omitted to nail down the normalization question explicitly.test(chat-completion-cache): cover forced-tool/none transition and om…
IgorGanapolsky commentedon Jul 24, 2026
Agentic systems are turning into always-on factories: one request fans out into model calls, tool calls, policy checks, and storage.
The hard part is rarely tokens/sec — it is control-plane reliability:
If you share one sanitized failing run (expected vs actual + tool result), I can map which pillar is broken. Architecture only.