Skip to content

fix(chatgpt): round-trip encrypted reasoning through /v1/messages - #49076

Merged
mateo-berri merged 16 commits into
BerriAI:mainfrom
Hugo-DDT:cursor/chatgpt-encrypted-reasoning-messages-1723
Nov 7, 2026
Merged

mateo-berri merged 16 commits into
BerriAI:mainfrom
Hugo-DDT:cursor/chatgpt-encrypted-reasoning-messages-1723

Conversation

@Hugo-DDT

@Hugo-DDT Hugo-DDT commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Handwritten 1-3 sentence description

A Claude Code session pointed at the proxy /v1/messages endpoint with a chatgpt/ model never received the encrypted reasoning the backend returned, so every later tool-call turn started reasoning from scratch. This puts that reasoning on the thinking block the client echoes back, and only strips it when the chat request will not be served by the Responses API.

Details

TLDR

Problem this solves:

  • chatgpt/ behind /v1/messages dropped encrypted reasoning, so the next tool-call turn sent none
  • Azure and OpenAI turns bridged to Responses lost it too

How it solves it:

  • every encrypted reasoning item becomes a signed thinking block on /v1/messages replies
  • each streamed summary keeps its text once, with its own item's signature
  • replayed blocks are kept whenever the request ends in Responses, a deployment's reasoning_summary included
  • a turn whose replayed block a backend rejects is sent again once without encrypted blocks
  • /v1/chat/completions replies are unchanged, and streamed signatures reach only /v1/messages clients
Why a retry instead of tagging the block

The block carries no marker of the account that minted it unless the router's opt-in encrypted_content_affinity is on. With it, each block is tagged with its deployment and a block from another deployment is dropped before the call (case 12). Without it, the proxy cannot tell ahead of time whether a backend will accept a block. When the backend answers 400 with invalid_encrypted_content, or says the encrypted content it got could not be verified, and the sent request carried encrypted reasoning, the turn is sent once more without any encrypted blocks. That covers the chat bridge and the openai/ Responses path. The cost is one failed round trip on each turn whose history still holds another account's block, and that turn also loses its own blocks. The rejected attempt logs no failure, is not retried by an SDK call's num_retries, and keeps its rate-limit slot and budget reservation for the turn sent again (case 17). When an SDK call's num_retries retries a 429 or 500 on that turn, the retry stops at the rejection, so the turn is still sent again without the block. The retry logs one warning with provider, model, status, and error code, never the block, and names the affinity setting. Tagging every block by default would change the block format clients store

User Flow

Before: Claude Code on a chatgpt/ model loses its reasoning after every tool call

  1. They point Claude Code at the proxy with ANTHROPIC_BASE_URL and pick a chatgpt/ model, or an Azure gpt-5.4+ deployment
  2. They ask for a task that needs reasoning before a tool call, and Claude Code sends POST https://litellm-domain/v1/messages
  3. The reply streams only a tool_use block, so ctrl+o shows no thinking line
  4. The next POST https://litellm-domain/v1/messages carries the tool result but no reasoning, so the model starts over

After: the same session keeps its reasoning across tool calls

  1. Same first two steps as Before
  2. The reply streams a redacted_thinking block before the tool_use block, and ctrl+o shows Thinking…
  3. Claude Code sends that block back with the tool result, so the model keeps its reasoning
  4. If they switch to a model on another account mid-session, later turns still answer 200

Relevant issues

Fixes #49046

Pre-Submission checklist

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • Every automated review on my PR's latest commit has finished, and I have addressed or replied to each of its findings before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live runs on the ChatGPT backend, Azure gpt-5.4-nano, and Gemini, with Claude Code v2.1.305 in tmux and 2 uvicorn workers per leg sharing one Redis. Output and scripts are in this gist

Proxy setup, the same on both legs

A copy of litellm/proxy/dev_config.yaml with router_settings and general_settings.coordination_redis pointed at the leg's Redis, plus these models

model_list:
  - model_name: gpt-sol
    litellm_params:
      model: chatgpt/gpt-6.1-sol
  - model_name: az-nano-resp
    litellm_params:
      model: azure/gpt-5.4-nano
      api_base: os.environ/AZ_BASE
      api_key: os.environ/AZ_KEY
      api_version: "2025-04-01-preview"
    model_info:
      mode: responses
  - model_name: az-nano
    litellm_params:
      model: azure/gpt-5.4-nano
      api_base: os.environ/AZ_BASE
      api_key: os.environ/AZ_KEY
      api_version: "2025-04-01-preview"
  - model_name: oa-nano-v1
    litellm_params:
      model: openai/gpt-5.4-nano
      api_base: os.environ/AZ_OPENAI_V1_BASE
      api_key: os.environ/AZ_KEY
  - model_name: az-nano-summary
    litellm_params:
      model: azure/gpt-5.4-nano
      api_base: os.environ/AZ_BASE
      api_key: os.environ/AZ_KEY
      api_version: "2025-04-01-preview"
      reasoning_summary: auto
  - model_name: gemini-control
    litellm_params:
      model: gemini/gemini-3.8-flash
      api_key: os.environ/GEMINI_API_KEY
export CHATGPT_TOKEN_DIR=<dir holding a signed-in ChatGPT auth.json>   # also AZ_BASE, AZ_KEY, GEMINI_API_KEY, LITELLM_MASTER_KEY
export AZ_OPENAI_V1_BASE=$AZ_BASE/openai/v1
.venv/bin/python litellm/proxy/proxy_cli.py --config config.yaml --port <free port> --num_workers 2 --detailed_debug --reload --use_v2_migration_resolver --timeout_worker_healthcheck 60

OpenAI's org had no credits, so oa-nano-v1 sends the openai/ provider to the Azure OpenAI v1 endpoint. Cases 4, 8, 9, and 16 send the After leg's turn 2 files to both legs, and cases 10, 12, and 14 build turn 2 from each leg's own turn 1. The base is unchanged since the earlier rounds, so the Before output for cases 1 to 16 stands, and case 17 is new this round. Case 17 mints a key with max_parallel_requests: 1 on each leg

Case 12's third proxy runs the After commit with the same config plus

router_settings:
  enable_pre_call_checks: true
  optional_pre_call_checks:
    - encrypted_content_affinity
Case Before e3387b6fc5f After ae89db72d29
1. Claude Code TUI no thinking block redacted_thinking kept
2-3. Turn 1, plain and streamed 0 of 3 blocks each 3 and 1 of 3 blocks
4. Turn 2 input plus cache 1738 either way 1751 with, 1738 without
5. Gemini control HTTP 200 HTTP 200
6. Streamed summary signature 0 chars on both 1940 and 1812 chars
7. Azure with tools, turn 1 tool_use only redacted_thinking
8. Azure turn 2 input 118 either way 143 with, 118 without
9. Account switch 200, then 400 200, then 200
10. OpenAI thinking turn 2 input 118 either way 145 with, 118 without
11. Chat completions stream 0 thinking blocks 0 thinking blocks
12. Two turns after a switch 200, 200 200, 200
13, 15. Claude Code switch to Azure, openai/ 0 errors, no thinking 0 errors, thinking kept
14. Switch to openai/ Responses 200, 200, no thinking 200, 200, thinking kept
16. Summary deployment turn 2 input 118 either way 148 with, 118 without
17. Retried turn, limit-1 key 1 success row, B 429 1 success row, B 429

Before (e3387b6)

Case 1: Claude Code, reasoning before a tool call

  1. Start Claude Code in a folder holding 1039.txt (alpha) and done.txt (beta): ANTHROPIC_BASE_URL=http://127.0.0.1:<port> ANTHROPIC_AUTH_TOKEN=$LITELLM_MASTER_KEY claude --model gpt-sol
  2. Type Without tools, find the smallest prime above 1000 whose digits sum to 13. Then read the file named after that number plus .txt with the Read tool, then read done.txt with the Read tool, and reply with the two words joined by a dash.
  3. It answers alpha-beta, ctrl+o shows no thinking line, and the session file has no thinking block
202be87c ['tool_use'] input_tokens 11364 cache_read 0
bc9372bd ['tool_use'] input_tokens 195 cache_read 11264
94bacc39 ['text'] input_tokens 290 cache_read 11264

before
before, ctrl+o transcript

Case 2: turn 1, non-streaming
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn1.json | jq -c '[.content[] | .type] , .usage'
["tool_use"]
{"input_tokens":1703,"output_tokens":30}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn1.json | jq -c '[.content[] | .type] , .usage'
["tool_use"]
{"input_tokens":167,"output_tokens":32,"cache_read_input_tokens":1536}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn1.json | jq -c '[.content[] | .type] , .usage'
["tool_use"]
{"input_tokens":167,"output_tokens":32,"cache_read_input_tokens":1536}
Case 3: turn 1, streaming
$ jq -c '. + {stream: true}' turn1.json | curl -sN $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- | grep '^data: ' | sed 's/^data: //' | jq -c 'select(.type == "content_block_start") | .content_block.type' | paste -sd, -
"tool_use"
$ jq -c '. + {stream: true}' turn1.json | curl -sN $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- | grep '^data: ' | sed 's/^data: //' | jq -c 'select(.type == "content_block_start") | .content_block.type' | paste -sd, -
"tool_use"
$ jq -c '. + {stream: true}' turn1.json | curl -sN $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- | grep '^data: ' | sed 's/^data: //' | jq -c 'select(.type == "content_block_start") | .content_block.type' | paste -sd, -
"tool_use","text"
Case 4: turn 2 replay with and without the block
$ jq -c '[.messages[1].content[] | .type]' turn2_with_block.json turn2_without_block.json
["redacted_thinking","tool_use"]
["tool_use"]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn2_with_block.json | jq -c '{input_tokens: .usage.input_tokens, cache_read: .usage.cache_read_input_tokens, blocks: [.content[] | .type]}'
{"input_tokens":1738,"cache_read":null,"blocks":["text"]}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn2_without_block.json | jq -c '{input_tokens: .usage.input_tokens, cache_read: .usage.cache_read_input_tokens, blocks: [.content[] | .type]}'
{"input_tokens":202,"cache_read":1536,"blocks":["text"]}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn2_with_block.json | jq -c '{input_tokens: .usage.input_tokens, cache_read: .usage.cache_read_input_tokens, blocks: [.content[] | .type]}'
{"input_tokens":1738,"cache_read":null,"blocks":["text"]}
Case 5: Gemini control
$ jq -c '.model = "gemini-control"' turn2_with_block.json | curl -s -o $OUT/control.json -w 'HTTP %{http_code}\n' $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @-
HTTP 200
$ jq -c '{type, blocks: [.content[]? | .type], error: .error.message}' $OUT/control.json
{"type":"message","blocks":["thinking","text"],"error":null}
Case 6: streamed reasoning summary on chatgpt/ and Azure Responses
$ jq -c '.model = "gpt-sol" | .thinking.summary = "detailed" | .stream = true' turn1.json | curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- | grep "^data: " | sed "s/^data: //" | jq -sc "{blocks: [.[] | select(.type == \"content_block_start\") | .content_block.type], thinking: ([.[] | select(.type == \"content_block_delta\") | .delta.thinking // empty] | join(\"\")), signature_chars: ([.[] | select(.type == \"content_block_delta\") | .delta.signature // empty] | join(\"\") | length), error: ([.[] | select(.type == \"error\") | .error.message] | first)}"
{"blocks":["tool_use"],"thinking":"","signature_chars":0,"error":null}
$ jq -c '.model = "az-nano-resp" | .thinking.summary = "detailed" | .stream = true' turn1.json | curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- | grep "^data: " | sed "s/^data: //" | jq -sc "{blocks: [.[] | select(.type == \"content_block_start\") | .content_block.type], thinking: ([.[] | select(.type == \"content_block_delta\") | .delta.thinking // empty] | join(\"\")), signature_chars: ([.[] | select(.type == \"content_block_delta\") | .delta.signature // empty] | join(\"\") | length), error: ([.[] | select(.type == \"error\") | .error.message] | first)}"
{"blocks":["thinking","tool_use","text"],"thinking":"**Deciding between Madrid and Oslo's temperatures**\n\nI need to figure out which city is colder, and I think it’ll probably be Oslo. So, the next step is to call the weather tool specifically for Oslo. This should give me the accurate temperature info I need. It’s important to check before I finalize anything, just to be sure I’m providing the right details! Let's get that weather data now!","signature_chars":0,"error":null}
$ jq -c '.model = "gpt-sol" | .thinking.summary = "detailed" | .stream = true' turn1.json | curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- | grep "^data: " | sed "s/^data: //" | jq -sc "{blocks: [.[] | select(.type == \"content_block_start\") | .content_block.type], thinking_chars: ([.[] | select(.type == \"content_block_delta\") | .delta.thinking // empty] | join(\"\") | length), signature_chars: ([.[] | select(.type == \"content_block_delta\") | .delta.signature // empty] | join(\"\") | length), error: ([.[] | select(.type == \"error\") | .error.message] | first)}"
{"blocks":["thinking","tool_use","text"],"thinking_chars":23,"signature_chars":0,"error":null}
Case 7: Azure gpt-5.4-nano with tools, turn 1
$ jq -c '.model = "az-nano"' turn1.json | curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- | jq -c '[.content[] | .type]'
["tool_use"]
$ jq -c '.model = "az-nano" | .stream = true' turn1.json | curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- | grep "^data: " | sed "s/^data: //" | jq -sc "{blocks: [.[] | select(.type == \"content_block_start\") | .content_block.type], thinking: ([.[] | select(.type == \"content_block_delta\") | .delta.thinking // empty] | join(\"\")), signature_chars: ([.[] | select(.type == \"content_block_delta\") | .delta.signature // empty] | join(\"\") | length), error: ([.[] | select(.type == \"error\") | .error.message] | first)}"
{"blocks":["tool_use","text"],"thinking":"","signature_chars":0,"error":null}
Case 8: Azure turn 2 replay with and without the block
$ jq -c '[.messages[1].content[] | .type]' az_turn2_with_block.json az_turn2_without_block.json
["redacted_thinking","tool_use"]
["tool_use"]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @az_turn2_with_block.json | jq -c '{input_tokens: .usage.input_tokens, cache_read: .usage.cache_read_input_tokens, blocks: [.content[] | .type]}'
{"input_tokens":118,"cache_read":null,"blocks":["text"]}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @az_turn2_without_block.json | jq -c '{input_tokens: .usage.input_tokens, cache_read: .usage.cache_read_input_tokens, blocks: [.content[] | .type]}'
{"input_tokens":118,"cache_read":null,"blocks":["text"]}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @az_turn2_with_block.json | jq -c '{input_tokens: .usage.input_tokens, cache_read: .usage.cache_read_input_tokens, blocks: [.content[] | .type]}'
{"input_tokens":118,"cache_read":null,"blocks":["text"]}
Case 9: switching accounts mid-session
$ jq -c '.model = "gpt-sol"' az_turn2_with_block.json | curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -o $OUT/switch_az_to_sol.json -w 'HTTP %{http_code}\n' -d @-; jq -c '{blocks: [.content[]? | .type], input_tokens: .usage.input_tokens, error: .error.message}' $OUT/switch_az_to_sol.json
HTTP 200
{"blocks":["text"],"input_tokens":202,"error":null}
$ jq -c '.model = "az-nano-resp"' sol_turn2_with_block.json | curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -o $OUT/switch_sol_to_az.json -w 'HTTP %{http_code}\n' -d @-; jq -c '{blocks: [.content[]? | .type], input_tokens: .usage.input_tokens, error: .error.message}' $OUT/switch_sol_to_az.json
HTTP 400
{"blocks":[],"input_tokens":null,"error":"litellm.BadRequestError: AzureException - {\n  \"error\": {\n    \"message\": \"The encrypted content gAAA...OA== could not be verified. Reason: Encrypted content could not be decrypted or parsed.\",\n    \"type\": \"invalid_request_error\",\n    \"param\": null,\n    \"code\": \"invalid_encrypted_content\"\n  }\n}\n\nThis error occurs when load balancing Responses API across deployments with different API keys.\n   Encrypted content is tied to the organization that created it and cannot be decrypted by other organizations.\n\n   Solution: Enable 'encrypted_content_affinity' to route follow-up requests to the correct deployment:\n\n   router_settings:\n     enable_pre_call_checks: true\n     optional_pre_call_checks:\n       - encrypted_content_affinity\n\n   Learn more: https://docs.litellm.ai/docs/response_api#encrypted-content-affinity-multi-region-load-balancing\n\nLiteLLM: model group 'az-nano-resp' failed with the error above. No fallback was attempted."}
Case 10: OpenAI thinking turn 2 with stop_sequences
$ jq -c '.model = "oa-nano-v1" | .thinking = {type: "enabled", budget_tokens: 2048} | del(.output_config)' turn1.json | curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- > $OUT/oa_turn1_response.json; jq -c '[.content[] | .type]' $OUT/oa_turn1_response.json
["redacted_thinking","tool_use"]
$ jq --slurpfile r $OUT/oa_turn1_response.json '.model = "oa-nano-v1" | .thinking = {type: "enabled", budget_tokens: 2048} | del(.output_config) | .stop_sequences = ["</answer>"] | .messages += [{role: "assistant", content: $r[0].content}, {role: "user", content: [{type: "tool_result", tool_use_id: ($r[0].content[] | select(.type == "tool_use") | .id), content: "-7 C, snow"}]}]' turn1.json > oa_turn2_with_block_base.json
$ jq '.messages[1].content |= map(select(.type | test("thinking") | not))' oa_turn2_with_block_base.json > oa_turn2_without_block_base.json
$ jq -c '[.messages[1].content[] | .type]' oa_turn2_with_block_base.json oa_turn2_without_block_base.json
["redacted_thinking","tool_use"]
["tool_use"]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @oa_turn2_with_block_base.json | jq -c '{input_tokens: .usage.input_tokens, blocks: [.content[]? | .type], error: .error.message}'
{"input_tokens":118,"blocks":["text"],"error":null}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @oa_turn2_without_block_base.json | jq -c '{input_tokens: .usage.input_tokens, blocks: [.content[]? | .type], error: .error.message}'
{"input_tokens":118,"blocks":["text"],"error":null}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @oa_turn2_with_block_base.json | jq -c '{input_tokens: .usage.input_tokens, blocks: [.content[]? | .type], error: .error.message}'
{"input_tokens":118,"blocks":["text"],"error":null}
Case 11: /v1/chat/completions streaming with tools
$ jq -c '.model = "az-nano"' chat_stream_body.json | curl -sN $PROXY/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'content-type: application/json' -d @- > $OUT/chat_stream_az-nano.sse; grep -c '^data: {' $OUT/chat_stream_az-nano.sse; grep -o '"thinking_blocks"' $OUT/chat_stream_az-nano.sse | wc -l; grep -c 'litellm_encrypted_reasoning' $OUT/chat_stream_az-nano.sse; grep -o '"name":"get_weather"' $OUT/chat_stream_az-nano.sse | head -1
9
       0
0
"name":"get_weather"
$ jq -c '.model = "gpt-sol"' chat_stream_body.json | curl -sN $PROXY/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'content-type: application/json' -d @- > $OUT/chat_stream_gpt-sol.sse; grep -c '^data: {' $OUT/chat_stream_gpt-sol.sse; grep -o '"thinking_blocks"' $OUT/chat_stream_gpt-sol.sse | wc -l; grep -c 'litellm_encrypted_reasoning' $OUT/chat_stream_gpt-sol.sse; grep -o '"name":"get_weather"' $OUT/chat_stream_gpt-sol.sse | head -1
9
       0
0
"name":"get_weather"
Case 12: turn 1 on chatgpt/, turns 2 and 3 on Azure
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn1.json > $OUT/c12_turn1.json; jq -c '[.content[] | .type]' $OUT/c12_turn1.json
["tool_use"]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn1.json > $OUT/c12_turn1.json; jq -c '[.content[] | .type]' $OUT/c12_turn1.json
["tool_use"]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn1.json > $OUT/c12_turn1.json; jq -c '[.content[] | .type]' $OUT/c12_turn1.json
["tool_use"]
$ jq --slurpfile r $OUT/c12_turn1.json '.model = "az-nano-resp" | .messages += [{role: "assistant", content: $r[0].content}, {role: "user", content: [{type: "tool_result", tool_use_id: ($r[0].content[] | select(.type == "tool_use") | .id), content: "-7 C, snow"}]}]' turn1.json > $OUT/c12_turn2_req.json
$ jq -c '[.messages[1].content[] | .type]' $OUT/c12_turn2_req.json
["tool_use"]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @$OUT/c12_turn2_req.json -o $OUT/c12_turn2.json -w 'HTTP %{http_code} in %{time_total}s\n'; jq -c '{blocks: [.content[]? | .type], input_tokens: .usage.input_tokens, error: .error.message}' $OUT/c12_turn2.json
HTTP 200 in 3.570568s
{"blocks":["text"],"input_tokens":118,"error":null}
$ jq --slurpfile r $OUT/c12_turn2.json '.messages += [{role: "assistant", content: $r[0].content}, {role: "user", content: "And would Madrid be warmer? Answer in one sentence."}]' $OUT/c12_turn2_req.json > $OUT/c12_turn3_req.json
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @$OUT/c12_turn3_req.json -o $OUT/c12_turn3.json -w 'HTTP %{http_code} in %{time_total}s\n'; jq -c '{blocks: [.content[]? | .type], input_tokens: .usage.input_tokens, error: .error.message}' $OUT/c12_turn3.json
HTTP 200 in 3.352200s
{"blocks":["text"],"input_tokens":170,"error":null}
Case 13: Claude Code switching models mid-session
$ ANTHROPIC_BASE_URL=http://127.0.0.1:26888 ANTHROPIC_AUTH_TOKEN=$LITELLM_MASTER_KEY claude --model gpt-sol   # in qa/projsw-base, isolated config dir
ready: input prompt is up
typed: Without tools, find the smallest prime above 1000 whose digits sum to 13. Then read the file named after that number plus .txt with the Read tool, then read done.txt with the Read tool, and reply with the two words joined by a dash.
claude-tui-send: submitted with C-m
typed: /model az-nano-resp
claude-tui-send: submitted with literal \r
typed: Read third.txt with the Read tool and reply with its word in uppercase.
claude-tui-send: submitted with C-m
typed: Read done.txt again with the Read tool and reply with its word spelled backwards.
claude-tui-send: submitted with C-m
qa/out-base/claude-code-switch-base.png
## Claude Code's own session record: model and content blocks of each assistant turn
gpt-sol ['tool_use'] []
gpt-sol ['text'] ['']
gpt-sol ['tool_use'] []
gpt-sol ['text'] ['']
gpt-sol ['text'] ['alpha-beta']
az-nano-resp ['tool_use'] []
az-nano-resp ['text'] ['']
az-nano-resp ['text'] ['GAMMA']
az-nano-resp ['tool_use'] []
az-nano-resp ['text'] ['']
az-nano-resp ['text'] ['ateb']
0
pane text: qa/out-base/pane-switch.txt, screenshot: qa/out-base/claude-code-switch-base.png

before, model switch

Case 14: turn 1 on chatgpt/, turns 2 and 3 on openai/ (Responses path)
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn1.json > $OUT/c14_turn1.json; jq -c '[.content[] | .type]' $OUT/c14_turn1.json
["tool_use"]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn1.json > $OUT/c14_turn1.json; jq -c '[.content[] | .type]' $OUT/c14_turn1.json
["tool_use"]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn1.json > $OUT/c14_turn1.json; jq -c '[.content[] | .type]' $OUT/c14_turn1.json
["tool_use"]
$ jq --slurpfile r $OUT/c14_turn1.json '.model = "oa-nano-v1" | .messages += [{role: "assistant", content: $r[0].content}, {role: "user", content: [{type: "tool_result", tool_use_id: ($r[0].content[] | select(.type == "tool_use") | .id), content: "-7 C, snow"}]}]' turn1.json > $OUT/c14_turn2_req.json
$ jq -c '[.messages[1].content[] | .type]' $OUT/c14_turn2_req.json
["tool_use"]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @$OUT/c14_turn2_req.json -o $OUT/c14_turn2.json -w 'HTTP %{http_code} in %{time_total}s\n'; jq -c '{blocks: [.content[]? | .type], input_tokens: .usage.input_tokens, error: .error.message}' $OUT/c14_turn2.json
HTTP 200 in 1.953178s
{"blocks":["text"],"input_tokens":118,"error":null}
$ jq '.stream = true' $OUT/c14_turn2_req.json > $OUT/c14_turn2_stream_req.json
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @$OUT/c14_turn2_stream_req.json -o $OUT/c14_turn2_stream.txt -w 'HTTP %{http_code} in %{time_total}s\n'; grep '^event:' $OUT/c14_turn2_stream.txt | sort | uniq -c | tr -s ' ' | tr '\n' ';'; echo; grep -o '"error":{[^}]*}' $OUT/c14_turn2_stream.txt | head -1
HTTP 200 in 1.882531s
 23 event: content_block_delta; 1 event: content_block_start; 1 event: content_block_stop; 1 event: message_delta; 1 event: message_start; 1 event: message_stop;
$ jq --slurpfile r $OUT/c14_turn2.json '.messages += [{role: "assistant", content: $r[0].content}, {role: "user", content: "And would Madrid be warmer? Answer in one sentence."}]' $OUT/c14_turn2_req.json > $OUT/c14_turn3_req.json
$ jq -c '[.messages[] | select(.role == "assistant") | [.content[]? | .type]]' $OUT/c14_turn3_req.json
[["tool_use"],["text"]]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @$OUT/c14_turn3_req.json -o $OUT/c14_turn3.json -w 'HTTP %{http_code} in %{time_total}s\n'; jq -c '{blocks: [.content[]? | .type], input_tokens: .usage.input_tokens, error: .error.message}' $OUT/c14_turn3.json
HTTP 200 in 2.524070s
{"blocks":["text"],"input_tokens":179,"error":null}
Case 15: Claude Code switching to an openai/ model mid-session
$ ANTHROPIC_BASE_URL=http://127.0.0.1:26888 ANTHROPIC_AUTH_TOKEN=$LITELLM_MASTER_KEY claude --model gpt-sol   # in qa/projsw-oa-nano-v1-base, isolated config dir
ready: input prompt is up
typed: Without tools, find the smallest prime above 1000 whose digits sum to 13. Then read the file named after that number plus .txt with the Read tool, then read done.txt with the Read tool, and reply with the two words joined by a dash.
claude-tui-send: submitted with C-m
typed: /model oa-nano-v1
claude-tui-send: submitted with literal \r
typed: Read third.txt with the Read tool and reply with its word in uppercase.
claude-tui-send: submitted with C-m
typed: Read done.txt again with the Read tool and reply with its word spelled backwards.
claude-tui-send: submitted with C-m
qa/out-base/claude-code-switch-oa-nano-v1-base.png
## Claude Code's own session record: model and content blocks of each assistant turn
gpt-sol ['tool_use'] []
gpt-sol ['text'] ['']
gpt-sol ['tool_use'] []
gpt-sol ['text'] ['']
gpt-sol ['text'] ['alpha-beta']
oa-nano-v1 ['redacted_thinking'] []
oa-nano-v1 ['tool_use'] []
oa-nano-v1 ['text'] ['GAMMA']
oa-nano-v1 ['redacted_thinking'] []
oa-nano-v1 ['tool_use'] []
oa-nano-v1 ['redacted_thinking'] []
oa-nano-v1 ['text'] ['ateb']
0
pane text: qa/out-base/pane-switch-oa-nano-v1.txt, screenshot: qa/out-base/claude-code-switch-oa-nano-v1-base.png

before, switch to openai/

Case 16: a deployment with reasoning_summary set, turn 2 with and without the block
$ jq -c '.model = "az-nano-summary" | .thinking = {type: "enabled", budget_tokens: 2048} | del(.output_config)' turn1.json | curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- | jq -c '[.content[] | .type]'
["thinking","tool_use"]
## turn 2 bodies are the After leg's fixtures
$ jq -c '[.messages[1].content[] | .type]' sum_turn2_with_block.json sum_turn2_without_block.json
["thinking","tool_use"]
["tool_use"]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @sum_turn2_with_block.json | jq -c '{input_tokens: .usage.input_tokens, blocks: [.content[]? | .type], error: .error.message}'
{"input_tokens":118,"blocks":["text"],"error":null}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @sum_turn2_without_block.json | jq -c '{input_tokens: .usage.input_tokens, blocks: [.content[]? | .type], error: .error.message}'
{"input_tokens":118,"blocks":["text"],"error":null}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @sum_turn2_with_block.json | jq -c '{input_tokens: .usage.input_tokens, blocks: [.content[]? | .type], error: .error.message}'
{"input_tokens":118,"blocks":["text"],"error":null}
Case 17: a turn sent again on a key limited to 1 parallel request

The base drops the block before the call and sends this turn once. It logged 1 success row, and B answered 429 whenever A was still in flight. In run 2 at 1.4 s, A had finished in 1.39 s, so B answered 200 (output, script)

After (ae89db7)

Cases 1, 4 to 7, 10, 11, and 16 ran on cadfe1d

Last updated: 917bfe3. QA ran on ae89db7, and on cadfe1d for cases 1, 4 to 7, 10, 11, and 16. Changes since QA: the first try of a turn sent again is no longer logged or counted, 3d53f0047d1 only adds types to two test callbacks, ba83c7205c4 stops an SDK call's own num_retries from sending that first try again, and aee2e638467 stops that retry from resending a rejected block after a 429 or 500 and keeps the global litellm.num_retries, 66f329f5408 stops those retries from starting a cancelled call again, and 917bfe3c27d only types a test parameter. Proof remains valid because no carried case sends a turn again, and every case goes through the proxy's router, which never takes the SDK retry path. Verification: the new handler tests, including four that fail without ba83c7205c4 and six that fail without aee2e638467, a cancellation test that fails without 66f329f5408, and 5862 related unit tests pass at aee2e63, and the adapter and retry helper tests pass at 917bfe3. /live-pr-risk passed at 66f329f, and 917bfe3c27d changes only that test

Case 1: Claude Code, reasoning before a tool call

  1. Same folder, command, and prompt as Before
  2. It answers alpha-beta, ctrl+o shows Thinking… before the first Read, and the session file keeps redacted_thinking
a3f086c0 ['redacted_thinking'] input_tokens 11364 cache_read 0
a3f086c0 ['tool_use'] input_tokens 11364 cache_read 0
a3f086c0 ['text'] input_tokens 11364 cache_read 0
21c2020d ['tool_use'] input_tokens 142 cache_read 11520
9a378c96 ['text'] input_tokens 109 cache_read 11648

after
after, transcript

Fixtures: turn 2 bodies from this proxy's own turn 1
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn1.json > $OUT/turn1_response.json; jq -c '[.content[] | .type]' $OUT/turn1_response.json
["tool_use"]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn1.json > $OUT/turn1_response.json; jq -c '[.content[] | .type]' $OUT/turn1_response.json
["redacted_thinking","tool_use"]
$ jq --slurpfile r $OUT/turn1_response.json '.messages += [{role: "assistant", content: $r[0].content}, {role: "user", content: [{type: "tool_result", tool_use_id: ($r[0].content[] | select(.type == "tool_use") | .id), content: "-7 C, snow"}]}]' turn1.json > turn2_with_block.json
$ jq '.messages[1].content |= map(select(.type | test("thinking") | not))' turn2_with_block.json > turn2_without_block.json
$ jq -c '.model = "az-nano"' turn1.json | curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- > $OUT/az_turn1_response.json; jq -c '[.content[] | .type]' $OUT/az_turn1_response.json
["redacted_thinking","tool_use"]
$ jq --slurpfile r $OUT/az_turn1_response.json '.model = "az-nano" | .messages += [{role: "assistant", content: $r[0].content}, {role: "user", content: [{type: "tool_result", tool_use_id: ($r[0].content[] | select(.type == "tool_use") | .id), content: "-7 C, snow"}]}]' turn1.json > az_turn2_with_block.json

$ jq '.messages[1].content |= map(select(.type | test("thinking") | not))' az_turn2_with_block.json > az_turn2_without_block.json

$ jq -c '.model = "gpt-sol"' turn1.json | curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- > $OUT/sol_turn1_response.json; jq -c '[.content[] | .type]' $OUT/sol_turn1_response.json
["redacted_thinking","tool_use"]
$ jq --slurpfile r $OUT/sol_turn1_response.json '.model = "gpt-sol" | .messages += [{role: "assistant", content: $r[0].content}, {role: "user", content: [{type: "tool_result", tool_use_id: ($r[0].content[] | select(.type == "tool_use") | .id), content: "-7 C, snow"}]}]' turn1.json > sol_turn2_with_block.json

$ jq '.messages[1].content |= map(select(.type | test("thinking") | not))' sol_turn2_with_block.json > sol_turn2_without_block.json

These ran in earlier rounds and built the turn 2 files, which went unchanged to both legs

Case 2: turn 1, non-streaming
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn1.json | jq -c '[.content[] | .type] , .usage'
["redacted_thinking","tool_use"]
{"input_tokens":167,"output_tokens":32,"cache_read_input_tokens":1536}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn1.json | jq -c '[.content[] | .type] , .usage'
["redacted_thinking","tool_use"]
{"input_tokens":167,"output_tokens":32,"cache_read_input_tokens":1536}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn1.json | jq -c '[.content[] | .type] , .usage'
["redacted_thinking","tool_use"]
{"input_tokens":167,"output_tokens":31,"cache_read_input_tokens":1536}
Case 3: turn 1, streaming
$ jq -c '. + {stream: true}' turn1.json | curl -sN $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- | grep '^data: ' | sed 's/^data: //' | jq -c 'select(.type == "content_block_start") | .content_block.type' | paste -sd, -
"tool_use"
$ jq -c '. + {stream: true}' turn1.json | curl -sN $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- | grep '^data: ' | sed 's/^data: //' | jq -c 'select(.type == "content_block_start") | .content_block.type' | paste -sd, -
"redacted_thinking","tool_use","text"
$ jq -c '. + {stream: true}' turn1.json | curl -sN $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- | grep '^data: ' | sed 's/^data: //' | jq -c 'select(.type == "content_block_start") | .content_block.type' | paste -sd, -
"tool_use"
Case 4: turn 2 replay with and without the block
$ jq -c '[.messages[1].content[] | .type]' turn2_with_block.json turn2_without_block.json
["redacted_thinking","tool_use"]
["tool_use"]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn2_with_block.json | jq -c '{input_tokens: .usage.input_tokens, cache_read: .usage.cache_read_input_tokens, blocks: [.content[] | .type]}'
{"input_tokens":1751,"cache_read":null,"blocks":["text"]}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn2_without_block.json | jq -c '{input_tokens: .usage.input_tokens, cache_read: .usage.cache_read_input_tokens, blocks: [.content[] | .type]}'
{"input_tokens":202,"cache_read":1536,"blocks":["text"]}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn2_with_block.json | jq -c '{input_tokens: .usage.input_tokens, cache_read: .usage.cache_read_input_tokens, blocks: [.content[] | .type]}'
{"input_tokens":215,"cache_read":1536,"blocks":["text"]}
Case 5: Gemini control
$ jq -c '.model = "gemini-control"' turn2_with_block.json | curl -s -o $OUT/control.json -w 'HTTP %{http_code}\n' $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @-
HTTP 200
$ jq -c '{type, blocks: [.content[]? | .type], error: .error.message}' $OUT/control.json
{"type":"message","blocks":["thinking","text"],"error":null}
Case 6: streamed reasoning summary on chatgpt/ and Azure Responses

Signatures of 1940 and 1812 chars (chatgpt/, Azure)

Case 7: Azure gpt-5.4-nano with tools, turn 1
$ jq -c '.model = "az-nano"' turn1.json | curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- | jq -c '[.content[] | .type]'
["redacted_thinking","tool_use"]
$ jq -c '.model = "az-nano" | .stream = true' turn1.json | curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- | grep "^data: " | sed "s/^data: //" | jq -sc "{blocks: [.[] | select(.type == \"content_block_start\") | .content_block.type], thinking: ([.[] | select(.type == \"content_block_delta\") | .delta.thinking // empty] | join(\"\")), signature_chars: ([.[] | select(.type == \"content_block_delta\") | .delta.signature // empty] | join(\"\") | length), error: ([.[] | select(.type == \"error\") | .error.message] | first)}"
{"blocks":["redacted_thinking","tool_use","text"],"thinking":"","signature_chars":0,"error":null}
Case 8: Azure turn 2 replay with and without the block
$ jq -c '[.messages[1].content[] | .type]' az_turn2_with_block.json az_turn2_without_block.json
["redacted_thinking","tool_use"]
["tool_use"]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @az_turn2_with_block.json | jq -c '{input_tokens: .usage.input_tokens, cache_read: .usage.cache_read_input_tokens, blocks: [.content[] | .type]}'
{"input_tokens":143,"cache_read":null,"blocks":["text"]}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @az_turn2_without_block.json | jq -c '{input_tokens: .usage.input_tokens, cache_read: .usage.cache_read_input_tokens, blocks: [.content[] | .type]}'
{"input_tokens":118,"cache_read":null,"blocks":["text"]}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @az_turn2_with_block.json | jq -c '{input_tokens: .usage.input_tokens, cache_read: .usage.cache_read_input_tokens, blocks: [.content[] | .type]}'
{"input_tokens":143,"cache_read":null,"blocks":["text"]}
Case 9: switching accounts mid-session
$ jq -c '.model = "gpt-sol"' az_turn2_with_block.json | curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -o $OUT/switch_az_to_sol.json -w 'HTTP %{http_code}\n' -d @-; jq -c '{blocks: [.content[]? | .type], input_tokens: .usage.input_tokens, error: .error.message}' $OUT/switch_az_to_sol.json
HTTP 200
{"blocks":["text"],"input_tokens":202,"error":null}
$ jq -c '.model = "az-nano-resp"' sol_turn2_with_block.json | curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -o $OUT/switch_sol_to_az.json -w 'HTTP %{http_code}\n' -d @-; jq -c '{blocks: [.content[]? | .type], input_tokens: .usage.input_tokens, error: .error.message}' $OUT/switch_sol_to_az.json
HTTP 200
{"blocks":["text"],"input_tokens":118,"error":null}
Case 10: OpenAI thinking turn 2 with stop_sequences
$ jq -c '.model = "oa-nano-v1" | .thinking = {type: "enabled", budget_tokens: 2048} | del(.output_config)' turn1.json | curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- > $OUT/oa_turn1_response.json; jq -c '[.content[] | .type]' $OUT/oa_turn1_response.json
["redacted_thinking","tool_use"]
$ jq --slurpfile r $OUT/oa_turn1_response.json '.model = "oa-nano-v1" | .thinking = {type: "enabled", budget_tokens: 2048} | del(.output_config) | .stop_sequences = ["</answer>"] | .messages += [{role: "assistant", content: $r[0].content}, {role: "user", content: [{type: "tool_result", tool_use_id: ($r[0].content[] | select(.type == "tool_use") | .id), content: "-7 C, snow"}]}]' turn1.json > oa_turn2_with_block_head.json
$ jq '.messages[1].content |= map(select(.type | test("thinking") | not))' oa_turn2_with_block_head.json > oa_turn2_without_block_head.json
$ jq -c '[.messages[1].content[] | .type]' oa_turn2_with_block_head.json oa_turn2_without_block_head.json
["redacted_thinking","tool_use"]
["tool_use"]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @oa_turn2_with_block_head.json | jq -c '{input_tokens: .usage.input_tokens, blocks: [.content[]? | .type], error: .error.message}'
{"input_tokens":145,"blocks":["text"],"error":null}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @oa_turn2_without_block_head.json | jq -c '{input_tokens: .usage.input_tokens, blocks: [.content[]? | .type], error: .error.message}'
{"input_tokens":118,"blocks":["text"],"error":null}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @oa_turn2_with_block_head.json | jq -c '{input_tokens: .usage.input_tokens, blocks: [.content[]? | .type], error: .error.message}'
{"input_tokens":145,"blocks":["text"],"error":null}
Case 11: /v1/chat/completions streaming with tools
$ jq -c '.model = "az-nano"' chat_stream_body.json | curl -sN $PROXY/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'content-type: application/json' -d @- > $OUT/chat_stream_az-nano.sse; grep -c '^data: {' $OUT/chat_stream_az-nano.sse; grep -o '"thinking_blocks"' $OUT/chat_stream_az-nano.sse | wc -l; grep -c 'litellm_encrypted_reasoning' $OUT/chat_stream_az-nano.sse; grep -o '"name":"get_weather"' $OUT/chat_stream_az-nano.sse | head -1
9
       0
0
"name":"get_weather"
$ jq -c '.model = "gpt-sol"' chat_stream_body.json | curl -sN $PROXY/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'content-type: application/json' -d @- > $OUT/chat_stream_gpt-sol.sse; grep -c '^data: {' $OUT/chat_stream_gpt-sol.sse; grep -o '"thinking_blocks"' $OUT/chat_stream_gpt-sol.sse | wc -l; grep -c 'litellm_encrypted_reasoning' $OUT/chat_stream_gpt-sol.sse; grep -o '"name":"get_weather"' $OUT/chat_stream_gpt-sol.sse | head -1
9
       0
0
"name":"get_weather"
Case 12: turn 1 on chatgpt/, turns 2 and 3 on Azure, without and with affinity

Turns 2 and 3 answered 200 and 200 without affinity (output) and with it (output), and only the affinity proxy tagged the turn 1 block with its deployment (output)

Case 13: Claude Code switching models mid-session
$ ANTHROPIC_BASE_URL=http://127.0.0.1:23779 ANTHROPIC_AUTH_TOKEN=$LITELLM_MASTER_KEY claude --model gpt-sol   # in qa/projsw-az-nano-resp-head, isolated config dir
ready: input prompt is up
typed: Without tools, find the smallest prime above 1000 whose digits sum to 13. Then read the file named after that number plus .txt with the Read tool, then read done.txt with the Read tool, and reply with the two words joined by a dash.
claude-tui-send: submitted with C-m
typed: /model az-nano-resp
claude-tui-send: submitted with literal \r
typed: Read third.txt with the Read tool and reply with its word in uppercase.
claude-tui-send: submitted with C-m
typed: Read done.txt again with the Read tool and reply with its word spelled backwards.
claude-tui-send: submitted with C-m
qa/out-head/claude-code-switch-az-nano-resp-head.png
## Claude Code's own session record: model and content blocks of each assistant turn
gpt-sol ['redacted_thinking'] []
gpt-sol ['tool_use'] []
gpt-sol ['text'] ['']
gpt-sol ['tool_use'] []
gpt-sol ['text'] ['alpha-beta']
az-nano-resp ['redacted_thinking'] []
az-nano-resp ['tool_use'] []
az-nano-resp ['text'] ['']
az-nano-resp ['text'] ['GAMMA']
az-nano-resp ['tool_use'] []
az-nano-resp ['text'] ['ateb']
0
pane text: qa/out-head/pane-switch-az-nano-resp.txt, screenshot: qa/out-head/claude-code-switch-az-nano-resp-head.png

after, model switch

Case 14: turn 1 on chatgpt/, turns 2 and 3 on openai/ (Responses path)
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn1.json > $OUT/c14_turn1.json; jq -c '[.content[] | .type]' $OUT/c14_turn1.json
["tool_use"]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @turn1.json > $OUT/c14_turn1.json; jq -c '[.content[] | .type]' $OUT/c14_turn1.json
["redacted_thinking","tool_use"]
$ jq --slurpfile r $OUT/c14_turn1.json '.model = "oa-nano-v1" | .messages += [{role: "assistant", content: $r[0].content}, {role: "user", content: [{type: "tool_result", tool_use_id: ($r[0].content[] | select(.type == "tool_use") | .id), content: "-7 C, snow"}]}]' turn1.json > $OUT/c14_turn2_req.json
$ jq -c '[.messages[1].content[] | .type]' $OUT/c14_turn2_req.json
["redacted_thinking","tool_use"]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @$OUT/c14_turn2_req.json -o $OUT/c14_turn2.json -w 'HTTP %{http_code} in %{time_total}s\n'; jq -c '{blocks: [.content[]? | .type], input_tokens: .usage.input_tokens, error: .error.message}' $OUT/c14_turn2.json
HTTP 200 in 1.919053s
{"blocks":["text"],"input_tokens":118,"error":null}
$ jq '.stream = true' $OUT/c14_turn2_req.json > $OUT/c14_turn2_stream_req.json
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @$OUT/c14_turn2_stream_req.json -o $OUT/c14_turn2_stream.txt -w 'HTTP %{http_code} in %{time_total}s\n'; grep '^event:' $OUT/c14_turn2_stream.txt | sort | uniq -c | tr -s ' ' | tr '\n' ';'; echo; grep -o '"error":{[^}]*}' $OUT/c14_turn2_stream.txt | head -1
HTTP 200 in 2.828153s
 24 event: content_block_delta; 1 event: content_block_start; 1 event: content_block_stop; 1 event: message_delta; 1 event: message_start; 1 event: message_stop;
$ jq --slurpfile r $OUT/c14_turn2.json '.messages += [{role: "assistant", content: $r[0].content}, {role: "user", content: "And would Madrid be warmer? Answer in one sentence."}]' $OUT/c14_turn2_req.json > $OUT/c14_turn3_req.json
$ jq -c '[.messages[] | select(.role == "assistant") | [.content[]? | .type]]' $OUT/c14_turn3_req.json
[["redacted_thinking","tool_use"],["text"]]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @$OUT/c14_turn3_req.json -o $OUT/c14_turn3.json -w 'HTTP %{http_code} in %{time_total}s\n'; jq -c '{blocks: [.content[]? | .type], input_tokens: .usage.input_tokens, error: .error.message}' $OUT/c14_turn3.json
HTTP 200 in 2.772860s
{"blocks":["redacted_thinking","text"],"input_tokens":165,"error":null}

The same steps at the earlier tip e021b42087e answered 400 invalid_encrypted_content on turn 2, its stream, and turn 3 (output)

Case 15: Claude Code switching to an openai/ model mid-session
$ ANTHROPIC_BASE_URL=http://127.0.0.1:23779 ANTHROPIC_AUTH_TOKEN=$LITELLM_MASTER_KEY claude --model gpt-sol   # in qa/projsw-oa-nano-v1-head, isolated config dir
ready: input prompt is up
typed: Without tools, find the smallest prime above 1000 whose digits sum to 13. Then read the file named after that number plus .txt with the Read tool, then read done.txt with the Read tool, and reply with the two words joined by a dash.
claude-tui-send: submitted with C-m
typed: /model oa-nano-v1
claude-tui-send: submitted with literal \r
typed: Read third.txt with the Read tool and reply with its word in uppercase.
claude-tui-send: submitted with C-m
typed: Read done.txt again with the Read tool and reply with its word spelled backwards.
claude-tui-send: submitted with C-m
qa/out-head/claude-code-switch-oa-nano-v1-head.png
## Claude Code's own session record: model and content blocks of each assistant turn
gpt-sol ['redacted_thinking'] []
gpt-sol ['tool_use'] []
gpt-sol ['text'] ['']
gpt-sol ['tool_use'] []
gpt-sol ['text'] ['alpha-beta']
oa-nano-v1 ['redacted_thinking'] []
oa-nano-v1 ['tool_use'] []
oa-nano-v1 ['text'] ['GAMMA']
oa-nano-v1 ['redacted_thinking'] []
oa-nano-v1 ['tool_use'] []
oa-nano-v1 ['redacted_thinking'] []
oa-nano-v1 ['text'] ['ateb']
0
pane text: qa/out-head/pane-switch-oa-nano-v1.txt, screenshot: qa/out-head/claude-code-switch-oa-nano-v1-head.png

after, switch to openai/

Case 16: a deployment with reasoning_summary set, turn 2 with and without the block
$ jq -c '.model = "az-nano-summary" | .thinking = {type: "enabled", budget_tokens: 2048} | del(.output_config)' turn1.json | curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- > $OUT/sum_turn1_response.json; jq -c '[.content[] | .type]' $OUT/sum_turn1_response.json
["thinking","tool_use"]
$ jq --slurpfile r $OUT/sum_turn1_response.json '.model = "az-nano-summary" | .thinking = {type: "enabled", budget_tokens: 2048} | del(.output_config) | .messages += [{role: "assistant", content: $r[0].content}, {role: "user", content: [{type: "tool_result", tool_use_id: ($r[0].content[] | select(.type == "tool_use") | .id), content: "-7 C, snow"}]}]' turn1.json > sum_turn2_with_block.json
$ jq '.messages[1].content |= map(select(.type | test("thinking") | not))' sum_turn2_with_block.json > sum_turn2_without_block.json
$ jq -c '[.messages[1].content[] | .type]' sum_turn2_with_block.json sum_turn2_without_block.json
["thinking","tool_use"]
["tool_use"]
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @sum_turn2_with_block.json | jq -c '{input_tokens: .usage.input_tokens, blocks: [.content[]? | .type], error: .error.message}'
{"input_tokens":148,"blocks":["text"],"error":null}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @sum_turn2_without_block.json | jq -c '{input_tokens: .usage.input_tokens, blocks: [.content[]? | .type], error: .error.message}'
{"input_tokens":118,"blocks":["text"],"error":null}
$ curl -s $PROXY/v1/messages -H "x-api-key: $LITELLM_MASTER_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @sum_turn2_with_block.json | jq -c '{input_tokens: .usage.input_tokens, blocks: [.content[]? | .type], error: .error.message}'
{"input_tokens":148,"blocks":["text"],"error":null}

The request sets no summary, so only the deployment's reasoning_summary: auto sends this chat route to Responses

Case 17: a turn sent again on a key limited to 1 parallel request
$ jq -c '{model, assistant_blocks: [.messages[] | select(.role == "assistant") | [.content[]? | .type]]}' c17_retried_turn.json
{"model":"az-nano","assistant_blocks":[["redacted_thinking","tool_use"]]}
$ curl -s -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'content-type: application/json' $PROXY/key/generate -d "{\"key_alias\": \"$ALIAS\", \"max_parallel_requests\": 1}" > $OUT/c17_key.json; jq -c '{key_alias, max_parallel_requests}' $OUT/c17_key.json
{"key_alias":"c17-head-1794064888","max_parallel_requests":1}
### One turn, then the key's spend logs
$ curl -s -m 120 $PROXY/v1/messages -H "x-api-key: $KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @c17_retried_turn.json -o $OUT/c17_turn.json -w 'HTTP %{http_code} in %{time_total}s\n'; jq -c '{blocks: [.content[]? | .type], error: .error.message}' $OUT/c17_turn.json
HTTP 200 in 2.645148s
{"blocks":["text"],"error":null}
$ curl -s -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'content-type: application/json' -G $PROXY/spend/logs/v2 --data-urlencode "key_alias=$ALIAS" --data-urlencode "start_date=$DAY 00:00:00" --data-urlencode "end_date=$NEXT 00:00:00" > $OUT/c17_spend.json; jq -c '{total, rows: [.data[] | {status, model_group: .model_group, error: (.metadata.error_information.error_code // null)}]}' $OUT/c17_spend.json
{"total":1,"rows":[{"status":"success","model_group":"az-nano","error":null}]}
### A second turn on the same key while the first is in flight
$ curl -s -m 120 $PROXY/v1/messages -H "x-api-key: $KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @c17_retried_turn.json -o $OUT/c17_race_A.json -w 'A HTTP %{http_code} in %{time_total}s' &
$ sleep $DELAY; curl -s -m 120 $PROXY/v1/messages -H "x-api-key: $KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @c17_plain_turn.json -o $OUT/c17_race_B.json -w 'B HTTP %{http_code} in %{time_total}s'
run 1, DELAY=0.8: A HTTP 200 in 2.252614s, B HTTP 429 in 0.019771s, B error: Rate limit exceeded for api_key: <key hash>. L
run 1, DELAY=1.4: A HTTP 200 in 4.189240s, B HTTP 429 in 0.074468s, B error: Rate limit exceeded for api_key: <key hash>. L
run 2, DELAY=0.8: A HTTP 200 in 2.846076s, B HTTP 429 in 0.014210s, B error: Rate limit exceeded for api_key: <key hash>. L
run 2, DELAY=1.4: A HTTP 200 in 1.722469s, B HTTP 429 in 0.016533s, B error: Rate limit exceeded for api_key: <key hash>. L

At the earlier tip cadfe1d1798, B answered 200 while A was still in flight in 2 of 4 runs (output)

Observations

  • Some tries got no reasoning on both legs, a backend choice

Type

Bug Fix

Caveats (if any)

Medium

  • Without encrypted_content_affinity, a turn whose history holds another account's block is sent twice and loses its own blocks. Affinity drops the block first (case 12)

Low

  • A Responses turn with only web search still drops replayed blocks, though xAI sent none live
  • A reasoning item with no encrypted_content still becomes no thinking block. Its summary streams as reasoning_content
  • Several reasoning items in one non-streamed reply are only unit tested
  • Deployment failure hooks still see the rejected first try

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
Open in Web Open in Cursor 
Bugbot summary

[!NOTE]
Medium Risk
Touches request translation, streaming, and retry/logging across Anthropic adapters and Responses bridges; behavior changes are scoped to messages clients but mis-routing or retry edge cases could affect multi-deployment sessions.

Reviewed by Cursor Bugbot for commit 917bfe3. Bugbot is set up for automated code reviews on this repo. Configure here.

Copilot AI balanced review requested due to automatic review settings October 29, 2026 13:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@CLAassistant

CLAassistant commented Nov 5, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

The chat bridge stored encrypted reasoning on reasoning_items, the Anthropic
adapter never read that field, stream chunks that only carried thinking
blocks were dropped, and the messages adapter stripped the signed blocks
before a chatgpt request could turn them back into reasoning items.

Co-authored-by: DDT <1786035110@qq.com>
@cursor
cursor Bot force-pushed the cursor/chatgpt-encrypted-reasoning-messages-1723 branch from f7b52a5 to ea9e29b Compare November 6, 2026 09:54
…1723

Resolve the messages adapter conflict with the Azure Responses replay
change by routing both through chat_completion_ends_in_responses_api,
which now also reads the deployment's mode. Drop duplicate type-only
imports and type two helpers so the lint gates stay flat.
@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

@greptile-apps

greptile-apps Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The latest change appears safe to merge and addresses the remaining numbered finding.

Summary

This PR carries encrypted reasoning through /v1/messages replies and follow-up requests.

  • Anthropic Messages replies carry signed encrypted reasoning into later turns.
  • Rejected encrypted reasoning gets one retry without the block.

Reviews (12) · Last reviewed commit: "test(anthropic): type the retry helper p..." · Reviewed by Greptile

Comment thread litellm/responses/bridge_check.py Resolved
@codspeed

codspeed Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

1 warning

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 43 untouched benchmarks


Comparing Hugo-DDT:cursor/chatgpt-encrypted-reasoning-messages-1723 (917bfe3) with main (268bc03)

Open in CodSpeed

@codecov

codecov Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

@veria-ai

veria-ai Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 2 · PR risk: 0/10

mateo-berri and others added 3 commits November 6, 2026 14:49
# Conflicts:
#	litellm/llms/anthropic/messages/adapters/transformation.py
# Conflicts:
#	litellm/completion_extras/litellm_responses_transformation/transformation.py
Quote the retry helper return type so importing litellm no longer evaluates a string alias with |. The warning keeps provider, model, status, and error code, and leaves the ciphertext out.
@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

mateo-berri added a commit that referenced this pull request Nov 7, 2026
The retry warning keeps provider, model, status, and error code, and leaves the ciphertext out.

Backport of the third #49076 commit to lts/1.108.x. Cherry-picked from c62c92d.

Co-authored-by: DDT <141296842+Hugo-DDT@users.noreply.github.com>
…eb search turns, and stream signatures only to Messages clients

The keep check now knows when the handler moves an OpenAI thinking turn to Responses, and when web_search_options sends an xAI turn there.

The Responses-to-chat stream iterator adds the signed reasoning chunk only when the caller is /v1/messages, so /v1/chat/completions streams stay unchanged.

The reasoning retry fires only when the translated request still carries encrypted reasoning.
@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

…m, and read the Messages marker before the bridge call
@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

…ted reasoning a backend rejects

The openai/ Responses path on /v1/messages had no retry, so a chatgpt/ block left in history answered 400 on every later openai/ turn. That path now sends the turn once more without encrypted reasoning, the same way the chat bridge does. The retry matches only invalid_encrypted_content or a "could not be verified" message naming the content, so the exception mapper's own hint no longer triggers it. The xAI web search keep path is backed out, since xAI sends no encrypted reasoning over the bridge.
mateo-berri added a commit that referenced this pull request Nov 7, 2026
…ted reasoning a backend rejects

The openai/ Responses path on /v1/messages had no retry, so a chatgpt/ block left in history answered 400 on every later openai/ turn. That path now sends the turn once more without encrypted reasoning, the same way the chat bridge does. The retry matches only invalid_encrypted_content or a "could not be verified" message naming the content, so the exception mapper's own hint no longer triggers it. The xAI web search keep path is backed out, since xAI sends no encrypted reasoning over the bridge.

Backport of 15e6017 from #49076 to lts/1.108.x, ported by hand onto the pass_through modules.

Co-authored-by: DDT <141296842+Hugo-DDT@users.noreply.github.com>
@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

…lients and read a deployment's reasoning_summary in the keep check
@mateo-berri

Copy link
Copy Markdown
Contributor

bugbot run

mateo-berri added a commit that referenced this pull request Nov 7, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

@mateo-berri

Copy link
Copy Markdown
Contributor

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

…learing the global num_retries

A transient 429 or 500 on a /v1/messages turn that carries a replayed encrypted
block was retried by an SDK call's num_retries loop, which sent the block again
until every retry was rejected. On async calls the loop then raised the first
transient error, so the turn failed instead of being sent again without the
block. The retry loop now stops at that rejection and passes it on, so the turn
is sent again once without encrypted blocks.

A rejected first try on a sync call also cleared the process-wide
litellm.num_retries, so later calls in the process were no longer retried. That
setting now stays as configured.
@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

Comment thread litellm/main.py Resolved
Comment thread litellm/main.py Resolved
The resent-failure retry predicate returned true for BaseException
subclasses such as asyncio.CancelledError, so a cancelled
acompletion_with_retries or aresponses_with_retries call could start
another provider call. Retry only Exception subclasses again, as
tenacity's default did, while still skipping a resent failure.
@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

Comment thread tests/unit/test_main.py Resolved
@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

@mateo-berri

Copy link
Copy Markdown
Contributor

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 917bfe3. Configure here.

@mateo-berri mateo-berri left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for the contribution!

@mateo-berri
mateo-berri merged commit d0c1600 into BerriAI:main Nov 7, 2026
94 checks passed
mateo-berri added a commit that referenced this pull request Nov 9, 2026
…08_x

fix(anthropic): backport #49076 encrypted reasoning round-trip to lts/1.108.x
mateo-berri added a commit to zoveki/litellm that referenced this pull request Nov 9, 2026
The chat bridge stored encrypted reasoning on reasoning_items, the Anthropic
adapter never read that field, stream chunks that only carried thinking
blocks were dropped, and the messages adapter stripped the signed blocks
before a chatgpt request could turn them back into reasoning items.

Backport of BerriAI#49076 to lts/1.108.x. Cherry-picked from ea9e29b (cursor/chatgpt-encrypted-reasoning-messages-1723).
The line keeps the reasoning helpers in common_utils and the adapters under pass_through,
gains resolve_model_and_provider_for_metadata, and streams redacted_thinking as its own block.

Co-authored-by: DDT <141296842+Hugo-DDT@users.noreply.github.com>
Co-authored-by: DDT <1786035110@qq.com>
mateo-berri added a commit to zoveki/litellm that referenced this pull request Nov 9, 2026
… when a backend cannot decrypt it

Backport of the second BerriAI#49076 commit to lts/1.108.x. Cherry-picked from 10c953d.

Adapted to the line: chat_completion_ends_in_responses_api has no deployment_model_info
parameter here, so only tools, reasoning_effort, and api_base are threaded through, and
the moved tests import the adapters from pass_through instead of messages.
pamanseau added a commit to KubeTEE-AI/litellm that referenced this pull request Nov 16, 2026
* fix(anthropic): stop logging rejected encrypted reasoning

The retry warning keeps provider, model, status, and error code, and leaves the ciphertext out.

Backport of the third #49076 commit to lts/1.108.x. Cherry-picked from c62c92d2ad0.

Co-authored-by: DDT <141296842+Hugo-DDT@users.noreply.github.com>

* fix(ollama): keep replayed tool arguments out of a rejected replay's error chain

Backport of #49183 to lts/1.108.x.
Cherry-picked from 0f7003a2aeb84651230b0fcbcb8403e4ed957af8 (fix-ollama-chat-stream-tool-call-index).

* fix(anthropic): start a new streamed block for each reasoning item

The backported Responses bridge tags each reasoning item's thinking block with its output index, but this line's Anthropic stream wrapper only opened a new block when the block type changed, so a second reasoning item of the same type was dropped. This ports the index check from #46863, which main already has.

* fix(anthropic): keep each streamed reasoning summary with its own item

A summary delta carries no item index, so the next item's summary joined the block that already got its signature. A thinking chunk after an indexed signature now opens a new block.

* fix(ollama): keep the repair warning's argument preview unless message logging is off

Backport of #49183 to lts/1.108.x.
Cherry-picked from 5ef6d62cca7e0a9a9e6db2609ca435e9b957de5e (fix-ollama-chat-stream-tool-call-index).
Adapted to the line. The repair warning lives in common_utils.py here.

* fix(anthropic): stream reasoning signatures only to Messages clients, keep xAI web search reasoning, and retry only when the sent request carried it

* fix(anthropic): send openai/ Messages turns again once without encrypted reasoning a backend rejects

The openai/ Responses path on /v1/messages had no retry, so a chatgpt/ block left in history answered 400 on every later openai/ turn. That path now sends the turn once more without encrypted reasoning, the same way the chat bridge does. The retry matches only invalid_encrypted_content or a "could not be verified" message naming the content, so the exception mapper's own hint no longer triggers it. The xAI web search keep path is backed out, since xAI sends no encrypted reasoning over the bridge.

Backport of 15e60170124 from #49076 to lts/1.108.x, ported by hand onto the pass_through modules.

Co-authored-by: DDT <141296842+Hugo-DDT@users.noreply.github.com>

* test(ollama): cover short and boundary-length repair warning previews

Backport of #49183 to lts/1.108.x.
Cherry-picked from 1f71ed377f459792b383bcd2bd1e5e4f4b2137ae (fix-ollama-chat-stream-tool-call-index).
Adapted to the line. The repair warning tests live in test_litellm_core_utils_prompt_templates_common_utils.py here.

* test(utils): drop a duplicate schema key the line already had on lts/1.108.x

The picks touch tests/unit/test_utils.py, so ruff now checks it, and
the line's model info schema lists supports_bedrock_native_structured_output
twice with the same value. The second copy goes

* fix(tool_choice): accept OpenAI allowed_tools on chat completions and flatten it for the Responses bridge (#49043)

Backport of #49043 to lts/1.108.x.
Cherry-picked from 28c80048ed47d53befd819f22b5ac1bb112ec3fd (main). Import conflicts in the Responses bridge and the Azure chat config resolved to the line's imports plus the helper import.

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(tool_choice): accept the Responses API's flat allowed_tools layout on chat completions

Backport of #50467 to lts/1.108.x.
Cherry-picked from e56c060363c741b24579ecc01279c992d2d339d1 (litellm_allowed_tools_flat_tool_choice).

* fix(tool_choice): drop an allowed_tools choice under drop_params and refuse conflicting flat entries

Backport of #50467 to lts/1.108.x.
Cherry-picked from 361aecae72f2a9153a12b3998c90515f679896fd (litellm_allowed_tools_flat_tool_choice). Azure chat config import conflict resolved to the line's imports.

* fix(tool-choice): keep the 400 when LiteLLM runs the tools, report auto after a drop

Backport of #50467 to lts/1.108.x.
Cherry-picked from 5f8f54a6ca0dbd5bfc443b55e4c49e34b6b8176e (litellm_allowed_tools_flat_tool_choice). Conflicts resolved by applying this commit's hunks to the line's simpler Responses bridge handler and responses() bridge branch, and appending its new tests to the line's test files.

* fix(tool-choice): refuse an excluded tool right before LiteLLM runs it

Backport of #50467 to lts/1.108.x.
Cherry-picked from e25a82c3f54ff54178b5ebb98fb1da6a753e1fe2 (litellm_allowed_tools_flat_tool_choice). Conflicts resolved by applying this commit's hunks to the line's code. Test hunks that need main-only test helpers are carried by the line's test adaptation commit.

* fix(tool-choice): drop the choice under drop_params and hand an excluded MCP call back unrun

Backport of #50467 to lts/1.108.x.
Cherry-picked from be0bb7a57e55712476b903a8244802024416bd96 (litellm_allowed_tools_flat_tool_choice). Conflicts resolved by applying this commit's hunks to the line's MCP handlers and responses(). Test hunks that need main-only test helpers are carried by the line's test adaptation commit.

* fix(tool-choice): refuse an allowed_tools entry whose name is not a string

Backport of #50467 to lts/1.108.x.
Cherry-picked from 2ac020c070c7fdd47ec912f5ae334923ec85356c (litellm_allowed_tools_flat_tool_choice). Test hunks that need main-only test helpers are carried by the line's test adaptation commit.

* fix(tool-choice): honor server-level and null-name mcp entries and stop hooks running excluded calls

Backport of #50467 to lts/1.108.x.
Cherry-picked from 939034d8a4bc3a9f7d337471ca97738df933e300 (litellm_allowed_tools_flat_tool_choice). Resolved: tool_precheck.py is not on the line, so server_labels_serving lives in litellm_proxy_mcp_handler.py. The line has no refused-turn path, so a hook turn holding a call the choice excludes answers 400 through refuse_tool_calls_the_choice_excludes. Test hunks that need main-only fixtures move to the line test commit.

* fix(tool-choice): match named mcp entries by server label and let built-in entries allow their interception call

Backport of #50467 to lts/1.108.x.
Cherry-picked from 4755727276c84a50107c3d5aef9d398e7d0d4603 (litellm_allowed_tools_flat_tool_choice). Resolved: kept only this pick's helper and import where the line lacks main's sandbox result types and core_helpers import. Test hunks that need main-only fixtures move to the line test commit.

* fix(tool-choice): leave a malformed allowed_tools choice unrewritten in the interception hooks

Backport of #50467 to lts/1.108.x.
Cherry-picked from cd076fb13ab (litellm_allowed_tools_flat_tool_choice).

* fix(tool-choice): resolve mcp entries like the gateway and keep the choice on every hook path

Backport of #50467 to lts/1.108.x.
Cherry-picked from 6be4f0b9ead (litellm_allowed_tools_flat_tool_choice). Resolved: the line has no tool_precheck.py or agentic_loop_refusal.py, so the MCP label resolver lives in litellm_proxy_mcp_handler.py and mirrors the line's own toolset resolution (admin view and granted toolsets, no access-group fallback), and the line's toolset listing now calls the same resolver. A chat hook turn holding a call the choice excludes answers 400 through the line's AgenticLoopSafetyError path. Main-only code in the conflicted hunks (internal_sub_call, first-round billing, Claude Code web search prefix) is left out. Test hunks that need main-only fixtures move to the line test commit.

* fix(allowed_tools): stop web search and code interpreter follow-ups forcing another call

Backport of #50467 to lts/1.108.x.
Cherry-picked from 478fb95d6b1 (litellm_allowed_tools_flat_tool_choice). Resolved: test imports merged with the line's, and the tests that need main's web search fixtures move to the line test commit.

* fix(openai): refuse an openai chat hook call the allowed_tools choice excludes

The line's OpenAI chat handler runs its own agentic hook loop instead of
the shared one main uses, so the picked exclusion check never reached
it. This checks the response before a hook runs, like the line's HTTP
handler already does after the picks, and answers 400 when the model
called a tool the choice leaves out. Chat calls with an allowed_tools
choice answer 400 on the line today, so no call that succeeds changes.

* test(tool-choice): adapt the picked tests to lts/1.108.x

The picked test files reach a few names the line does not have, so this
ports them to the line's code without bringing those features over.
The MCP response tests stub the line's own proxy_server imports, the
web search tests get main's perplexity_search fixture, and the chat and
Responses hook tests expect what the line does. An excluded chat hook
call answers 400, and native /v1/responses hooks run as before since
the line cannot end a Responses turn early

* fix(allowed_tools): stop the Responses web search follow-up forcing another call

Backport of #50467 to lts/1.108.x.
Cherry-picked from 08eb785ee22 (litellm_allowed_tools_flat_tool_choice). Resolved: adds main's _WEB_SEARCH_FUNCTION_CALL test constant, which came with #38885 and is not on the line, and wraps one long test line.

* fix(responses): send a malformed allowed_tools choice as given on lts/1.108.x

On the line, native /v1/responses with gateway MCP tools passes any
tool_choice through to the provider, with or without drop_params, and
providers that accept a malformed allowed_tools choice answer 200. The
picks answer 400 for it without drop_params and drop it under
drop_params, so this keeps the line's behavior. The choice reaches the
provider unchanged and the gateway enforces nothing for it, while a
choice that parses still gets the picked rewrite and enforcement

* fix(tool-choice): keep an allowed_tools type with a function as a named tool choice on lts/1.108.x

On the line, a tool_choice of type allowed_tools that also carries a
function passes validation and reaches the provider as a named function
choice, so Anthropic and other providers that read function.name force
that tool and answer 200. The picks treat it as a malformed allowed_tools
choice, which answers 400 without drop_params and drops the choice with
it. This keeps the line's behavior, so such a choice is not read as an
allowed_tools choice anywhere and goes on as the named choice it was

* style(tool-choice): format called_tool_names on lts/1.108.x

* fix(tool-choice): keep the caller's allowed_tools choice on agentic follow-up calls on lts/1.108.x

On main the agentic follow-up keeps the caller's allowed_tools choice
under a reserved kwarg that main lists among its loop control keys.
That list does not exist on the line, so the line's acompletion dropped
the key and a follow-up call could run a hook call the caller's choice
leaves out. acompletion now passes the key on when the caller sent it or
sent an allowed_tools choice, and leaves every other call as it was

* fix(types): widen untyped request dicts for the allowed_tools refusal on lts/1.108.x

The line types the hook loop's kwargs and optional_params as plain
dicts, so passing them to the picked refusal helpers added unknown
argument errors that main does not have. They are now cast to
Mapping[str, object] at the call sites, and the loop settings reader
takes a Mapping

The new cast import in the agentic loop would grow the banned-API
count, so the duplicate function-level cast import in
llm_http_handler.py goes, since the module already imports cast

* fix(anthropic): keep every non-streamed reasoning item for Messages clients and read a deployment's reasoning_summary in the keep check

Backport of #49076 to lts/1.108.x.
Ported from cadfe1d1798 (cursor/chatgpt-encrypted-reasoning-messages-1723) onto the line's pass_through/adapters layout.

* fix(anthropic): keep every non-streamed reasoning item for Messages clients on the line's bridge

This line's transform_request rewrites the logging call_type to responses, so the non-streamed reply read the Messages-client check after the rewrite and kept only the last reasoning item. The bridge handler now reads it before the request transform and passes it to transform_response, as the streamed path already does.

* fix(anthropic): log only the outcome of a turn sent again, and keep every reasoning item in Messages logs on the line

A /v1/messages turn whose replayed encrypted block the backend rejects is sent
once more without it. The rejected first attempt no longer fires failure
callbacks or releases its parallel slot and budget reservation, so only the
turn sent again is logged and counted.

Success logs for a Messages client now keep every reasoning item of a
non-streamed turn, matching the reply the client got.

* test(anthropic): type the parameters of the failure-recording test loggers

Backport of 3d53f0047d1 (#49076) to lts/1.108.x.

* test(anthropic): type the success logger and use a fixed token expiry in the bridge tests

* fix(anthropic): skip the SDK num_retries retry for an attempt that is sent again

* fix(anthropic): stop SDK retries from resending a rejected block or clearing the global num_retries

A transient 429 or 500 on a /v1/messages turn that carries a replayed encrypted
block was retried by an SDK call's num_retries loop, which sent the block again
until every retry was rejected. On async calls the loop then raised the first
transient error, so the turn failed instead of being sent again without the
block. The retry loop now stops at that rejection and passes it on, so the turn
is sent again once without encrypted blocks.

A rejected first try on a sync call also cleared the process-wide
litellm.num_retries, so later calls in the process were no longer retried. That
setting now stays as configured.

* fix(anthropic): stop a cancelled SDK retry from starting another call

The resent-failure retry predicate returned true for BaseException
subclasses such as asyncio.CancelledError, so a cancelled
acompletion_with_retries or aresponses_with_retries call could start
another provider call. Retry only Exception subclasses again, as
tenacity's default did, while still skipping a resent failure.

* test(anthropic): type the retry helper parameter in the cancellation test

* test(integration): backport main's Gateway.rows, Scenario.model name and MCP ack fix to lts/1.108.x

Backports test-only changes from main so the line's integration signal stops reporting stale helpers as failures.

Gateway.database_url and Gateway.rows come from #47213 (fc27dfdc7d), and the name keyword on Scenario.model comes from #48917 (ebf84f8649). Only those hunks of tests/integration/_support/client.py are taken. The rest of #47213 gives each owned proxy its own database and Redis, which none of the line's tests need, and the workers field from a later main change is left out.

The MCP progress and cancellation test wraps its upstream in _acknowledge_notifications_after_dispatch, from #49967 (b5952f243a). The helper is already on the line.

* fix(bedrock)!: backport #47429 deployment guardrailConfig on /bedrock pass-through routes to lts/1.108.x

The router-model pass-through dropped a Bedrock deployment's guardrailConfig, so converse, converse-stream, invoke and invoke-with-response-stream reached Bedrock unguarded while /v1/chat/completions on the same deployment was blocked. The signer now sends it the way Bedrock reads it, in the body for Converse and as the X-Amzn-Bedrock-Guardrail* headers for Invoke, replacing any guardrail the caller sent

Adapted to the line, which does not have #47040. The guardrail headers are merged through merge_bedrock_invoke_headers without the request metadata header, and the unit tests drop the request metadata assertions and the identity argument that only #47040 needs

* refactor(bedrock): drop the unused converse URL helper and cover a trailing-slash converse pass-through

* test(e2e): repin the Together reasoning replay backend to MiniMax-M3.1-Flash

MiniMax-M3 reached its 2026-11-06 deprecation date in the cost map, which names MiniMax-M3.1-Flash as its successor

(cherry picked from commit 90f7ab6fe6837152b21617c8cb7dc73a541c42e9)

* fix(sdk): backport #45961 to lts/1.108.x

completion() loaded the MCP gateway handler for any request with tools,
and the handler pulls in mcp 2.x only code, so tool calls failed on
installs with mcp 1.x (google-adk 1.36.1) or no mcp at all

Cherry-picked from 783cf06f31. Only conflict was import context in
litellm/responses/main.py

(cherry picked from commit 783cf06f31e55e8f7ab84c2d7c708b0d90e68001)

* fix(router): serve a team's wildcard deployment before a proxy-wide wildcard (#48490)

Backport of #48490 to lts/1.108.x.
Cherry-picked from 76d001a7c06d5dbf93f9191b247db0ef981b6c66 (main). The line has no wildcard_routed_models helper and no get_models_by_pattern, so only wildcard_route_deployments is changed.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(videos)!: keep model_id in proxy video ids so per-model keys resolve on status/content (#44614)

Backport of #44614 to lts/1.108.x.
Cherry-picked from ba61ecdf570c91a2409560138170ff9ab1fe7ad7 (main). The line has no splits_stream_timeout on _update_kwargs_with_deployment, so only the _video_call_pin wrapping is added around deployment selection. The line has no litellm/llms/base_llm/videos/async_task_utils.py, so its _id_safe_name dedupe is applied to the dashscope video transformation that holds the same helper on this line.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore: bump litellm 1.108.3 -> 1.108.4

litellm 1.108.3 already shipped, so the next release off this line is 1.108.4

* fix(proxy): check s3 URIs inside encoded file ids and on queued chat completions

The s3 media check now decodes model-embedded and unified file ids, since
completion() turns them back into the raw s3 URI they carry. The
/queue/chat/completions endpoint skips common request processing, so it now
runs the same check. The media walk goes level by level instead of
recursing, which the recursive_detector code-quality step requires.

(cherry picked from commit 5b45c632c57fd46b9dda537b694edcda2e0f41d3)

* fix(anthropic-messages): apply additional_drop_params before the unsupported param checks

* fix(proxy): let a key send managed file ids it may read and refuse s3 sources hidden in model_file_id_mapping

The s3 media gate refused every LiteLLM managed file id whose upload sits under the
managed prefix, even when the key's own user or team created it. It now asks the
managed files hook which unified ids the key may read and skips only the objects
those ids embed. A forged or unowned id, or a proxy without the hook, still gets a 403.

The gate also reads the provider file ids in a client-supplied model_file_id_mapping,
so an alias can no longer point a non-admin request at an unlisted or managed object.

Markdown documents (.md, .markdown) now map to Bedrock's md document format, so an
s3 Markdown source without an explicit format no longer fails with an unknown media type.

On lts/1.108.x the gate still walks the body before it reads the key role, so callers that pass a
specced UserAPIKeyAuth mock keep working, and it reads self.data and general_settings directly
since the line has no raw_request_fields.

(cherry picked from commit 37f085c966dd83e84b646eb3acbdfbaef7b6b0db)

* fix(mcp): refuse a hook call the allowed_tools choice excludes on every surface

The line has no terminal-response path outside Anthropic Messages, so the check skipped /v1/responses and streamed hook loops and the excluded call ran. It now raises AgenticLoopToolChoiceError there too

* fix(mcp): scope each MCP entry to its own allowed_tools and keep the allowlist on follow-ups

A label now covers only the tools its own allowed_tools lists, so a second label on the same server no longer widens it. Responses MCP follow-ups, streamed or not, keep an allowed_tools choice in auto mode instead of dropping it. The new Responses tests fake only the upstream MCP listing, so LiteLLM's own discovery runs

* fix(agentic): answer a 400 when an allowed_tools choice excludes a hook call on /v1/responses

The line cannot end a Responses turn early, so the refusal was re-raised as a
plain error and reached clients as a retryable 500. It now answers the same 400
the chat path gives for an excluded call. The stream path still raises the
safety error its iterator catches.

* fix(mcp): match an entry's allowed_tools against the tool name the gateway lists

A server with a display name override lists the display name, and the listing
filter matches allowed_tools against it, but the label scope only checked the
original name, so no spelling worked. The scope now also accepts the listed
name and its unprefixed part, like the listing filter. Tests now cover the
whole gateway and toolset entries and a display-named tool.

* fix(proxy): keep inline tool-result video dropped and let an admitted client AWS identity skip the s3 bucket list

A tool result forwards a video to Bedrock only when it is an s3Location. An inline
base64 video is dropped with a warning, as it was before this change, since Converse
rejects video in a toolResult block

A non-admin key that sends an AWS identity its deployment admits through
configurable_clientside_auth_params (aws_role_name, aws_web_identity_token or
aws_profile_name) skips the bucket list, because Bedrock then reads the object with
that identity. The managed-prefix 403 still applies

The check now skips the body walk when no string starts with s3:// and no file_id or
model_file_id_mapping key is present, which keeps large image bodies cheap

Adapted for lts/1.108.x. The per-frame check on WS /v1/responses is not ported,
because this line has no Responses WebSocket frame guard to hook it into. The gate
keeps this line's admits_s3_bucket allowlist and its order of walking the body
before the admin role check

(cherry picked from commit 6662cb34d768bd68a7dc34ea27c363a1bd7f6578)

* feat(decisions): support Vercel AI Gateway decision models

Backport of #50642 to lts/1.108.x.
Cherry-picked from bdd017e908e2380de83b059b08c9363431227666 (mishdotdev:vercel-ai-gateway-decisions).
Adapted to the line: the line registers decisions configs in its own if-chain and lazy-import tables, and its OpenAI decisions config has no resolve_credential hook, so only resolve_api_key is overridden.

* test(decisions): cover Vercel AI Gateway dispatch through the proxy

Backport of #50642 to lts/1.108.x.
Cherry-picked from dd1057b018f8bce2408627a15ea7e7a65c2e33b1 (mishdotdev:vercel-ai-gateway-decisions).
Adapted to the line: the new test is appended to the line's own copy of the test file, whose tail differs from main's.

* feat(decisions): add Vercel AI Gateway cost map rows for Jev

Backport of #50642 to lts/1.108.x.
Cherry-picked from 3f9539201cef70aa965fc494fbdb3179b6560c1f (mishdotdev:vercel-ai-gateway-decisions).

* fix(decisions): bill Vercel AI Gateway decisions at the charge Vercel reports

Backport of #50642 to lts/1.108.x.
Cherry-picked from 7c40b75f5657ec067f9a6fe2c7c91f8662c81a78 (mishdotdev:vercel-ai-gateway-decisions).
Adapted to the line: its decisions usage is never None, so the None guards are dropped, and the resolve_credential override stays out as in the first pick.

* fix(decisions): bill Vercel gateway charges with margin and BYOK, forward providerOptions

Backport of #50642 to lts/1.108.x.
Cherry-picked from 1f50a7c91f5032cbd350ab71af204245f2c640a7 (mishdotdev:vercel-ai-gateway-decisions).
Adapted to the line: the line has no upstream_usage_cost billing path (main got it in #48953), so the reported charge still goes out as the line's response-cost header and the operator discount and margin do not apply to it there, the same as OpenRouter decisions on the line. The image check is resolved once per request beside the line's box-question check, and the tests read the billed cost the way the line bills it.

* fix(proxy): check s3 media on every Responses WebSocket frame on the LTS line

The managed Responses WebSocket handler calls litellm.aresponses once per
response.create frame, and the HTTP s3 media check only saw the first frame as a raw
JSON string. A non-admin key could send an s3 object from an unlisted bucket, or one
under LiteLLM-managed storage, to Bedrock over WS /v1/responses

The WS route now hands the handler a frame check bound to the connection's key,
general_settings, managed files hook and router. The handler runs it on the params it
is about to forward for every frame, the first one included, and a refused frame gets
an invalid_request_error frame with the 403 message while the socket keeps serving
later frames

* fix(decisions): stop forwarding caller providerOptions to Vercel and send Jev images

Backport of #50642 to lts/1.108.x.
Cherry-picked from c8122bc6a5305fd8ce0bc3c1d534581d4dd1227a (mishdotdev:vercel-ai-gateway-decisions).
Adapted to the line: decisions/call.py goes back to the line's own version, since the line never had the per-model support record main edits here, and the discount and margin test this commit touches on main is not on the line.

* fix(decisions): report provider charges on the cost header and read Vercel's cache split from the serving provider

Backport of #50642 to lts/1.108.x
Cherry-picked from ec26925103f82742866219bd73d19f88cc3de84e (mishdotdev:vercel-ai-gateway-decisions)

The Jev 1.13.0 and 1.14.0 rows record their image probes

Conflicts resolved to main's result. The call.py hunk is dropped because the line already reports provider charges on the response-cost header. The proxy test keeps only its paid call, since the line's proxy drops a zero response-cost header for every provider

* fix(proxy): let a client AWS identity skip the s3 bucket list only when Bedrock signs with it

An admitted aws_role_name, aws_web_identity_token or aws_profile_name now skips the
bucket list only when it decides the credentials get_credentials picks, on every
deployment of the model group. A web identity token without a role and a session
name does not, since get_credentials then ignores it. A profile does not when the
deployment or the environment sets a role, since the role branch runs first. In
those cases Bedrock still reads with the proxy's credentials, so the bucket list
applies

BaseAWSLLM.identity_choosing_params mirrors the branch order of get_credentials, and
both read the same AWS_* environment fallbacks through one helper

Ported to lts/1.108.x, where the model group is read with Router.get_model_list

* fix(mcp): match an entry's allowed_tools only against a server's own prefix, and a tool no server resolves only by its exact name

* fix(proxy): keep the s3 bucket list for non-admin keys that send a client AWS identity

A client AWS identity field does not prove that the caller can read the bucket. A
role or profile name picks credentials the proxy holds, and Bedrock reads the object
with them. So the bucket list now applies to every non-admin s3 media part, whatever
AWS identity fields the request sends. allow_client_side_credentials: true still
turns it off

This removes BaseAWSLLM.identity_choosing_params and the router lookup that only the
skip needed

Ported to lts/1.108.x

* fix(cost): bill transcriptions by the seconds the provider reports

Line port of the usage.seconds billing slice onto lts/1.108.x: normalized_audio_duration_seconds and _get_transcription_usage_duration from #35600 (ecb8ef5ca8), and the provider-seconds-first _transcription_duration_for_cost from #50727 (dfb1aac8b1), with their unit tests. The rest of #35600 stays on main.

* fix(transcription): treat libsndfile's unknown frame count as no duration

Backport of #50842 to lts/1.108.x.
Cherry-picked from e6a0534785 (litellm_transcription_unknown_frame_count).

* refactor(transcription): inline the unknown frame count check

Backport of #50842 to lts/1.108.x.
Cherry-picked from b6d25f2752 (litellm_transcription_unknown_frame_count).

* fix(transcription): read the frame count once to keep the type gate flat

Backport of #50842 to lts/1.108.x.
Cherry-picked from 54dcd9230f (litellm_transcription_unknown_frame_count).

* test(transcription): annotate the new passthrough test's locals as Final

Backport of #50842 to lts/1.108.x.
Cherry-picked from eb0f251b47 (litellm_transcription_unknown_frame_count).

* feat(openai)!: honor per-deployment workload identity on non-chat routes (#39966)

Backport of #39966 to lts/1.108.x.
Cherry-picked from b39eadcb8228e05b76506df78fa2723628494669 (main).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): refuse a tool whose display name only looks like an allowed name, and let the chat follow-up answer

An MCP entry's allowed_tools now matches a tool only by its exact listed name or a spelling routing
accepts for its real name, so a display name that carries the server's own prefix no longer passes for
another tool. The chat MCP follow-up keeps an allowed_tools choice in auto mode, so the model can answer
with text after the tool ran.

* chore: bump litellm 1.108.4 -> 1.108.5

litellm 1.108.4 already shipped, so the next release off this line is 1.108.5

* fix(mcp): match a toolset label's tools by exact native name

A toolset label served any tool whose name a routing spelling of a granted name matched, so a toolset granting the literal name "server-tool" also served the real tool "tool" that only another toolset grants. It now serves a tool only when the toolset grants its exact native name, the same check tool_is_granted uses when the gateway lists tools

* feat(bedrock): search knowledge bases by image and return image hits

vector_stores.search and asearch take a query_image (base64 jpeg, png, gif or webp), and query is optional when it is sent. Bedrock sends it as an IMAGE retrievalQuery, and a caller's own retrievalQuery in extra_body still wins

Bedrock IMAGE hits now come back as image content items instead of being dropped. The media type comes from the source file mime type metadata, then the data URI label, then application/octet-stream

A request with both query and query_image, with neither, or with a malformed image gets a 400. So does query_image on a provider without image search, unless drop_params is on. The proxy accepts query_image in place of query on /v1/vector_stores/{id}/search

Spend logs redact image data when prompts are not stored. Emulated file_search skips image-only hits, RAG rerank maps its indices to the text documents it ranked, and the dashboard tester renders image hits

(cherry picked from commit 55587c70122fc0fa4369cc23886e753a91b47ee0)

* fix(vector-stores): keep image hits to image searches and reject a caller retrievalQuery

Port of the main branch change. Text searches return the same text hits as before, image hits come back only for query_image searches, a caller retrievalQuery answers 400 or drops under drop_params, a file_search tool carrying query_image searches again, and a non-string query next to an image answers 400

On this line the typed views use as_str_mapping and a local list helper, since decoded_json_object and as_object_list are not on lts/1.108.x

(cherry picked from commit 65a7206199c4c51b46084f8417bf7235d33fd378)

* fix(compresr): let the retrieve call run under an allowed_tools choice on lts/1.108.x

The line's CompresrGuardrail did not list compresr_retrieve in server_fulfilled_tool_names, so the allowed_tools exclusion check counted the model's retrieve call as excluded and the request answered 400. Main has declared it since #49227, and this adds the same attribute with a regression test

* fix(usage): restore per-user spend breakdown on the user aggregated route (#49464)

Backport of #49464 to lts/1.108.x. Cherry-picked from 639625d6791e9236cb12e72790fdf7f00fc55323 (main).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style: preserve spacing around the user metadata helper

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(translation): expect encrypted reasoning in message responses (#50576)

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 7617e66acfe1f3f9bb2984646c7da7c2f7d9db1b)

* test(integration): decode video ids with the product decoder in the callback delivery test

Ports only the tests/integration/observability/test_callback_delivery.py hunk of #47600. Video ids on this line now carry a ;deployment_id: suffix after #50693, so splitting on video_id: kept the suffix. The shares_worker_state pytestmark from that hunk's context belongs to a change not on this line and is left out

(cherry picked from commit e78f7b4602c15a5328f6f211c4bc9f3d664e1d50)

* test(integration): let an owned proxy opt into a private database and Redis

Ports the harness part of #47213 (process.py, workers.py, the remaining client.py hunks and its unit tests) so owned_proxy and owned_proxy_process accept state=OwnedState.PRIVATE and boot on a scratch database and their own redis-server, and Scenario cleanup checks read the database the proxy writes to.

Adapted for this line. Main makes private the default and marks 189 test files with shares_worker_state to keep them on the worker's state. This line keeps the worker's state as the default, so no existing test changes behavior and only callers that pass state=OwnedState.PRIVATE go private. The isolated_gateway rewrite and the conftest marker plumbing are left out, and the unit tests cover the shared default plus the explicit private opt-in

(cherry picked from commit fc27dfdc7d9be704a1c5bffb9c954caeae9dc886)

* test(integration): run the body upload disconnect test on a private database

Ports only the tests/integration/observability/test_guardrail_effects.py hunk of #49310. The test asserts the only spend row is its own control call, so on the shared worker database it picked up rows other tests wrote

(cherry picked from commit fbd126e5182040a5ed8b97d54d41b5febc94452b)

* test(compresr): annotate the allowed_tools choice test's locals as Final

* test(integration): give each kill-burst its own prompts so cached control replies cannot stand in for late traffic (#50569)

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 0d0bf8f75f63a9c76ab75d121cdd6b7f498f2607)

* fix: make scheduled key, user, and team spend resets idempotent (#50298)

Backport of #50298 to lts/1.108.x.
Cherry-picked from ac08eb566314a2dec9f73ede0bb8c687118b5608 (main).

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): add execute_raw to the PrismaBatch protocol

Prerequisite for the #50298 backport to lts/1.108.x, which queues guarded raw spend resets on the batch.
Hand-ported from 24a7bf6af728272eca2ba3277a2f3cdfa7e60863 (#48260, main).

* fix(azure): accept explicit v1 Responses bases (#48072) (#48118)

Backport of #48118 to lts/1.108.x.
Hand-ported from 96abce842b (main): the line lacks #46377's api-version resolution helpers.

* fix(azure): send no api-version on Foundry project /openai/v1 Responses bases

Backport of the main fix on branch litellm_azure_responses_project_v1_api_version to lts/1.108.x.
Cherry-picked from 2a8cdecb15 and adapted to the line's get_complete_url.

* fix(vector-stores): redact query images and keep store search params out of image checks

(cherry picked from commit 0574a8e11965c95dac58ffe037064ff43bf6846b)

* fix(mcp): backport #50717 to lts/1.108.x

Cherry-pick of e6db6206d1 with module paths moved to litellm/proxy/_experimental/mcp_server.

* fix(responses): keep unset-effort gpt-5.4/5.5 tool calls on chat completions (#50968)

Backport of #50968 to lts/1.108.x.
Cherry-picked from d47c03175bba41a39c4c3f8a947e28938708eef2 (main).

Line adaptation: the line has no bridge_reasoning_effort helper or deployment-pinned
routing mode, so the bridge-flag tests pass the effort directly and skip the mode pin.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(vertex_ai): bill gemini image edits by token usage (#51038)

Backport of #51038 to lts/1.108.x.
Cherry-picked from 53e1e9b87b36dabed967b17338948535020b7abc (main).

Line adaptation: the line has no map_vertex_image_size or the Gemini image-size tests, so only
the usage test and the token-usage response path come across.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(mcp): backport #51065 to lts/1.108.x

The connect check regression test mocked the credential reader, so a broken
stored grant still passed. It now saves the grant with the real writer into an
in-memory table and lets the connect check read and decode it.

Cherry-picked from 478d9bf5d2 on main, with module paths moved to
_experimental/mcp_server.

* fix(vector-stores): redact images nested in extra_body and keep a store's extra_body retrievalQuery out of searches

(cherry picked from commit c99fd3631b256caf20bdc5eab530226d02be0b80)

* fix(responses): read reasoning_effort from extra_body for the bridge gate

Minimal prerequisite for #50968 on lts/1.108.x: the bridge_reasoning_effort and
str_keyed_mapping helpers from 897dda7920 (#46263, main), so a reasoning_effort sent
in extra_body still bridges gpt-5.4+ function-tool calls now that an unset effort
stays on chat completions.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(vector_stores): keep a caller's image query through store and router params

A deployment's retrievalQuery no longer reaches a routed search, a caller's query fields survive a store's own extra_body, and an extra_body image stops before the provider. file_search forwards rewrite_query, both retrieval query spellings are redacted, and the cost error stops echoing the response when message logging is off

(cherry picked from commit c65634d4ade59d5c06dabacb946b77161aa79365)

* feat(azure_ai): support gpt-image edits on Azure AI Foundry

Foundry hosts gpt-image deployments on the same /openai/deployments/{deployment}/images/edits route as Azure OpenAI, but the azure_ai image-edit factory only knew MAI and FLUX, so gpt-image-2 and the gpt-image-2.5 models raised a ValueError before any request was sent

Reuse AzureImageEditConfig for the request, URL, and multipart handling, and resolve the api_base and credentials with the azure_ai helpers

Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
(cherry picked from commit 4c29e1846a7de81bc986f23058c91816be2e5581)

* fix(azure_ai): default gpt-image edits to an edit-capable api-version

Foundry gpt-image edits now fall back to 2025-04-01-preview when no api_version is set, since the shared Azure default 2025-02-01-preview answers 404 on /images/edits. The config returns merged headers instead of updating the caller's dict, and the tests check the request each gpt-image model sends

(cherry picked from commit 233a0d99f12c939f1050b67324e83aeb0c42dc23)

* fix(azure_ai): ignore the Azure OpenAI api-version env for Foundry gpt-image edits

The proxy CLI always sets AZURE_API_VERSION to 2025-02-01-preview, which
Foundry answers with 404 on image edits. gpt-image edits now read only
litellm_params, AZURE_AI_API_VERSION, and litellm.api_version before the
2025-04-01-preview default, matching the other azure_ai edit configs

(cherry picked from commit c93fd388d4d34e865b426b1d53297f469734fad4)

* fix(azure_ai): route custom Foundry deployments, require a Foundry api_base, and honor caller auth for gpt-image edits

A deployment name the edit factory does not recognize now uses the gpt-image route instead of raising, since Foundry deployment names are free-form. A gpt-image edit with no Foundry api_base raises like the MAI and FLUX configs instead of borrowing AZURE_API_BASE. The shared edit handler passes the call's extra_headers to validate_environment, so a caller's api-key or Authorization header is sent alone

(cherry picked from commit 6be34701e6bc69d43659e5b9068cdd149cf5c9cf)

* fix(azure_ai): keep the shared edit handler unchanged and send caller auth headers like azure_ai chat

(cherry picked from commit 31348e4c5f90c85fbf4e981e73ccb258eea9431e)

* fix(cost): leave image search responses out of the cost error text

(cherry picked from commit b892162eb3dfc7c7d8bf24d54130ee9e794832c3)

* fix(vector_stores): accept line-wrapped base64 in query_image data

Bedrock takes inlineContent wrapped with \n or \r\n line breaks, as plain base64 writes it, so the check ignores only those before decoding and the data goes to Bedrock as sent

(cherry picked from commit c2cad843b07609f3d216a46db6150c186568d595)

* chore: bump litellm 1.108.5 -> 1.108.6

litellm 1.108.5 already shipped, so the next release off this line is 1.108.6

* fix(usage): keep spend when metadata lookups hit database failures (#50253)

* fix(usage): tolerate entity metadata database failures

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(usage): keep spend when metadata lookups hit database failures

Treat transport errors and the query engine crash on a closed connection as database
failures in the key, user and guardrail lookups of the aggregated, keys, users,
model_top_keys and cache_leakage_keys routes, and keep prisma out of module-level imports
so a proxy install without prisma still imports the usage endpoints.

* fix(usage): fall back only on database outages, on every daily activity route

Data-layer Prisma errors such as a rejected query or a missing table return 500 again, a key whose details failed to load reports key_exists null instead of false, and the non-aggregated daily activity routes keep spend when the key or entity metadata lookup hits an outage

* fix(proxy): skip the key metadata lookup when the daily rows carry no keys

* fix(usage): keep the guardrail breakdown fallback for any Prisma error

A missing guardrail usage table or column made the Usage call return 500 after the outage-only narrowing. The guardrail lookup now falls back on outages and on any Prisma error, as it did before this PR

* fix(usage): keep spend when the team, organization, customer, or agent name lookup drops

---------

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 5d66f2d8ff55378ee01c0cb04f3193330dc09e83)

* fix(ci): install tzdata so zoneinfo tests pass on runners without legacy tz aliases (#51057)

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit ebd5a15b22ba012d2f5a5ee654afc93cd51add5e)

* fix(baseten): forward reasoning_effort to Baseten Model API (#40152)

* fix(baseten): forward reasoning_effort to Baseten Model API

Baseten's Model API accepts a top-level reasoning_effort parameter and
validates it server-side (https://docs.baseten.co/inference/model-apis/reasoning),
but BasetenConfig.get_supported_openai_params did not list it, so LiteLLM
silently dropped it when drop_params=True and raised UnsupportedParamsError
otherwise.

Add reasoning_effort to the supported params so map_openai_params forwards
it verbatim, mirroring the hosted_vllm and deepseek providers. Baseten
returns a clear 400 for values outside a model's supported set and ignores
the parameter on non-reasoning models, so verbatim passthrough is safe for
the whole catalog.

[Claude_Code]

* fix(baseten): forward reasoning_effort to Baseten Model API

Baseten's Model API accepts a top-level reasoning_effort parameter and
validates it server-side (https://docs.baseten.co/inference/model-apis/reasoning),
but BasetenConfig.get_supported_openai_params did not list it, so LiteLLM
silently dropped it when drop_params=True and raised UnsupportedParamsError
otherwise.

Add reasoning_effort to the supported params so map_openai_params forwards
it verbatim, mirroring the hosted_vllm and deepseek providers. Baseten
returns a clear 400 for values outside a model's supported set and ignores
the parameter on non-reasoning models, so verbatim passthrough is safe for
the whole catalog.

---------

Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
(cherry picked from commit 13335266f59a822dcafaadf7329c6a938867c265)

* fix(baseten): list current Model APIs instead of retired 2023 deployment IDs (#46945)

* fix(baseten): list current Model APIs instead of retired 2023 deployment IDs

`baseten/*` wildcard expansion and `/v1/models` read a hardcoded set of
three 2023 deployment IDs (Falcon 7B / WizardLM / MosaicML) that no longer
exist, while the model cost map already carries current Baseten entries.

- Populate `baseten_models` from the model cost map, like fireworks_ai.
- Add `BasetenConfig.get_models()` against the Model API `/v1/models`
  endpoint and register it in `ProviderConfigManager.get_provider_model_info`,
  so `check_provider_endpoint: true` returns the live catalog.
- Remove the retired deployment IDs from `constants.py`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(baseten): mark retired Model APIs deprecated and skip them in listings

15 Baseten entries in the model cost map are no longer servable: the Model
API returns 410 "model version ... has been deprecated" (DeepSeek-V3.1
returns 400 deactivated), and none appear in `inference.baseten.co/v1/models`.

- Set `deprecation_date` on those entries (kept for spend tracking).
- Skip entries past their `deprecation_date` when building `baseten_models`,
  so `baseten/*` expansion only lists models that still serve.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(baseten): assert the custom api_base model listing on the returned models

The custom api_base test only inspected the patched client's call args, which trips the
TQ002 mock-echo gate. Serve the custom URL with respx and assert on the models that come
back instead, so a wrong host or a broken parse both fail.

* fix(baseten): drop newly retired models on cost map reload and fake discovery at the HTTP boundary

A cost map refresh only ever added to the Baseten set, so a model that gained a past
deprecation_date after boot stayed listed under baseten/* until restart. Discard it on reload.

Tests now serve the catalog with respx instead of patching our own HTTP client, the
isinstance registration test is gone since the get_valid_models test already proves
discovery end to end, and new test locals carry Final.

---------

Co-authored-by: AlexKer <AlexKer@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
(cherry picked from commit 3eeee3e76118783671bd7c9dd44a192eedf917ad)

* fix(router): route decisions calls only to deployments whose provider supports decisions (#48175)

* fix(router): route decisions calls only to deployments whose provider supports decisions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(router): account for indirect route helper coverage

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(router): pin the chat leg so the mixed-group check is deterministic

Weights pick the Bedrock leg for chat instead of relying on simple-shuffle landing on both legs. The route filter now passes the candidate list through unchanged in type, so the pre-existing selector lines lose their pyright ignores

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 4710c3b30e5c1459dbb11a060ad733ee06f06c18)

* fix(responses): keep every MCP tool round in streaming follow-up input (#47698)

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
(cherry picked from commit f1795e6d447a4e5a3660de0d2fde5e1f44f52929)

* fix(responses): skip failed spend-log turns when rebuilding a session from previous_response_id (#48647)

* fix(responses): skip failed spend-log turns when rebuilding a session from previous_response_id

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(responses): guard the failed-turn status read and annotate the regression test locals

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit df9091deb666f0d689c7df01056374569019c03e)

* fix(responses): reject or drop Responses params the chat completions bridge cannot translate (#48655)

* fix(responses): reject or drop Responses params the chat completions bridge cannot translate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(responses): keep additional_drop_params working for Responses fields the bridge cannot translate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Update litellm/responses/main.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* style(responses): drop the blank line ruff format rejects after the dataclass header

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
(cherry picked from commit e32d8a4d03f62ca0ec123e0a9377f6be32653bc0)

* fix(vector_stores): drop rewrite_query false from the chat file_search tool

False asks for the default search every store already runs, and MongoDB
rejects the field outright, so forwarding it made a MongoDB chat lose the
store's context for a no-op flag. True is still forwarded.

(cherry picked from commit ee74a39a776)

* fix(responses): serialize sequence_number on every bridged streaming event (#44944)

* fix(responses): serialize sequence_number on every bridged streaming event

* test(responses): cover sequence_number on bridged streaming events

* fix(responses): read undeclared fields through dict accessors and number guardrail block continuations

* fix(responses): scope dict reads that see undeclared fields to Responses stream events

Moves the extras-aware __getitem__, get, __contains__ and items from the
shared response base onto a new base for the 37 stream event classes, so
usage, Gemini, Interactions and moderation objects keep reading __dict__
and Gemini responses keep _hidden_params out of .get

* fix(router): renumber Responses stream events only after a fallback takes over

A stream that never falls back now keeps its provider's own sequence numbers, repeats included, so a starting_after resume still matches them. After a fallback the wrapper continues above the highest number the client saw

* fix(guardrails): keep Responses stream numbers unique when LLM Shield flushes held text

The restorer's flush copied the last delta and kept its sequence_number, so a
native stream repeated a number and a bridged stream sent the flush unnumbered.
A flush now takes the done event's slot and later events move up by one, and a
stream the provider never closed numbers its flush after the last event.

Also pins the created_at in the streaming iterator test fixture instead of
reading the clock.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 6f3251c7e31fb6fed73a786becc6c62f2df56ce7)

* fix(decisions): keep the Vercel AI Gateway key off the OpenAI credential fallback on lts/1.108.x

The line gained the Decisions resolve_credential hook in the backport of #39966 after this branch was cut, so the Vercel provider would inherit OpenAI's fallback to OPENAI_API_KEY. Take main's override from #50642, which returns the gateway key as is.

* test(e2e): remove the secret manager suite (#51241)

The secret manager lanes ran the proxy against a Vault dev container and a
Conjur container that we start ourselves, so nothing in them needed a real
third-party service. They have not run a single test since late October,
because the co-located proxy cannot boot in the lane pod, and they were
soft_fail outside the promotion gate, so they only added noise.

The AWS test in the folder was skipped on every lane, since no aws lane
ever existed.

The four coverage registry rows go with the tests, because the registry is
the e2e denominator and this behavior belongs in the integration tier if it
gets covered again. The Vault and CyberArk clients keep their unit tests.

(cherry picked from commit 587902becf7d48d843befca9d7ea359a216e39d7)

* fix(proxy): stop request body fields from conflicting with admin-set deployment fields

Six fields that also configure Azure sign-in on a deployment (tenant_id, client_id, client_secret, azure_username, azure_password, azure_scope) could be set in a request body, where they conflicted with the values an admin set on the deployment

They are now refused at the body root, in extra_body and in litellm_embedding_config unless an admin opted in with allow_client_side_credentials or configurable_clientside_auth_params. Credential fields and user_config deployments also refuse values in the os.environ/ and oidc/ format, which only admin config uses. Deployment management routes are unchanged

Routes where these names are the API's own fields keep accepting them. That covers client_secret on the realtime transcription_sessions routes, client_id and client_secret on the MCP server OAuth form and on /introspect, and all six as metadata tags

Co-authored-by: Oliver Jensen <oliver@berri.ai>
(cherry picked from commit 6ba6b041e517bd424e8fb4d75aa724962abe289d)

* test(proxy): drop the two request body classifications the field ban removed

client_secret and azure_password are no longer accepted in a request body by default, so the registry's list of credential params a client may send no longer includes them

(cherry picked from commit 2b7e860982d0bdd60f7b5f3ec88d60ce871c3c78)

* test(proxy): import Mapping for the backported request body tests

* fix(proxy): check user_config search_tools for admin-format values too

The source change checks user_config search_tools entries as well as model_list and default_litellm_params. This line's walker skips search_tools, so the backport walks them for this check

* fix(vertex_ai): bill gemini image text, thinking and priority tokens at their own rates (#51017)

Backport of #51017 to lts/1.108.x.
Cherry-picked from 461b56b14b2a024ff73e59f56c01db169dfb902f (main).
Conflicts resolved: vertex_gemini_transformation.py import block (the line does not import map_vertex_image_size) and test_image_usage_transformation.py (the line lacked the ValidationError import and the rejects test, both taken from the pick).

* feat(gemini): file-backed batch jobs on /v1/batches with output download and spend tracking (#51032)

Backport of #51032 to lts/1.108.x as the prerequisite of #51279.
Cherry-picked from a0f2562088 (main).
Conflicts resolved: http_handler.py (the line has no _send_streaming_get, so only the anchor_handler_to rename was taken, including the line's own call site), tests/unit/batches/test_batch_utils.py and test_main.py (the line lacks the hosted_vllm batch tests the pick was formatted next to, so only the new gemini tests were taken). Line adaptation: join_api_base and _strip_trailing_slashes_keeping_the_scheme copied from main's url_utils (#47535) because the gemini files transformation needs them and the line never received that helper.

Line adaptation: the three join_api_base call sites validate GeminiModelInfo.get_api_base through TypeAdapter(str), because the line still annotates that helper as str | None while main returns str, so the type gate would otherwise count three new reportArgumentType errors.

* fix(gemini): attribute /gemini batchGenerateContent batches to the calling key (#51279)

Backport of #51279 to lts/1.108.x.
Cherry-picked from 801e139c35af5eca132e8761cb11be19621218c4 (main).
Conflicts resolved: gemini_passthrough_logging_handler.py import block (the line keeps its import re) and test_gemini_passthrough_logging_handler.py, where the line lacks the gemini pass-through pricing probe tests, so only the new batch tests and their asyncio and base64 imports were taken.
Line adaptation: gemini_deployment_id reads deployments through llm_router.get_model_list(model_name=...), the line's name for what main calls llm_router.registry().deployments(model_name=...).

* fix(bedrock): check OpenAI vector store filter values per operator and keep nested groups within the AWS limit

AWS in and notIn take only a non-empty list of strings, so an in or nin list holding a number now goes out as orAll of equals or andAll of notEquals, and a one-item list as a single equals or notEquals

Each OpenAI filter now refuses keys it does not define and checks its value by operator: eq and ne take a scalar, gt, gte, lt and lte take a number, in and nin take a non-empty list. An empty filters object is invalid too. Each of these answers 400 before any AWS call, or is dropped under drop_params, where before it reached AWS and failed there or, for a stray key, was silently ignored

AWS rejects a RetrievalFilter nested more than 2 levels deep, so a group whose member is the same kind of group, written by the caller or made from a numeric in or nin, now takes that member's items. A mix of and and or nested deeper than AWS allows still fails at AWS

_aws_group_members joins the recursion detector's ignore list, since it walks only a validated filter that pydantic-core stops at 255 levels

Backport to lts/1.108.x.
Cherry-picked from b15ac0fe6b and 2232dba0db (litellm_bedrock_vector_store_openai_filters), whose main PR follows as the forward-port.

* fix(azure_ai): send reasoning_effort to Foundry Kimi K3 and keep its reasoning_content

Foundry models whose loaded catalog row declares reasoning_effort_levels now take reasoning_effort as a level string, rounded up to the nearest declared level, and an effort object is unwrapped to its level. Fireworks-hosted FW- models keep reasoning_content in replayed history, so Kimi K3 can read back what it decided in its earlier thinking. The gate reads the catalog as loaded, so levels a router copies onto the shared row for one deployment do not open reasoning_effort for an undeclared sibling

* fix(responses): preserve mixed reasoning and answer deltas on lts

Partial backport of #46754 to lts/1.108.x. Source merge commit a8113b0dd721804c752a6f1952367fb642205224. Preserve the reasoning/answer split only, excluding the unrelated Anthropic compaction changes.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): look up managed file ids only when they can change the s3 refusal

(cherry picked from commit 0043e3020f5258d6542cad2031f70b2928b89101)

* fix(bedrock): check every file id field, skip MCP tool calls, accept registered S3 media types

(cherry picked from commit 2b13898be4959f51255d7a40978cc6515a522a8a)

* fix(proxy): refuse unlisted s3 media on the rag query and adapter routes

(cherry picked from commit 1d85ba2c1de440543a38b1ca58f0636d8c1bb631)

* style(proxy): format the merged collections.abc import

(cherry picked from commit 91bb8946180b52a708944a7c939e337704834d0e)

* fix(proxy): walk tuples when checking s3 media and skip plain number lists

(cherry picked from commit 5d8f25696341df2acf559ada53ef66ff53dd448d)

* fix(proxy): skip numbers when walking a request body for s3 media

A body with large token-id lists and any s3:// string or file_id key made
the full walk build a node per number, taking seconds per request. The walk
and its prefilter now skip int, float, bool and None values.

(cherry picked from commit 57e73a8f7de294abbc90bef55bdd58acb30f24a6)

* test(proxy): check the s3 refusal through the real request setup

(cherry picked from commit 03624164af4ba7dc35112041a201775436ddd4a6)

* test(proxy): exercise real setup for allowed s3 media

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 5f11068f76d77e05519fcc90a7087a2d9ef7fe95)

* fix(proxy): preserve lts allowlist contract after backport

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(responses): preserve explicit media formats on lts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(model-armor): fail closed on remote s3 documents

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit f9ad9f8aad6e9df611d8b4f470aeacee7ac0ff85)

* chore: bump litellm 1.108.6 -> 1.108.7

litellm 1.108.6 already shipped, so the next release off this line is 1.108.7

* feat(azure-ai): support Entra ID auth for Microsoft-Decision-1 on /v1/decisions and /v1/systemone (#46539)

Backport of #46539 to lts/1.108.x.
Cherry-picked from caad4d62045e76c1a8428ab8f316d351ca9a0a7e (main).

* refactor(azure): expose the api_base resolver and the cached Entra token provider

Prerequisite for the backport of #48529 to lts/1.108.x.
Hand-ported from the litellm/llms/azure/common_utils.py hunks of 9cf09c18fe (#46377, BaseAzureLLM.resolve_azure_api_base) and c0d88b46c9 (#47348, cached_azure_ad_token_refresh_provider) on main. The rest of those two commits is not picked.

* fix(azure): add keyless Decisions authentication (#48386) (#48529)

Backport of #48529 to lts/1.108.x.
Cherry-picked from e5757934fd7e1e37ecf566cd712ea66313315ccf (main).

* fix(files): match every spelling of a provider file id in the ownership check

enforce_provider_file_access matched the stored provider file id by exact string, and a Gemini output file
can be addressed as files/<id>, files%2F<id>, <id>/ or its full URL, all of which the Gemini file transformation
sends to the same file. Those spellings found no ownership row and fell through the no-row pass-through path.

canonical_provider_file_id collapses the spellings to the bare id; the ownership check looks up both the
spelling the caller sent and the canonical one, and store_batch_output_file_ownership stores the canonical id.

(cherry picked from commit 85e6a98866910c98d699bbbaa9b0c3af677c2719)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(files): unwrap model-encoded ids and index canonical provider ids in ownership rows

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 97c9152552a80695cedd8eb72544e2b4ddebe640)

* test(files): assert the ownership lookup outcome instead of its query shape

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 07abf10d3ef4fbd4f79ad4c69802f30d5646acb5)

* fix(files): map provider ids back to their unified file by canonical spelling

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 214e17884a61f51ffa8f307bf94e75db2bd4c834)

* chore(files): record the bound on the ownership lookup's in filter

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit d2a5f0799444c7e05295452b5efaed0e1c945318)

* fix(files): reduce any scheme or case wrapping files/<id> the way gemini does before the ownership lookup

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit e2dad6c914d337471ef9d8b6c08db72502b5f16b)

* test(files): assert the provider id lookup covers the requested ids instead of its exact list

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(files): bind the reverse lookup match with a walrus instead of a Final inside the loop

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 32dabac9f8537bc431fb99ef95a11f79a64f8370)

* fix(files): keep non-Gemini storage URLs opaque so an upload name cannot index another key's provider id

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit c9c9d1024816573500b2fda53d4f025e3ad8adaa)

* test(files): judge the provider id resolver by the mapping it returns only

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit b145f34c0b1bed76607aa7c7d6614bcd870b3f22)

* fix(mcp): treat the mcp_all scope as a wildcard any grant covers (#51578)

Datadog answers a request for mcp_all with its granular scopes, so the connect
check never found mcp_all and users looped back to Connect. Reconnecting also
re-asked every granular scope from the old grant, which overflowed the sign-in
cookie. Any non-empty grant now covers mcp_all, and a request for mcp_all does
not re-as…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: chatgpt/ provider drops encrypted reasoning on /v1/messages, so it is never replayed between tool calls

4 participants