[ShawnGarello]

Linked issue: #1216

Purpose of change

The Gemini integration now sends images, audio, video and documents returned by a tool to the model. A tool result holding media becomes one functionResponse whose parts are its media blocks in order; its text stays in response. Before, media in a tool result failed the request.

Tools can return screenshots, recordings or documents the model must read before it continues, and Google documents media in function response parts for Gemini 3 models. This is part of the plan in the issue, after #1331 (Anthropic) and #1364 (OpenAI Responses), whose tool-result media table this docs change edits. The fields below cover the adapter change; the docs part updates the Gemini row of that table and the Gemini section.

Runtime flow

GeminiChatModelConnection.chat converts each non-system message with GeminiChatUtils.convertToContent. For a tool message, each media block of the ToolResultBlock becomes a FunctionResponsePart through convertMedia, and the text blocks are joined into response.result, giving one Part.fromFunctionResponse(name, {"result": text}, parts) in a user turn. User messages call convertMedia with Part::fromUri and Part::fromBytes, tool results with the FunctionResponsePart ones. Media in an assistant message still throws.

Key decisions

• Text stays in response and media goes in parts, without a $ref from response to a part. The API makes $ref optional, and using it would change the response of every result with media. No displayName is set.
• One helper decides URL or Base64 and decodes the data for both paths. It takes the SDK's factory methods, because message parts and function response parts are separate types.
• No model check: aliases, base_url proxies and new model names make one brittle. An older model's rejection comes back as the service's error.
• The tool result's error flag and the function response id stay unsent, as for text-only results.

Behavioral Semantics

Interaction decisions

Message	Text only	Media	Malformed Base64
User	unchanged	unchanged	raises, unchanged
Tool result	unchanged	functionResponse.parts in block order; text in response	raises, as in a user message
Tool result, is_error=true	unchanged, flag not sent	as with is_error=false, flag not sent	as with is_error=false
Assistant	unchanged	raises, role reason (new wording)	raises the same, before decoding

Behavioral contracts

1. A tool result with media is one user turn with one functionResponse: name as for any tool result, response = {"result": <its text blocks joined>} ("" if none), and parts = one part per media block, in block order.
2. A Base64 block becomes inlineData (mimeType = the block's media type, data = the decoded bytes); a URL block becomes fileData (fileUri, mimeType). No displayName is set.
3. Each media block carries the same values as in a user message.
4. All four media kinds are accepted; media types aren't restricted.
5. A text-only tool result is sent as before, byte for byte.
6. Malformed Base64 data in a tool result raises IllegalArgumentException with the same message as in a user message.
7. Media in an assistant message raises UnsupportedContentBlockException with the reason only user messages and tool results can carry media, not assistant messages. The message names no URL or data.
8. A tool result's error flag is not sent, with or without media, and nothing is added to response.
9. Each tool message becomes its own turn with its own function name and response, text-only or not.

Failure behavior

Malformed Base64 raises IllegalArgumentException, and media in an assistant message UnsupportedContentBlockException (a subclass), while the request is built, so nothing is sent; media is never dropped or stringified. chat() rethrows both unwrapped. A media type, URL or model the service rejects comes back as the service's error.

Tests

Contract	GeminiMultimodalTest
1	testToolResultMediaIsSentInOrder (full structure); testMediaOnlyToolResultSendsEmptyText
2	testToolResultMediaIsSentInOrder
3	testToolResultMediaIsSentInOrder (each part equals the same block's user-message part)
4	testToolResultMediaIsSentInOrder (image, audio, video, document)
5	testTextOnlyToolResultIsUnchanged (exact JSON; empty and error results too)
6	testMalformedBase64InToolResultFails; testChatKeepsMediaErrorTypes (through chat())
7	testMediaInAssistantMessagesFails
8	testErrorToolResultWithMediaSendsNoErrorField; testTextOnlyToolResultIsUnchanged
9	testParallelToolResultsKeepTheirCorrelation
live	GeminiToolMediaLiveTest: a tool returns a red PNG; the answer must say "red"

Coverage by risk: every media kind and both sources appear in one tool result, with text between media blocks.

Not verified:

• The live test needs GEMINI_API_KEY. CI doesn't run it: no workflow sets that variable, and secrets aren't passed to fork PRs. It passed against the Gemini Developer API with gemini-3.6-flash.
• URL sources in tool results, models before Gemini 3 and Vertex AI are not tested against the service.
• A Python agent using this Java connection through cross-language resources isn't tested with tool media.
• No end-to-end agent job returns media from a tool.

Test runs, test-file changes and implementation invariants

• mvn -B test -pl integrations/chat-models/gemini: 84 passed (GeminiMultimodalTest 12; GeminiChatModelConnectionTest 65, unchanged; the live class doesn't run without a key).
• mvn spotless:check -pl integrations/chat-models/gemini and ./tools/check-license.sh pass.
• With GeminiChatUtils.java reverted, the new media, media-only, assistant-reason, malformed-Base64, error-flag, parallel and chat() tests fail, and the text-only test passes.
• The former assistant-and-tool media test is split: the assistant half is contract 7's test; the tool half is replaced by the tests for contracts 1 and 6.
• The live test defaults to gemini-3.6-flash, a stable Gemini 3 model with a free tier. GEMINI_MULTIMODAL_MODEL and GEMINI_BASE_URL override the model and endpoint. It passes the first answer back unchanged, so the tool call's thought signature goes with it.
• The SDK serializes every function response with a parts list, so a text-only result keeps its empty "parts":[].
• DataContentBlock's subtypes are text and the four media types, so Gemini cannot send tool result block stays only for a future subtype.

API

No signature or dependency change. For a caller who changes nothing: text-only tool results are sent unchanged; media in a tool result is now sent instead of raising UnsupportedContentBlockException; the reason for media in an assistant message adds "and tool results".

Documentation

• doc-needed
• doc-not-needed
• doc-included

Was this patch authored or co-authored using generative AI tooling?

• Yes
• No

Generated-by: Claude Code 2.1.319 (Claude Opus 5.5)

[wenjin272]

Thanks for adding multimodal tool-result support for Gemini! I left two comments on supported media types and simplifying the user-facing documentation.

[wenjin272]

Could we restrict tool-result media to the supported MIME types (PNG/JPEG/WebP, PDF, and plain text), and explicitly reject audio/video? Gemini accepts audio/video in user messages, but rejects them in functionResponse.parts. Gemini CLI encountered the same 400 errors and added filtering for these types. The tests and documentation should reflect this distinction.

[wenjin272]

Could we simplify the Media in tool results documentation for Gemini, Anthropic, and OpenAI Responses to focus on supported media types, similar to Media input accepted by the integration above? The provider-specific request structures and conversion details seem unnecessary for users.