[ShawnGarello]

Linked issue: #1216

Purpose of change

The Anthropic integration now sends images and documents returned by a tool to the model, in Java and Python. A tool result holding media becomes one tool_result block whose content keeps the result's text, image and document blocks in order. Before, any media in a tool result failed the request.

Tools can return screenshots, charts or documents the model must read before it continues, and Anthropic accepts them inside tool_result.content. This is the first part of the plan in the issue; the other integrations still reject media in tool results. The fields below cover the adapter change; the docs part adds a tool-result media table to the providers page and updates the Tool Use warning.

Runtime flow

Java: AnthropicChatUtils.convertToAnthropicMessage passes a tool message to convertToolResult. A result of only TextBlocks is sent as content(String), the concatenated text, as before. Otherwise each block becomes a ToolResultBlockParam.Content.Block in order: ofText, ofImage through toImageBlockParam (now shared with user messages), or ofDocument through the existing toDocumentBlockParam. Other media throws.

Python: convert_to_anthropic_message already built the tool_result content from the result's blocks; its media check now accepts tool messages too, so _to_media_block runs for them.

Key decisions

• Tool results use the user-message helpers, so one place decides which media Anthropic accepts.
• Text-only results keep their wire form, so existing requests don't change.
• No dependency change: anthropic-java 2.12.0 has ToolResultBlockParam.Content.Block.ofImage/ofDocument, and the Python adapter sends plain dicts.

Behavioral Semantics

Interaction decisions

Message	Text only	Supported media	Unsupported media	Undecodable Base64 plain text
User	unchanged	unchanged	unchanged	unchanged
Tool result	unchanged	tool_result.content blocks in order	as for a user message, same message	as for a user message
Assistant	unchanged	raises, role reason (new wording)	role reason	role reason

Behavioral contracts

1. A tool result with media is sent as one tool_result block: tool_use_id is the call ID, is_error the result's flag, and content its blocks in order: TextBlock as text, ImageBlock as image, DocumentBlock as document titled by the block's name.
2. Media in a tool result follows the user-message rules: JPEG, PNG, GIF or WebP images by URL or Base64; PDF by URL or Base64; plain text by Base64, sent as decoded UTF-8.
3. Java and Python send the same JSON for a tool result with media.
4. A text-only tool result is sent as before: Java one concatenated string, Python a list of text blocks.
5. Audio, video, other image or document types, and plain text by URL in a tool result raise UnsupportedContentBlockException (Java) / UnsupportedContentBlockError (Python) with the same message as in a user message. The message names no URL or data.
6. Undecodable Base64 plain text in a tool result raises IllegalArgumentException / ValueError, not the unsupported type.
7. Media in an assistant message raises with the reason only user messages and tool results can carry media, not assistant messages.
8. Each tool message becomes its own tool_result block with its own call ID and content, text-only or not.

Failure behavior

Every failure above is raised while the request is built, so nothing is sent; media is never dropped or stringified. Java chat() rethrows UnsupportedContentBlockException unchanged and wraps other build errors in RuntimeException, as for user messages. Media the model rejects comes back as the service's error.

Tests

Contract	Java AnthropicMultimodalTest / Python test_anthropic_multimodal.py
1, 2	testToolResultMediaIsSentInOrder / test_tool_result_media_is_sent_in_order; testErrorToolResultWithMedia / test_error_tool_result_with_media
3	both order tests assert the same full tool_result
4	testTextOnlyToolResultIsUnchanged (exact JSON) / test_text_only_tool_result_is_unchanged
5	testUnsupportedToolResultMediaFailsAsForUserMessages / test_unsupported_tool_result_media_fails_as_for_user_messages
6	testInvalidBase64TextDocumentInToolResultFails / test_invalid_base64_text_document_in_tool_result_fails
7	testMediaInAssistantMessagesFails / test_media_in_assistant_messages_fails
8	testParallelToolResultsKeepTheirCorrelation / test_parallel_tool_results_keep_their_correlation
live	test_anthropic_reads_image_returned_by_tool (test_anthropic_chat_model.py): a tool returns a red PNG; the answer must say "red"

Coverage by risk: each accepted media kind and source appears in a tool result in both languages, and each unsupported kind is compared with its user-message error.

Not verified:

• The live test needs TEST_API_KEY (an Anthropic key). CI doesn't run it: no workflow sets that variable, and secrets aren't passed to fork PRs. It has not been run locally either.
• No test sends a document, or an error result with media, in a tool result to the real API.
• No end-to-end agent job returns media from a tool.

Test runs, test-file changes and implementation invariants

• mvn -B test -pl integrations/chat-models/anthropic: 118 passed (AnthropicMultimodalTest 13).
• pytest -k "not e2e_tests" -m "not integration" flink_agents/integrations/chat_models/anthropic flink_agents/api: 1201 passed, 15 skipped. Whole unit suite: 2877 passed, 16 skipped.
• ./tools/lint.sh --check (ruff 0.11.13, Spotless), ruff format --check and ./tools/check-license.sh pass.
• With the two adapter files reverted, the new tool-media, assistant-reason, unsupported, undecodable and parallel tests fail in both languages, and the text-only tests pass.
• The former assistant-and-tool media test is split: the assistant half is contract 7's test; the tool half is replaced by the tests for contracts 1 and 5. The five unsupported blocks are a shared list used by the user and tool tests.
• The live test calls the connection directly, with TEST_MODEL or the default model.
• Java picks the text-only path with allMatch(TextBlock), so an empty result is still sent as "".
• ToolResultBlock holds only text and media blocks, so no other block type reaches toToolResultContent.
• ToolResponse.error(...) builds a text-only result, so media with is_error comes only from a ToolResultBlock built directly.

API

No signature or dependency change. For a caller who changes nothing: text-only tool results are sent unchanged; supported media in a tool result is now sent instead of raising; unsupported media still raises the same type, now with the user-message reason instead of only user messages can carry media, not tool messages. The assistant reason adds "and tool results".

Documentation

• doc-needed
• doc-not-needed
• doc-included

Was this patch authored or co-authored using generative AI tooling?

• Yes
• No

Generated-by: Claude Code 2.1.309 (Claude Opus 5.5)

[wenjin272]

Thanks for taking this on. LGTM