Repository navigation
[integration][java][python] Reject media blocks in providers that send text only - #1186
Conversation
wenjin272
left a comment
There was a problem hiding this comment.
Thanks for the contribution and for making unsupported media handling explicit across these integrations! I found one remaining path in Java watsonx that bypasses the new check; please see the inline comment.
| Map<String, Object> modelParams, | ||
| Object outputSchema) { | ||
| // Media blocks are not sent yet; fail rather than drop them (#1059). | ||
| UnsupportedContentBlockException.rejectMedia("IBM watsonx.ai", messages); |
There was a problem hiding this comment.
The media check only runs in the four-argument chat(), but BaseChatModelSetup.chat() calls the three-argument overload, which delegates directly to doChat(). That path still silently drops media via getText() and sends only the text.
Could we delegate the three-argument overload to the four-argument one and add a regression test for that entry point? I reproduced this with a local HTTP endpoint: a text-plus-image message was sent as text only without throwing.
There was a problem hiding this comment.
Good catch; fixed in a7487b0, the three-argument chat() now delegates, with a regression test.
|
The cross-language failure is a Maven download error; could someone rerun that job? |
wenjin272
left a comment
There was a problem hiding this comment.
Thanks for the update. LGTM.
There are some conflicts with the main branch that need to be resolved.
a7487b0 to
9b2cc1f
Compare
|
Rebased onto main and resolved the DashScope rename conflict; thanks for the review! |
Linked issue: #1059
Purpose of change
A chat message with media sent to a provider that cannot carry it now fails with
UnsupportedContentBlockException/UnsupportedContentBlockErrorinstead of being sent as its text alone. This is the last Phase 2 item of #1059, "explicit unsupported-block errors for providers not yet migrated", after #1164 (OpenAI Chat Completions, Azure OpenAI, vLLM) and #1178 (Ollama).Covered: Java Anthropic, Gemini, Amazon Bedrock, IBM watsonx.ai and OpenAI Responses; Python Anthropic, DashScope and IBM watsonx.ai.
Runtime flow
chat()first callsUnsupportedContentBlockException.rejectMedia(provider, messages)(Java) orUnsupportedContentBlockError.reject_media(provider, messages)(Python).Key decisions
forBlock/for_blockfactory from [integration][java][python] Send multimodal user messages through OpenAI Chat Completions #1164, so every provider reports the same message:{provider} cannot send an image block (image/png, base64 source): this integration sends text only.chat(), before anytrythat wraps provider errors, so callers receive the documented type (the issue [integration][java][python] Send user images to Ollama #1178 review caught for Ollama).Behavioral Semantics
Interaction decisions
UnsupportedContentBlockException/UnsupportedContentBlockError; nothing is sentBehavioral contracts
chat(), not a wrapper.Java and Python behave identically.
Failure behavior
Tests
testMediaBlocksFailExplicitlyinAnthropicChatModelConnectionTest,GeminiChatModelConnectionTest,BedrockChatModelConnectionTest,WatsonxChatModelConnectionTest(both overloads),OpenAIResponsesModelConnectionTest, each throughchat()and asserting the exact messagetest_text_only_providers_reject_media.py, parametrized over Anthropic, DashScope and watsonx, throughchat()Not verified: a live provider; the check runs before any request, so none is needed for these contracts.
Implementation invariants and supporting evidence
rejectMedia/reject_mediaonly readgetBlocks()/.blocksand useforBlock/for_block, so messages match [integration][java][python] Send multimodal user messages through OpenAI Chat Completions #1164 and [integration][java][python] Send user images to Ollama #1178.chat()overload reaches the check: Anthropic, Bedrock, Gemini and (since 9b2cc1f) watsonx route the three-argument overload into the four-argument one; OpenAI Responses checks in the three-argument overload, which the base four-argument one calls.API
New public helpers:
UnsupportedContentBlockException.rejectMedia(String, List<ChatMessage>)(Java) andUnsupportedContentBlockError.reject_media(provider, messages)(Python).Compatibility: text-only requests are unchanged. A message with media to one of these providers used to be sent as its text only; it now fails.
Documentation
doc-neededdoc-not-neededdoc-includedThe provider-support note in
chat_models.mdnames these integrations and the error; the Bedrock limitation note says media raises.Was this patch authored or co-authored using generative AI tooling?
If yes, include a
Generated-by: <tool name and version> (<model name and version>)line, for exampleGenerated-by: Claude Code 2.1.226 (Claude Opus 4.6), in the commit message so it reaches Git history. Repeat the same line here for reviewer visibility. See the ASF generative tooling guidance.Generated-by: Claude Code 2.1.259 (Claude Opus 5.5)