Skip to content

[integration][java][python] Reject media blocks in providers that send text only - #1186

Merged
wenjin272 merged 2 commits into
apache:mainfrom
Zhuoxi2000:multimodal-reject-unsupported
Oct 7, 2026
Merged

wenjin272 merged 2 commits into
apache:mainfrom
Zhuoxi2000:multimodal-reject-unsupported

Conversation

@Zhuoxi2000

@Zhuoxi2000 Zhuoxi2000 commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Linked issue: #1059

Purpose of change

A chat message with media sent to a provider that cannot carry it now fails with UnsupportedContentBlockException / UnsupportedContentBlockError instead of being sent as its text alone. This is the last Phase 2 item of #1059, "explicit unsupported-block errors for providers not yet migrated", after #1164 (OpenAI Chat Completions, Azure OpenAI, vLLM) and #1178 (Ollama).

Covered: Java Anthropic, Gemini, Amazon Bedrock, IBM watsonx.ai and OpenAI Responses; Python Anthropic, DashScope and IBM watsonx.ai.

Runtime flow

  1. Each covered connection's chat() first calls UnsupportedContentBlockException.rejectMedia(provider, messages) (Java) or UnsupportedContentBlockError.reject_media(provider, messages) (Python).
  2. The first media block in any message, of any role, throws; text-only messages go on to the unchanged conversion and request.

Key decisions

Behavioral Semantics

Interaction decisions

Messages Result
text only, any roles unchanged request
any media block, any role UnsupportedContentBlockException / UnsupportedContentBlockError; nothing is sent

Behavioral contracts

  1. A covered connection throws for the first media block before sending anything.
  2. The error is the documented type itself when it leaves chat(), not a wrapper.
  3. The message names the provider, block type, media type and source type, never the data or URL.
  4. Requests without media are unchanged.

Java and Python behave identically.

Failure behavior

  • Media in a message: thrown before any request is built or sent; the chat action's error strategy applies, and RETRY fails the same way each attempt since the error is deterministic.
  • No other failure path changes.

Tests

Contract Java Python
1, 2, 3 testMediaBlocksFailExplicitly in AnthropicChatModelConnectionTest, GeminiChatModelConnectionTest, BedrockChatModelConnectionTest, WatsonxChatModelConnectionTest (both overloads), OpenAIResponsesModelConnectionTest, each through chat() and asserting the exact message test_text_only_providers_reject_media.py, parametrized over Anthropic, DashScope and watsonx, through chat()
4 the existing request-building tests of each connection the existing provider tests

Not verified: a live provider; the check runs before any request, so none is needed for these contracts.

Implementation invariants and supporting evidence

API

New public helpers: UnsupportedContentBlockException.rejectMedia(String, List<ChatMessage>) (Java) and UnsupportedContentBlockError.reject_media(provider, messages) (Python).

Compatibility: text-only requests are unchanged. A message with media to one of these providers used to be sent as its text only; it now fails.

Documentation

  • doc-needed
  • doc-not-needed
  • doc-included

The provider-support note in chat_models.md names these integrations and the error; the Bedrock limitation note says media raises.

Was this patch authored or co-authored using generative AI tooling?

  • Yes
  • No

If yes, include a Generated-by: <tool name and version> (<model name and version>) line, for example Generated-by: Claude Code 2.1.226 (Claude Opus 4.6), in the commit message so it reaches Git history. Repeat the same line here for reviewer visibility. See the ASF generative tooling guidance.

Generated-by: Claude Code 2.1.259 (Claude Opus 5.5)

@github-actions github-actions Bot added doc-included Your PR already contains the necessary documentation updates. fixVersion/0.4.0 priority/major Default priority of the PR or issue. labels Oct 1, 2026

@wenjin272 wenjin272 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution and for making unsupported media handling explicit across these integrations! I found one remaining path in Java watsonx that bypasses the new check; please see the inline comment.

Map<String, Object> modelParams,
Object outputSchema) {
// Media blocks are not sent yet; fail rather than drop them (#1059).
UnsupportedContentBlockException.rejectMedia("IBM watsonx.ai", messages);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The media check only runs in the four-argument chat(), but BaseChatModelSetup.chat() calls the three-argument overload, which delegates directly to doChat(). That path still silently drops media via getText() and sends only the text.

Could we delegate the three-argument overload to the four-argument one and add a regression test for that entry point? I reproduced this with a local HTTP endpoint: a text-plus-image message was sent as text only without throwing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch; fixed in a7487b0, the three-argument chat() now delegates, with a regression test.

@github-actions github-actions Bot added doc-included Your PR already contains the necessary documentation updates. and removed doc-included Your PR already contains the necessary documentation updates. labels Oct 5, 2026
@Zhuoxi2000

Copy link
Copy Markdown
Contributor Author

The cross-language failure is a Maven download error; could someone rerun that job?

@wenjin272 wenjin272 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. LGTM.

There are some conflicts with the main branch that need to be resolved.

@Zhuoxi2000
Zhuoxi2000 force-pushed the multimodal-reject-unsupported branch from a7487b0 to 9b2cc1f Compare October 7, 2026 12:57
@Zhuoxi2000

Copy link
Copy Markdown
Contributor Author

Rebased onto main and resolved the DashScope rename conflict; thanks for the review!

@github-actions github-actions Bot added doc-included Your PR already contains the necessary documentation updates. and removed doc-included Your PR already contains the necessary documentation updates. labels Oct 7, 2026
@wenjin272
wenjin272 merged commit e057ca6 into apache:main Oct 7, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-included Your PR already contains the necessary documentation updates. fixVersion/0.4.0 priority/major Default priority of the PR or issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants