[Zhuoxi2000]

Linked issue: #1059

Purpose of change

A user ChatMessage with image blocks now reaches Ollama with its images, in Java and Python, so vision models such as qwen2.5vl or llava can see them. Until now the Ollama connections sent only the text and dropped the images silently. This is the second provider step of #1059 Phase 2, after #1164.

Runtime flow

1. Each message is converted as before (convertToOllamaChatMessages in Java, __convert_to_ollama_messages in Python), including assistant tool calls (forwarded in Java since [integration][java] Forward assistant tool calls in the Ollama chat history #1166).
2. The message's media blocks are then checked in order: a user message's Base64 ImageBlocks are decoded and attached as the message's images; any other media throws before the request is sent.
3. A message without media is sent exactly as before, with no images field.

Key decisions

• Images are attached next to the text, not interleaved: Ollama's /api/chat has one content string and one images list per message, so block order is kept among the images but not between text and images.
• Only Base64 images: Ollama takes inline image data and cannot fetch a URL, and fetching it in the connection would make the framework an HTTP client for arbitrary URLs.
• The Base64 data is decoded to bytes. ollama4j takes bytes and re-encodes them; ollama-python treats a string as a file path first, so bytes avoid an accidental file read.
• Unsupported media raises the shared UnsupportedContentBlockException / UnsupportedContentBlockError from [integration][java][python] Send multimodal user messages through OpenAI Chat Completions #1164.

Behavioral Semantics

Interaction decisions

Role	Blocks	Result
user	text only, or none	sent as before, no images
user	Base64 images, with or without text	content = text projection, images = the images in block order
user	an image by URL, audio, video or a document	UnsupportedContentBlockException; nothing sent
system / assistant / tool	any media	UnsupportedContentBlockException; nothing sent

Behavioral contracts

1. A message without media is sent unchanged, without images.
2. A user message's Base64 images are sent as images, in block order, and serialize back to the original Base64 strings.
3. The message content is the text projection.
4. An image by URL, and any audio, video or document block, throws UnsupportedContentBlockException naming the block but not its data or URL.
5. Media in a system, assistant or tool message throws UnsupportedContentBlockException.
6. Image data that is not valid Base64 throws IllegalArgumentException / ValueError ("could not be decoded"), not the unsupported-block error.
7. These errors reach callers of the public chat() with their own types; Java no longer wraps runtime exceptions in a plain RuntimeException.

Java and Python behave identically for each contract.

Failure behavior

• Unsupported media or role, and undecodable image data: thrown while building the request; nothing is sent, and the chat action's error strategy applies. These errors are deterministic, so RETRY fails the same way each attempt.
• A model without vision support: Ollama's own error propagates unchanged.

Tests

Contract	Java: OllamaMultimodalTest	Python: test_ollama_multimodal.py
1	testTextOnlyMessageHasNoImages	test_text_only_message_has_no_images
2, 3	testBase64ImagesAttachedInBlockOrder	test_base64_images_attached_in_block_order
4	testUnsupportedMediaFailsExplicitly	test_unsupported_media_fails_explicitly
5	testImagesOutsideUserMessagesFail	test_images_outside_user_messages_fail
6	testInvalidBase64Fails	test_invalid_base64_fails
7	testChatKeepsMediaErrorTypes	the tests above all call chat()
End to end	MultimodalChatIntegrationTest (it-java)	multimodal_chat_test.py (it-python)

The end-to-end tests run an agent on Flink whose input row carries a 64x64 PNG, through the chat request event and built-in chat action to the CI Ollama server, with qwen3.5:2b (about 2.7 GB; override with OLLAMA_VISION_MODEL). Opt-in live tests (OllamaMultimodalLiveTest, test_ollama_multimodal_live.py) send the same image straight through the connection.

Not verified:

• The opt-in live tests: not run. The end-to-end tests ran in CI on 4356394 and passed in both languages, with qwen3.5:2b answering "Red" for the red square.
• Interleaving text and images: not representable in Ollama's API (see Key decisions).

Implementation invariants and supporting evidence

• ollama4j 1.1.5 OllamaChatMessage.images is List<byte[]> serialized with FileToBase64Serializer; ollama-python 0.6.1 Message.images holds Image values whose serializer base64-encodes bytes and treats strings as paths first.
• The Java conversion keeps [integration][java] Forward assistant tool calls in the Ollama chat history #1166's tool-call forwarding for assistant messages alongside the images.
• Verified locally: Ollama module tests 24/24 with spotless; the e2e module test-compiles; Python Ollama and chat-message tests pass, with ruff.

API

No new API; it uses the UnsupportedContentBlockException / UnsupportedContentBlockError added in #1164.

Compatibility: text-only requests are unchanged. User images used to be dropped silently; they are now sent. Other media, and media in non-user messages, now fail instead of being dropped.

Documentation

• doc-needed
• doc-not-needed
• doc-included

A multimodal note in the Ollama section of chat_models.md, and Ollama added to the provider-support note.

Was this patch authored or co-authored using generative AI tooling?

• Yes
• No

If yes, include a Generated-by: <tool name and version> (<model name and version>) line, for example Generated-by: Claude Code 2.1.226 (Claude Opus 4.6), in the commit message so it reaches Git history. Repeat the same line here for reviewer visibility. See the ASF generative tooling guidance.

Generated-by: Claude Code 2.1.259 (Claude Opus 5.5)

[Zhuoxi2000]

The only failure is the ReAct e2e test, which also failed on main: https://github.com/apache/flink-agents/actions/runs/36512441540

[wenjin272]

Thanks for adding Ollama image support and the live tests! I left three comments below based on the review and local testing.

[wenjin272]

Could we add multimodal tests to the existing Java and Python E2E modules, so they run in the it-java and it-python CI jobs?

Unlike the OpenAI tests, these do not require external API credentials: both IT jobs already install and start an Ollama server. With a small vision model provisioned, we can exercise the full agent execution path with real image input in CI, rather than leave this coverage behind an opt-in environment variable.

[Zhuoxi2000]

Good idea; I'll add Java and Python E2E image tests running in it-java/it-python.

[Zhuoxi2000]

Added in 4356394: Java and Python E2E tests send a real image through an agent.

[wenjin272]

Could we preserve the media validation exception types through the public chat() API? toOllamaImages() throws UnsupportedContentBlockException or IllegalArgumentException, but doChat() wraps both in a generic RuntimeException. I reproduced this through chat(), so callers cannot catch the exceptions documented by this PR, unlike the Python implementation.

Please narrow the catch or rethrow runtime exceptions unchanged, and cover these cases through chat() rather than only buildRequest().

[Zhuoxi2000]

Done in d2bc595: chat() rethrows them unchanged, with tests through chat().

[wenjin272]

Both Java and Python live tests failed locally with Ollama 0.17.5 and qwen3.5:2b: the image processor rejects the 16×16 fixture with height:16 or width:16 must be larger than factor:32. Changing only the image dimensions to 64×64 made both tests pass. Could we use a larger fixture?

Also, both tests inherit think=true, while the suggested qwen2.5vl:3b does not support thinking. Please explicitly configure think=false for that model so the tests exercise image handling rather than fail on an unrelated parameter.

[Zhuoxi2000]

Done: the fixture is now 64x64 and both live tests set think=false.

[wenjin272]

Thanks for addressing my comments. LGTM