[weiqingy]

Linked issue: #912

Purpose of change

When an agent asks for structured output and its chat model can enforce a schema natively, the final answer now comes from a provider call carrying the schema, so the provider enforces the format rather than the prompt. This completes #280: providers could apply a schema, but the runtime never handed one over.

Nothing changes when the schema does not resolve to native; under the default AUTO that covers every unlisted model and every RowTypeInfo schema.

Runtime flow

1. ReActAgent asks the chat model setup whether the schema will travel natively. If yes, the loop prompt describes the schema's fields without asking for JSON; otherwise it keeps the JSON instruction.
2. The chat model action asks the same once per request, before retrying.
3. The tool-calling loop runs as today.
4. On a native final answer (no tool calls), the same attempt makes one more call with the schema and no tools: a user message holding a conversion instruction and the answer's text. Its response is parsed.

Key decisions

• The extra call happens only on the native path. The prompt path keeps a single call.
• The extra call sends only the answer, as one user message, so no tool traffic, turn-order rule or provider-specific response field is involved.
• The extra call binds no tools (Gemini drops a native schema when tools are bound) and omits tool_choice, tool_choice_option and parallel_tool_calls, also inside additional_kwargs.
• The setup's strategy is the policy; a connection only encodes. It carries the schema without checking its model list, so NATIVE on an unlisted model keeps the schema.

Behavioral Semantics

Interaction decisions

Strategy	Connection's answer	Result
PROMPT	any	Prompt path
AUTO	native recommended	Native: extra call, field instruction
AUTO	feasible, or infeasible	Prompt path
NATIVE	native recommended, or feasible	Native
NATIVE	infeasible (e.g. RowTypeInfo)	Fails before any model call
any	setup bridged to the other language	Prompt path; NATIVE fails before any model call
any	no connection bound, check not overridden	Fails before any model call, naming open()

Behavioral contracts

1. The extra call happens only when the strategy resolves to native and the answer has no tool calls.
2. Its only message is a user message: the conversion instruction, then the answer's text.
3. It carries the schema, binds no tools, and omits the tool-choice parameters.
4. Its response is the one parsed.
5. A chat call given tools (Java 4-arg, Python tools=) skips the bound prompt, skill-discovery message and bound tools; otherwise chat prepares the request as before.
6. Tool-choice parameters are omitted whenever a call carries a schema and binds no tools; the native check sees the same parameters.
7. For a non-null schema, the native check requires a bound connection.
8. An unhonorable NATIVE fails with a message naming the connection and the schema.
9. ReActAgent uses the field instruction only on a native answer; any other answer or a failure keeps the JSON instruction.
10. A recovered run replays both calls from the durable record without calling the model.

Failure behavior

• A failed, truncated or unparseable extra call uses the existing retry budget; a retry repeats the attempt, not completed tool rounds.
• A failing native check fails like any model failure: a Java router tries its next candidate, otherwise a failed ChatResponseEvent is sent.
• Bridged setups refuse explicit tools or a schema.
• Cancellation propagates everywhere, including from the ReActAgent check.

Tests

Tests per contract

Contract	Pinned by
1	ChatModelActionRetryTest; test_chat_model_action_retry.py
2, 3	nativeSchemaConvertsTheFinalAnswerInASeparateSchemaCarryingCall; test_finalization_call_carries_answer_directive_and_schema
4	parse-from-final tests in both languages
5	BaseChatModelTest, test_chat_model_base.py; bridge refusals in PythonChatModelSetupTest, test_java_chat_model.py
6	tool-param tests in both base test files
7	testWillApplyNativeStructuredOutputRequiresOpen; test_will_apply_native_structured_output_requires_open
8	strategy x answer grids in both base test files; bridge tests
9	ReActAgentTest; test_react_agent_schema_instruction.py
10	ChatModelInvokerTest replay; test_flink_runner_context_reconcilable.py

End to end against Ollama (qwen3:1.7b), ReActAgentTest#testReActAgentNativeStructuredOutput and test_react_agent_native_structured_output_on_remote_runner run the ReAct tool loop with a POJO / BaseModel schema under AUTO. A recording connection checks that the loop requests carried the tools and no schema, that the last request carried the schema and no tools, and that the model called a tool; the typed result is checked too. Forcing the prompt strategy makes both tests fail.

Every commit was mutation-tested, and all non-equivalent mutants were killed.

Not verified:

• No call to a hosted provider; only a local Ollama model runs end to end.
• Whether a loop answer written from the field instruction always lets the conversion call fill every field.
• The setup's answer is not stored durably; a configuration change before recovery may switch paths, and the journal then discards the later records.

Known limit: Java POJO instructions carry no field descriptions, since Jackson's schema generator drops them (as on the JSON path).

API

• New on BaseChatModelSetup: willApplyNativeStructuredOutput and chat(messages, tools, modelParams, outputSchema) (Java); will_apply_native_structured_output and keyword-only tools / output_schema on chat (Python; prompt_args with tools raises). Users configure structured_output_strategy instead.
• Behavior change: a custom setup that binds no connection and receives an output schema now fails unless it overrides the native check.
• Behavior change for direct connection.chat(..., outputSchema) callers: an unlisted model now gets the native parameter, so Anthropic drops its JSON prefill, Azure and DashScope reject a caller-supplied response_format, and Python raises TypeError for a schema Pydantic cannot render.
• The [api][integrations][java][python] Ask each connection whether it could apply a schema natively #1129 connection contract now says the native branch carries the schema unless the answer is infeasible. A ReActAgent with a schema registers one more prompt resource; an older plan gets no instruction on the native path.

Documentation

• doc-needed
• doc-not-needed
• doc-included

User documentation for structured output across providers follows in a separate PR.

Was this patch authored or co-authored using generative AI tooling?

• Yes
• No

Generated-by: Claude Code 2.1.294 and GitHub Copilot CLI 1.0.95 (Claude Opus 5.5)

[wenjin272]

Thanks for putting this together, @weiqingy, and for combining the remaining wiring into one PR! I left a few inline comments on simplifying the setup/finalization interfaces, removing the connection-null fallback, and handling tool-specific parameters in the vLLM finalization request.

[wenjin272]

Could we simplify the finalization path and keep its orchestration in the chat action/invoker?

The finalization input should be just the last assistant response plus the conversion directive, with the schema passed separately. That response already has no tool calls. If the final answer itself is incomplete, format conversion should not try to repair it using earlier context. This would remove the need to reconstruct the full history, call prepareRequestMessages again, or filter it through withoutToolTraffic.

Instead of a dedicated chatStructured method or a new request type, could we add an overload along these lines?

chat(messages, tools, modelParams, outputSchema)

This overload would use the supplied messages and tools without injecting the setup's bound prompt or tools. The existing chat(messages, promptArgs, modelParams) could prepare the normal request and delegate to it. The action/invoker would call the explicit overload with the last response, conversion directive, empty tools, and schema.

[weiqingy]

Done in e783465. The setup now has chat(messages, tools, modelParams, outputSchema), which sends messages and tools as given. The existing chat prepares the request and delegates to it. chatStructured, the history rebuild and withoutToolTraffic are gone. Python can't overload, and a new keyword on chat would reach the provider through **kwargs in subclasses that override it, so the Python twin is a separate chat_explicit(messages, tools, output_schema=None, **kwargs).

I changed one thing in the shape. The request is a single user message, the directive followed by the answer text, instead of [assistant(answer), user(directive)]. On Bedrock an empty answer becomes an assistant message with no content, and some vLLM/Ollama chat templates may refuse a conversation that opens with an assistant turn. Does that trade-off work for you?

One thing I'm unsure about on the ReAct side. On the native path the loop call no longer carries the schema instruction (613b275, 7e65097), and the conversion call now sees only the answer. So nothing tells the model which fields to fill, and a missing field either fails to parse or gets made up. Should we keep the schema instruction on the native path, or at least its field list? Or do you see a better place for that hint?

[wenjin272]

Could we remove the connection == null fallback from willApplyNativeStructuredOutput?

The framework calls open() before exposing the setup for use, so a normal setup should already have its connection initialized here. Treating a missing connection as a reason to select the prompt path hides a lifecycle or implementation error.

Special setups that intentionally operate without a connection can override this method, as the bridge setup already does. The tests expecting automatic fallback for a missing connection should be adjusted accordingly.

[weiqingy]

Done in d6a22a3. A null schema still returns false. Otherwise the gate now requires a connection and fails with a message that names open() and suggests overriding the method. I noted the compatibility impact in the PR description.

[wenjin272]

The finalization call removes tools but still inherits tool_choice. With vLLM and additional_kwargs={"tool_choice": "auto"}, this produces an invalid request: When using tool_choice, tools must be set.

Could we clear the tool-specific parameters for finalization and add a regression test?

[weiqingy]

Fixed in 445374f. When the call carries a schema and no tools, it drops tool_choice, tool_choice_option and parallel_tool_calls, at the top level and inside additional_kwargs. A plain chat call is unchanged. Regression tests added in Java and Python.

[wenjin272]

Thanks for the updates! A few follow-ups:

1. For Python, could we extend chat with keyword-only parameters instead of introducing chat_explicit? The provider setup classes do not override chat, and we can adapt the cross-language wrapper and test implementations together. We do not need to preserve compatibility with external custom subclasses for this change.

2. A single user message containing the conversion directive and the final answer works for me.

3. Good catch regarding the schema information in the ReAct loop. For this PR, let's keep the required fields and their descriptions visible to the model during the loop, without requiring JSON output there. The final conversion should still only format the final answer, without inferring missing information.

Could you also open a follow-up issue for the next release to support binding tools and the output schema together during the main loop? For providers/models that support this combination, the schema would make the required fields visible throughout the loop, so we could remove both the separately injected field instructions and the extra final conversion call.

[weiqingy]

@wenjin272 Thanks for the review. All four points are in, and main is merged to clear the conflicts (62906bd).

1. Keyword-only chat (d95f5b8). chat_explicit is gone. Python now has chat(messages, prompt_args=None, *, tools=None, output_schema=None, **kwargs). Passing tools (even []) sends the messages and tools as given; leaving it out prepares the request as before. prompt_args together with tools raises, since that path skips the bound prompt and would drop them silently. A call with only output_schema still takes the prepared path, so [api][python] Add explicit output schema parameter to the chat path (structured-output foundation) #843's routing is unchanged. JavaChatModelSetup.chat raises NotImplementedError for tools or output_schema, which mirrors the Java PythonChatModelSetup refusal. The test setups that override chat(messages, **kwargs) needed no change, since none of them is reached with tools or a schema.

2. Kept as is, thanks.

3. Schema fields in the ReAct loop (68e93bb). On the native path the loop prompt now reads "The final response must include the information for every required field of the following schema. It does not need to be in JSON format.", followed by the schema. It is a second prompt resource placed where the JSON instruction goes, so the two paths differ only in wording. The conversion directive is unchanged and still forbids adding or inferring information. One limit: Python shows field descriptions and the required list, but Java POJO schemas show neither, because Jackson's schema generator drops descriptions and marks a field required only with @JsonProperty(required = true). The JSON prompt on main has the same gap, so I left it out of this PR.

4. Follow-up: [Feature][chat-models] Apply the output schema on the tool-calling request when the provider supports both #1222. One question for that issue: when a connection reports it can take tools and the schema together, should AUTO use the single-call path by default, or should it be opt-in? I'd lean toward opt-in first. The OpenAI and Anthropic connections already answer NATIVE_RECOMMENDED with tools bound, but that answer only says the request can carry the schema, not that every loop turn should be constrained by it. As an AUTO default, it would change every loop request for those users at once. Once a model family has been checked, AUTO could adopt it. Would that fit how you see AUTO?

[wenjin272]

The code changes look good to me now. Could we also add an e2e test covering the native structured-output path through the tool-calling loop? The existing ReAct e2e tests use RowTypeInfo, so they still take the prompt path.

It should exercise a tool call, the final native conversion, and parsing of the structured result. Please also verify that the conversion request actually carries the native schema, so the test cannot pass through prompt fallback.

[wenjin272]

Thanks for addressing my comments. LGTM

[weiqingy]

@wenjin272 Thanks for the review! I merged main, which brings in the ChatResult refactor from #1185, and added the e2e tests in 63d8c70. They are ReActAgentTest#testReActAgentNativeStructuredOutput and test_react_agent_native_structured_output_on_remote_runner.

Each test runs the ReAct tool loop on Ollama (qwen3:1.7b) with a POJO / BaseModel schema under AUTO. A recording connection captures every request, and the tests check three things:

• the model called a tool, and the loop requests carried the tools and no schema;
• the last request carried the schema and no tools;
• the output was parsed into the schema type with the right value.

The prompt path never sends a schema, so a fallback run fails. I checked this by forcing the PROMPT strategy, and both tests failed.

The Java test checks the schema handed to the connection's chat, not the wire request, because the Ollama request building isn't public. OllamaChatModelConnectionTest#buildRequestSetsFormatForPojoSchema already covers the POJO → format step, so I'd keep it as is. WDYT?

Like the other ReAct e2e tests, the new ones rely on the small model calling the tool and getting the math right, so they can need a rerun. The Python one needed two on flink-2.2.