Repository navigation
[Feature][Python] Support model routing #1333
Description
Search before asking
- I searched the existing issues; [Umbrella] Pluggable model routing (MODEL_ROUTER): delivery status, Python parity, documentation, and follow-ups #1062 tracks this work at the umbrella level, but there is no dedicated Python implementation issue.
Description
Parent: #1062.
Flink Agents 0.4 supports model routing through the Java API. Python currently exposes MODEL_ROUTER only for deserializing Java plans and rejects Python-side router registration.
Add Python model routing support with behavior aligned with the Java functionality delivered in 0.4. This issue covers declaration, registration, and execution, so Python applications can use a router wherever the supported chat APIs accept a model name.
Acceptance criteria
- Provide Python router and strategy declarations and support registration on an Agent or its execution environment.
- Support rule-based, LLM-judge, and custom routing strategies, with serializable declarations and plan-time validation consistent with Java.
- Preserve the select-not-delegate contract: strategies select a candidate, while the framework owns model invocation, retries, and tool execution.
- Align default-model selection, abstention, invalid selections, candidate fallback order, and failure handling with Java.
- Keep the selected model throughout the tool-calling loop and preserve routing decisions and model calls through durable execution and replay.
- Provide equivalent routing events, response metadata, and metrics, including judge usage attribution.
- Add deterministic tests for the above contracts, relevant cross-language coverage, and Python examples and documentation.
Scope
Use Java 0.4 routing behavior as the parity baseline. New strategies such as DecisionModel routing (#1325) remain separately tracked. YAML resource declarations will be tracked in a separate issue that depends on this work. Java annotation support remains in #1133.
Are you willing to submit a PR?
- I'm willing to submit a PR!
Activity
sangkyoonnam commentedon Oct 29, 2026
@purushah are you planning to take this one, since #1062 is with you? If not, I'd like to work on the Python routing support here. @wenjin272 could you assign it to me if that works for you both?
wenjin272 commentedon Oct 29, 2026
Thanks for volunteering, @sangkyoonnam! I'll assign this to you for now. @purushah, please let us know if you were already planning to work on this or have any concerns, and we can coordinate.
purushah commentedon Oct 29, 2026
@sangkyoonnam yes, that works for me. Thank you for taking it. I am not planning to work on the Python side. One thing to keep in mind: #1330 adds a decision-model strategy on the Java side and is under review, so the strategy list may grow by one. If you have questions about the Java behavior, ping me.
sangkyoonnam commentedon Oct 29, 2026
I'll implement this natively in the Python chat model action, with router descriptors that use the same keys and strategy tags as Java.
RemoteExecutionEnvironment.applyalready sends the Python plan JSON to Java for validation, soAgentPlan.validateRoutingStrategieswould check Python-declared routers too.I'll split this into three PRs, with the LLM judge last as in the Java work. First, the Python
chat()gets split into per-model attempts, since fallback needs a unit that retries one candidate without sending the final failure event. That PR shouldn't change behavior. Then routers with the rule-based and custom strategies, fallback, theroute:<router>andchat:<router>:<candidate>durable IDs,ModelRoutingEventand the latency histogram. Then the LLM judge. Until a follow-up adds it, Python would reject the decision-model strategy from #1330 at plan time.One Java change comes with the second PR.
validateCustomExecutorcallsClass.forNameon the executor class, so a Python custom strategy fails Java validation. I'll skip that check when the router comes from aPythonResourceProviderand validate the class on the Python side instead. Rule patterns will also have to compile in both Python and Java, since Java validates them too; I'll document that restriction and add tests for the known matching differences. Let me know if you'd rather have this in one PR or handle the custom check another way.