[purushah]

What

Implements the Strategies.llm(...) follow-up promised in discussion #897: the engine — not the strategy — executes the judge call, on the same durable, metered, observable chat path as any model call. Tracking: #1062.

Revised after review (thanks @wenjin272, @weiqingy): the PR now adopts the declaration/executor architecture proposed in review — one serializable RoutingStrategy declaration at the API layer, executors in Plan, dispatch by a language-neutral strategy type — plus all eleven review items. The user-facing builder is unchanged:

env.addResource("judge", ResourceType.CHAT_MODEL, judgeModelDescriptor);  // any plain chat model

env.addResource("router", ResourceType.MODEL_ROUTER,
    ModelRouter.of("small", "big")
        .describe("small", "fast and cheap: chit-chat, lookups")
        .describe("big",   "expensive: code, SQL, multi-step reasoning")
        .strategy(Strategies.llm("judge"))          // optional: .withMaxContextChars(n)
        .defaultModel("small")
        .fallback(true)
        .build());

Architecture (revised per review)

• RoutingStrategy is a declaration, not behavior: RoutingStrategyType (RULE_BASED | LLM_JUDGE | CUSTOM) + arguments (+ executor class for CUSTOM). There is no route() at the API layer — the previous UnsupportedOperationException-throwing judge class is gone.
• Execution lives in Plan, dispatched by type: rule matching in the resolver over patterns pre-compiled by the router; judge prompt/verdict logic in LlmJudgeRoutingExecutor (package-private pure helpers). No instanceof on concrete classes.
• Plan JSON is language-neutral: strategy_type: "llm_judge" + strategy_args — no Java class names for built-ins. A Java-emitted snapshot is committed and the Python cross-language test deserializes it verbatim, so Python parity ([Umbrella] Pluggable model routing (MODEL_ROUTER): delivery status, Python parity, documentation, and follow-ups #1062) needs only Python-side executors.
• The resolver owns the single durable boundary (route:<router>) for every strategy type — no executor can skip persistence; replay short-circuits execution, so neither the strategy nor the judge is re-invoked on recovery.
• Custom logic implements CustomRoutingExecutor with the data-only RoutingContext (v1's select-don't-delegate fence, verbatim): a custom executor cannot make hidden model calls. The engine-facing signature stays plan-internal.

Judge semantics (unchanged from the original design, per #897)

• Judge chat under durable id "judge:<router>" (engine retries, trace events, token attribution); decision persisted with source and judge-inclusive decision_ms under "route:<router>"; abstains persist as abstains, so replay after a candidate-set change resolves to the current default.
• Verdicts constrained to candidate names; unparseable / ambiguous / non-candidate verdicts abstain to the default with the cause recorded.
• Failure policy: exhausted retries honor the error-handling strategy (FAIL loud, IGNORE abstain-with-cause). Cancellation is determined from the thread's interrupt state plus explicit cancellation types (isCancellation); a bare InterruptedIOException (the shape Okio uses for ordinary HTTP timeouts) follows the normal failure policy — with a regression test.
• Judge input is full-fidelity (review): the complete message list, plus the rendered request when the anchor candidate binds a prompt — via the same BaseChatModelSetup.prepareRequestMessages the chat path uses, so the judge's view cannot drift from what the selected model receives. Opt-in cost cap: withMaxContextChars(n) (rendered request + SYSTEM pinned, newest-first fill, truncation flagged as judge_context_truncated).
• Fail-fast validation: strategy argument rules live in the declaration constructor (bad configs fail at Strategies.llm(...)/plan construction — no reflective validation path to bypass). Plan construction verifies the judge model is registered and its descriptor binds no prompt/tools/skills; a slim runtime backstop catches instance-level bindings plan-time cannot see. Custom executor classes are checked (exists / implements / constructor) without instantiation.
• Python: add_resource(..., MODEL_ROUTER) raises an explicit not-yet-supported error.

Deliberate trade-offs (reviewed, kept)

• The judge path performs a second durable write (route: in addition to judge:) to keep the route-record schema uniform and preserve replayed decision_ms.
• The judge's rendered view anchors to the default candidate's bound prompt. Candidates binding different prompts see their own rendering only at answer time — rendering per-candidate before the decision is circular. Documented in the resolver javadoc.
• Full judge context means judge cost scales with conversation length; the cap is deliberately opt-in rather than a default, so the out-of-box behavior matches what the selected model sees.

Claims → tests

claim	test
verdict routes to named candidate, durable ids, metadata	llmJudgeVerdictRoutesToNamedCandidate
unparseable / non-candidate / ambiguous verdicts abstain	llmJudgeUnparseableVerdictAbstainsToDefault, llmJudgeVerdictOutsideCandidatesAbstainsToDefault, parseVerdictAcceptsOnlyCandidates
replay: stored abstain → current default; strategy not re-run	storedAbstainReplaysToCurrentDefaultWithoutRerunningStrategy
replay: original decision_ms reported	storedDecisionReplaysWithOriginalDecisionMs
judge timeout (InterruptedIOException) under IGNORE abstains, thread not left interrupted	judgeInterruptedIOTimeoutAbstainsToDefaultUnderIgnore
judge sees full conversation / rendered bound prompt	judgeSeesFullConversation, judgeSeesRenderedBoundPrompt
cap pins SYSTEM + rendered template + newest; flags truncation	maxContextCharsCapsJudgeInputAndFlagsTruncation, contextCapPinsRenderedTemplateMessages
SYSTEM + empty USER + promptArgs does not route blind	judgeMessagesFallBackToPromptArgsWithSystemAndEmptyUser
judge token keys actually produced	judgeTokenCountsLandInDecisionMetadata
bad-args judge / missing judge model / non-plain judge fail at plan construction	judgeWithMissingJudgeModelFailsAtPlanConstruction, typoedJudgeModelFailsAtPlanConstruction, judgeWithBoundPrompt/ToolsFailsAtPlanConstruction
instance-level bound prompt caught at runtime (FAIL loud / IGNORE degrade)	llmJudgeRejectsInstanceLevelPromptBoundJudgeSetupUnderFail, llmJudgePromptBoundSetupAbstainsToDefaultUnderIgnore
custom executors never instantiated at plan time	customExecutorIsNotInstantiatedDuringPlanConstruction
Java plan deserializes in Python	committed Java snapshot + test_python_can_deserialize_java_plan_with_judge_router

Compatibility impact

• API change (0.4.0 is unreleased, so no released users affected): custom strategies now implement CustomRoutingExecutor.route(strategy, context) instead of RoutingStrategy.route(context); Strategies.of(...) is now Strategies.custom(...); the plan wire format uses strategy_type/strategy_args (+ strategy_executor_class for CUSTOM) instead of strategy_clazz.
• Plain-chat agents and rule-routed agents are behaviorally untouched.
• Event consumers see the decision_source value llm_judge and optional metadata keys judge_model, judge_prompt_tokens, judge_completion_tokens, judge_context_truncated. (decision_source is no longer duplicated inside the metadata map — it contradicted the top-level key on the judge+fallback path.)

Test evidence

• 702 Java tests green (api 378, plan 324; 13 pre-existing skips) including the new replay-path, cancellation, context-fidelity, and plan-validation suites; Python suites green including the Java-snapshot cross-language test.
• End-to-end re-run post-restructure on a standalone Flink 2.3.0 cluster (dist jar + flink run of ModelRoutingJudgeExample, local Ollama): judge verdicts routed with full metadata; fallback composition exercised live.

Generative AI was used for this change.

Generated-by: Claude Code 2.1.239 (Claude Fable 5)

[wenjin272]

Thanks for opening the follow-up PR @purushah. Could you please create a corresponding github tracking issue and link the design discussion, #964, and this PR? The issue should capture the complete scope: what was delivered in V1, the planned V2 implementation, Python-side parity, and user documentation. Any additional follow-ups can then be linked to it so we can track the feature end to end.

I’ll start reviewing this PR as soon as possible.

[purushah]

Done — opened #1062 to track the feature end to end (design discussion #897, v1 #964, this PR, plus Python parity and user documentation as follow-ups). @wenjin272

[wenjin272]

Thanks for the work on this—the implementation looks solid overall. I’ve left a few minor comments.

Since LLM-based routing has a special framework-managed execution model, it also prompted some broader thoughts about the current routing API design, particularly the separation between API-level strategy declarations and Plan-level execution. It would be good to align on this design before finalizing the PR.

[weiqingy]

Thanks for taking this on. A few questions inline.

[weiqingy]

Thanks for working through the five earlier points. All five look closed to me. A few new questions inline on the restructure.

[weiqingy]

Thanks for working through these. All six from the last round check out. Three small things inline, all in the new code.

[wenjin272]

Thanks for the thoughtful updates and for addressing the previous comments. I’ve left a couple of follow-up thoughts about the execution-path abstraction: the declarations are now unified, while rule, custom, and LLM execution still follow different paths, with some execution details remaining in the API layer. A common Plan-side RoutingExecutor abstraction may provide a cleaner boundary. This is mainly something I think is worth discussing.

[wenjin272]

Thanks for addressing my comments. Only two minor comments.

Could we add a dedicated routing package in the Plan module like API module and move the routing resolver, executor interface, registry, and implementations there? This would keep the routing logic better organized.

[weiqingy]

Thanks for working through these. All three from the last round are fixed, and I left two more comments inline.

[weiqingy]

Thanks for working through these. All of my comments are resolved, nothing further from me.

[wenjin272]

Thanks for addressing the comments. LGTM