Support Custom LLM Models for LlmAsAJudgeCriterion

Problem

Currently, LlmAsJudge (used by FinalResponseMatchV2Evaluator and other LLM-based evaluators) requires the judge model to be registered in LLMRegistry. When using a custom model that is not pre-registered (e.g., openai/Qwen3-72B), the evaluation fails with:

ValueError: Model openai/Qwen3-72B not found.

This happens because LlmAsJudge._setup_auto_rater() (in llm_as_judge.py:165) calls LLMRegistry.resolve(), which only resolves models that have been explicitly registered via LLMRegistry.register().

Current Behavior

# In llm_as_judge.py:162-166
def _setup_auto_rater(self) -> BaseLlm:
    model_id = self._judge_model_options.judge_model
    llm_registry = LLMRegistry()
    llm_class = llm_registry.resolve(model_id)  # Raises ValueError if not found
    return llm_class(model=model_id)

Users configure the judge model via judge_model_options in evaluation criteria:

{
  "criteria": {
    "final_response_match_v2": {
      "threshold": 0.8,
      "judge_model_options": {
        "judge_model": "openai/Qwen3-72B",
        "num_samples": 5
      }
    }
  }
}

However, models like openai/Qwen3-72B (used via LiteLLM or other custom providers) are not registered in the registry, causing evaluation to fail.

Proposed Solution

Enhance LlmAsAJudgeCriterion and LlmAsJudge to support custom LLM models by:

Option A: Support Direct BaseLlm Instance
- Allow judge_model_options.judge_model to accept either:
  - A string (model ID) for registered models (current behavior)
  - A pre-instantiated BaseLlm instance for custom models
- Modify _setup_auto_rater() to handle both cases
Option B: Support Custom Model Configuration
- Extend JudgeModelOptions to include optional fields for custom model instantiation:
  - model_type: Provider type (e.g., "litellm", "custom")
  - model_config: Additional configuration (e.g., base_url, api_key, etc.)
- Implement fallback logic in _setup_auto_rater() to create custom models when registry lookup fails
Option C: Registry Fallback with Model Factory
- Add a model factory function that can instantiate common model types (LiteLLM, etc.) when registry lookup fails
- Allow users to register custom model factories for their specific use cases

Expected Behavior

Users should be able to configure custom LLM models for evaluation without requiring pre-registration:

{
  "criteria": {
    "final_response_match_v2": {
      "threshold": 0.8,
      "judge_model_options": {
        "judge_model": "openai/Qwen3-72B",
        "model_type": "litellm",
        "model_config": {
          "base_url": "http://xxx.xxx.xxx.xx:8000/v1",
          "api_key": "EMPTY"
        },
        "num_samples": 5
      }
    }
  }
}

Or use a custom model instance programmatically:

from google.adk.models.litellm import LiteLlm

custom_judge = LiteLlm(
    model="openai/Qwen3-72B",
    base_url="http://xxx.xxx.xxx.xx:8000/v1",
    api_key="EMPTY"
)

eval_metric = EvalMetric(
    metric_name=PrebuiltMetrics.FINAL_RESPONSE_MATCH_V2,
    criterion=LlmAsAJudgeCriterion(
        threshold=0.8,
        judge_model_options=JudgeModelOptions(
            judge_model=custom_judge,  # Direct instance
            num_samples=5
        )
    )
)

Impact

This enhancement would:

Improve flexibility for users using custom or private model deployments
Remove the need to modify ADK codebase to register custom models
Enable easier integration with LiteLLM and other LLM providers
Maintain backward compatibility with existing registered models

Related Code

google/adk/models/registry.py:86-102 - LLMRegistry.resolve()
google/adk/evaluation/llm_as_judge.py:162-166 - _setup_auto_rater()
google/adk/evaluation/final_response_match_v2.py:150 - Uses LlmAsJudge

@ankursharmas

Hey @ankursharmas @nathan-tsien @hangfei
I'd like to help implement this feature. I think Option A works better, and only 2 files require change -

1. eval_metrics.py:

judge_model: Union[str, BaseLlm] = Field(default="gemini-2.5-flash")

2. llm_as_judge.py:

# In _setup_auto_rater():
def _setup_auto_rater(self) -> BaseLlm:
    jm = self._judge_model_options.judge_model
    if isinstance(jm, BaseLlm):
        return jm
    return LLMRegistry().resolve(jm)(model=jm)

# In evaluate_invocations() (line ~180):
llm_request = LlmRequest(
    model=self._judge_model.model,  # ← Extract string from BaseLlm
    ...
)

Usage:
custom_judge = LiteLlm(model="openai/Qwen3-72B", base_url="...", api_key="EMPTY") eval_metric = EvalMetric( criterion=LlmAsAJudgeCriterion( judge_model_options=JudgeModelOptions(judge_model=custom_judge) ) )

Let me know if this works..
Happy to submit a PR!

@nathan-tsien

@nathan-tsien Thank you for reporting this. I understand the request is currently under review. Please keep me posted if anything else is needed. Thanks for your patience.

@klateefa

Hello @klateefa , is this feature request still up to date ? I'd like to help if @chinmay-ratnaparkhe is not working on it anymore

@morganroux

Hey @morganroux , apologies.
I'm currently not working on this, had a few things come up. You may take it over.

@chinmay-ratnaparkhe

Thank you @chinmay-ratnaparkhe 👍
@klateefa @ankursharmas can you confirm I can start working on this please ?

feat: support BaseLlm instances as judge_model in evaluation

@nathan-tsien

Hi @nathan-tsien, thank you for the suggestion! This feature is already supported. You can configure the judge_model_options parameter on the evaluation criteria classes to specify any custom LLM model name. We are closing this issue, but please feel free to reopen it or open a new one if you have further questions. Thanks!

feat: support BaseLlm instances in evaluation and user simulation

Navigation Menu

Repository navigation

Support Custom LLM Models for LlmAsAJudgeCriterion #3400

Support Custom LLM Models for LlmAsAJudgeCriterion

Problem

Current Behavior

Proposed Solution

Expected Behavior

Impact

Related Code

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support Custom LLM Models for LlmAsAJudgeCriterion #3400

Description

Support Custom LLM Models for LlmAsAJudgeCriterion

Problem

Current Behavior

Proposed Solution

Expected Behavior

Impact

Related Code

Activity

chinmay-ratnaparkhe commented on Nov 10, 2025

klateefa commented on Nov 26, 2025

morganroux commented on Feb 18, 2026

chinmay-ratnaparkhe commented on Feb 19, 2026

morganroux commented on Mar 1, 2026

DeanChensj commented on Jun 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions