You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
RubricBasedEvaluator drops per-invocation rubric verdicts when invocations carry different rubrics #7301
RubricBasedEvaluator keeps the effective rubric list as instance state: format_auto_rater_prompt calls create_effective_rubrics_list(actual_invocation.rubrics), which sets self._effective_rubrics_list, and convert_auto_rater_response_to_score later reads get_effective_rubrics_list() to map parsed verdicts back to rubrics.
LlmAsJudge.evaluate_invocations formats the prompt for every invocation first (the loop at the top of the method), then gathers all sample tasks. So by the time any response is converted, _effective_rubrics_list is whatever the last invocation's rubrics were. A verdict for a rubric that is on an earlier invocation but not on the last one is discarded with the warning Rubric ... not found in the rubrics provided to the metric.
Observed on 2.9.0; the same code is on main (2.10.0): rubric_based_evaluator.py lines ~374–417 and ~445, llm_as_judge.py lines ~215 and ~238.
To Reproduce
An eval case with two invocations, where invocation 0 carries rubric a and invocation 1 carries rubric b (per-invocation rubrics, type matching the metric), and no criterion-level rubrics.
Run RubricBasedFinalResponseQualityV1Evaluator.evaluate_invocations with any judge model that answers both prompts.
Invocation 0's result has no rubric score for a (the warning above is logged); invocation 1's has b.
With the same rubric on both invocations the bug is invisible, which is why criterion-level rubrics never hit it.
Expected behavior
Each sample's response is converted against the rubrics of the invocation it was sampled for.
Suggested fix
Make the effective list per invocation rather than per evaluator: return it from create_effective_rubrics_list and carry it alongside the LlmRequest into _evaluate_single_sample, or resolve it inside convert_auto_rater_response_to_score from the invocation that produced the response. The per-rubric majority in MajorityVotePerInvocationResultsAggregator is unaffected.
Desktop
Python 3.13, google-adk 2.9.0 (also present on main)
Picking this one up now — opening a PR shortly. Flagging it here so nobody duplicates the work; if someone is already on it, say so and I will drop mine.
This issue has been automatically marked as stale because it has not had recent activity for 7 days after a maintainer requested clarification. It will be closed if no further activity occurs within 7 days.
Hi @erauner12, thank you for reporting this issue!
This has been resolved by PR #7366 and merged into the main branch in commit bc220d2.
We are closing this issue as resolved. If you find that the issue persists on the latest ADK release or if this was closed in error, please feel free to reopen it or leave a comment! Thank you again for your contribution.
Activity
chelsealong commentedon Sep 26, 2026
Picking this one up now — opening a PR shortly. Flagging it here so nobody duplicates the work; if someone is already on it, say so and I will drop mine.
llalitkumarrr commentedon Sep 29, 2026
Hello @erauner12,
Could you please share the eval cases and a minimal reproducible code with us? This will help us reproduce the issue and analyze it better?
adk-bot commentedon Oct 7, 2026
This issue has been automatically marked as stale because it has not had recent activity for 7 days after a maintainer requested clarification. It will be closed if no further activity occurs within 7 days.
DeanChensj commentedon Oct 14, 2026
Hi @erauner12, thank you for reporting this issue!
This has been resolved by PR #7366 and merged into the main branch in commit bc220d2.
We are closing this issue as resolved. If you find that the issue persists on the latest ADK release or if this was closed in error, please feel free to reopen it or leave a comment! Thank you again for your contribution.