You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(frontend): /m live evaluations like /w - #8178
The /w vs /m comparison found several differences on the live evaluation screens (findings evaluations-live-01, -02 and -04):
The new live evaluation drawer said "No filters • Sampling 100% • Live traffic" while its filter button showed 1. It also had no "Run on historical data" control, and an empty divider sat above the first section.
The live run's overview showed "1 query" instead of the filter. Cost, Duration and Tokens showed "—", so the radar said "Insufficient metrics".
The run header had "Pause", while the runs list menu says "Stop evaluation". A stopped live run still showed "Running".
The chart at 0.00% next to 95.45% in the table is already fixed in #8164, which this PR is based on.
Changes
Drawer.
The Query summary counts the locked "trace type is invocation" filter, like the filter button: "1 filter • Sampling 100% • Live traffic".
"Run on historical data" shows with its start and end dates, off and disabled with the tooltip "Not available yet", exactly as in /w. The API does not run live evaluations over past traces yet.
The first section has no top divider.
Overview. "Filters / Queries" shows the query itself and its sampling rate, as /w prints them:
trace_type is invocation AND references in [{"id":"…"}]
Sample rate: —
The text comes from the stored query, read with POST /queries/revisions/retrieve through a new runQueryConfigQueryAtomFamily, and from a pure formatter, formatQueryFiltering / formatQuerySampleRate. Both live in @agenta/entities/evaluationRun.
Cost, duration and tokens of a live run. A live run calls no app, so what it spends per trace is its evaluators' spend. The metric catalog now adds up the evaluators' means per trace. Errors were already added up across the evaluators.
On the parity data: Duration 483.86 ms (judge 483.65 + regex 0.21), Tokens 114, and a non-zero Cost. /w printed 243 ms, which is the average of the two evaluators, not the time a trace takes. Auto runs still read their app step only; a test pins that.
Stopped state. The header button says "Stop" and "Resume" (tooltips "Stop evaluation" and "Resume evaluation"), the words the runs list menu and /w use. A stopped live run's Status reads "Stopped" instead of "Running". The server keeps a stopped run's status as running and turns off is_active. I kept the header button, which /w does not have: it is the only way to stop or resume from the run page.
Not in this PR: the evaluator details in the drawer (evaluations-live-05: prompt, model and schema). That is a larger piece, best next to the evaluator drawer work.
Without beta developer mode: these screens are developer views (the Evaluations view is in the hidden stage). A user without beta developer mode reaches them only on a deployment that forces the view on, the same as #8164.
Based on #8164 (feat/m-parity-evaluation-logic); PR base = that branch.
Tests
evaluation-run-metrics.test.ts (entities):
a live run's duration, tokens and cost are the per-trace sum of its evaluators', on the captured parity run;
an auto run still reads only its app step.
The old assertion that a live run has no duration is updated.
evaluation-run-live-query-text.test.ts (entities): nested groups, list values, no filter, sampling rate.
runSummaryTable.render.test.tsx:
the query row shows the filter and the sampling rate, and waits while the query loads;
a stopped live run shows "Stopped".
liveEvaluationSheet.render.test.tsx:
the summary counts the invocation filter (updated from "No filters");
the historical data switch and its dates are disabled.
/m, /w and entities type checks pass; pnpm run lint passes; Prettier passes on the changed files.
Quick look on the parity data: the live overview shows the filter, 476 ms, 115 tokens, a cost, the radar, and 96.75% in both the card and the table. The drawer shows "1 filter" and the disabled historical control.
What to QA
Open the live evaluation "Capital finder live checks". Filters / Queries shows the filter. Cost, Duration and Tokens have values, and the radar draws.
Click Stop. The button turns into Resume, and Status reads "Stopped". Click Resume to restore it.
Evaluations, Live Evals, New evaluation. The Query line says "1 filter". "Run on historical data" is off and disabled, with the "Not available yet" tooltip.
Live run summaries now show query filters and sampling rates, with a loading placeholder while query details are unavailable.
Added a disabled historical-data option with date fields marked as unavailable.
Query summaries include the locked invocation filter.
Live run metrics now include evaluator-reported duration, token, and cost data when invocation details are unavailable.
Bug Fixes
Inactive live runs are labeled “Stopped,” and the run action is labeled “Stop” rather than “Pause.”
The pull request adds query configuration and historical-data controls to live evaluation views. It updates live-run status and query details in run summaries, and changes metric aggregation when invocation steps are absent.
Changes: Live evaluation controls
Layer / File(s)
Summary
Query controls and summary web/mobile/src/features/evaluations/live/HistoricalDataControl.tsx, web/mobile/src/features/evaluations/live/LiveEvaluationForm.tsx, web/mobile/src/features/evaluations/live/LiveSection.tsx, web/mobile/tests/unit/liveEvaluationSheet.render.test.tsx
The Query section adds disabled historical-data controls. The query summary counts the locked invocation filter, and tests cover the control defaults and updated counts.
Run overview details
Layer / File(s)
Summary
Query configuration and formatting web/packages/agenta-entities/src/evaluationRun/state/runQuery.ts, web/packages/agenta-entities/src/evaluationRun/core/liveQuery.ts, web/packages/agenta-entities/src/evaluationRun/{core,state,index}.ts, web/packages/agenta-entities/tests/unit/evaluation-run-live-query-text.test.ts
A query atom retrieves filtering and sampling configuration. Formatting helpers render nested filters and valid sample rates. The package exports these additions, with tests for the formatted output.
Overview query and status display web/mobile/src/features/evaluation-run/overview/{useRunOverview,RunOverviewView,runSummaryRows}.tsx, web/mobile/src/features/evaluation-run/{RunDetailsHeader,RunOnlineToggle}.tsx, web/mobile/tests/unit/runSummaryTable.render.test.tsx
The overview loads query configuration and displays its formatted filter and sample rate, with a loading marker while data is pending. Inactive live runs display “Stopped,” and the live-run action uses “Stop” labels and iconography.
When invocation steps are absent, cost, duration, and token metrics use evaluator steps. Multiple candidates combine per trace. Tests cover live-run metrics and auto-run duration.
sequenceDiagram
participant RunOverviewView
participant useRunOverview
participant runQueryConfigQueryAtomFamily
participant buildRunSummaryRows
RunOverviewView->>useRunOverview: Request overview data
useRunOverview->>runQueryConfigQueryAtomFamily: Read configuration for query ID
runQueryConfigQueryAtomFamily-->>useRunOverview: Return query configuration or pending state
useRunOverview-->>RunOverviewView: Return overview with query state
RunOverviewView->>buildRunSummaryRows: Pass query configuration
buildRunSummaryRows-->>RunOverviewView: Return formatted query details or loading marker
Loading
Possibly related PRs
Agenta-AI/agenta#8042: Adds the run overview, live-run toggle, and metric catalog that this pull request extends.
Agenta-AI/agenta#8125: Adds inactive live-run status and evaluator error aggregation, which this pull request updates and extends.
Merge Risk: Moderate · up to e9224
A run can show query settings it did not use and misleading metrics or status. Correct those displays before merging.
Pre-merge checks: 5
Check name
Status
Explanation
Title check
The title describes the goal of aligning /m live evaluations with /w.
Description check
The description covers the live evaluation UI, query display, metrics, and stopped state changes.
Docstring Coverage
Docstring coverage is 100.00% which is sufficient. The required threshold is 60.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage
Explanation
Docstring coverage is 100.00% which is sufficient. The required threshold is 60.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 7 functions across 18 files.
Finishing Touches
Generate docstrings
Commit to this branch
Create a new PR
Generate unit tests (beta)
Commit to this branch
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Preview use finished or expired. Environment absence verified; retained storage requires separate provider accounting.
Post /preview as a new comment to request a one-hour preview. Repeating it while active does not extend the hour.
Preview data is disposable. Expiry is enforced by the next successful cleanup sweep, not an exact timer.
Environment deletion stops its compute. Provider-retained storage and permanent services may still be billed.
Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.
Validate the filtering cast with zod before it reaches formatQueryFiltering.
Line 30 casts data?.filtering to QueryFilteringText without validation. The guideline requires zod validation at the boundary, with safeParseWithLogging. It says Fern types under-declare backend extra="allow" fields. A malformed condition (for example null inside conditions) makes "conditions" in condition in formatQueryFiltering throw. The overview then crashes instead of logging a drift error.
Add a local zod schema for QueryFilteringText. Parse data?.filtering with safeParseWithLogging from @agenta/entities/shared. Fall back to null on failure.
As per coding guidelines: "Keep zod validation at the boundary" and "Use safeParseWithLogging from @agenta/entities/shared for the validation."
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 5 remain after this review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Context
The /w vs /m comparison found several differences on the live evaluation screens (findings evaluations-live-01, -02 and -04):
The chart at 0.00% next to 95.45% in the table is already fixed in #8164, which this PR is based on.
Changes
Drawer.
Overview. "Filters / Queries" shows the query itself and its sampling rate, as /w prints them:
The text comes from the stored query, read with
POST /queries/revisions/retrievethrough a newrunQueryConfigQueryAtomFamily, and from a pure formatter,formatQueryFiltering/formatQuerySampleRate. Both live in@agenta/entities/evaluationRun.Cost, duration and tokens of a live run. A live run calls no app, so what it spends per trace is its evaluators' spend. The metric catalog now adds up the evaluators' means per trace. Errors were already added up across the evaluators.
On the parity data: Duration 483.86 ms (judge 483.65 + regex 0.21), Tokens 114, and a non-zero Cost. /w printed 243 ms, which is the average of the two evaluators, not the time a trace takes. Auto runs still read their app step only; a test pins that.
Stopped state. The header button says "Stop" and "Resume" (tooltips "Stop evaluation" and "Resume evaluation"), the words the runs list menu and /w use. A stopped live run's Status reads "Stopped" instead of "Running". The server keeps a stopped run's status as running and turns off
is_active. I kept the header button, which /w does not have: it is the only way to stop or resume from the run page.Not in this PR: the evaluator details in the drawer (evaluations-live-05: prompt, model and schema). That is a larger piece, best next to the evaluator drawer work.
Without beta developer mode: these screens are developer views (the Evaluations view is in the hidden stage). A user without beta developer mode reaches them only on a deployment that forces the view on, the same as #8164.
Based on #8164 (
feat/m-parity-evaluation-logic); PR base = that branch.Tests
evaluation-run-metrics.test.ts(entities):evaluation-run-live-query-text.test.ts(entities): nested groups, list values, no filter, sampling rate.runSummaryTable.render.test.tsx:liveEvaluationSheet.render.test.tsx:pnpm run lintpasses; Prettier passes on the changed files.What to QA