[mmabrouk]

Do not merge into v0.128.4. This goes into the next release. Merge #8040 and #8043 first.

Context

The /m evaluations view had no way to start a live evaluation. The Live tab had no create button, its empty state had no start button, and the All tab menu had no Live item. In /w the same flow lives in the "Online evaluation" drawer, which is built on antd and has a few bugs (listed below). The view stays behind the hidden evaluations developer view.

Changes

A user with edit_evaluation_runs now gets "Live evaluation" in the All tab create menu, a "New evaluation" button on the Live tab, and a "Create Evaluation" button in the Live empty state. An app scoped list offers no Live item, because live evaluations are project wide (the Live tab is already hidden there).

Each entry opens a new sheet with three sections:

• General: Name (required) and Description.
• Query: the shared trace filter dialog with the trace_type is invocation row locked at the top, and a sampling rate slider (0 to 100, default 100). The header says, for example, "2 filters • Sampling 25% • Live traffic".
• Evaluator: one evaluator, from the latest revision of each online capable, non-human evaluator. Under the select, the sheet shows the evaluator type and the names of its output metrics.

Create sends three requests, through the generated API clients:

POST /simple/queries/            {query: {slug: "support-replies-a1b2c3d4", name, description, data: {filtering, windowing: {rate: 0.25}}}}
POST /queries/revisions/retrieve {query_ref: {id: <query id>}}
POST /simple/evaluations/        {evaluation: {name, description, flags: {is_live: true, is_closed: false, is_active: false},
                                  data: {status: "pending", query_steps: {<query revision id>: "auto"}, evaluator_steps: {<evaluator revision id>: "auto"}, repeats: 1}}}

The payload is the /w payload. The API ignores is_active: false and starts the run at once. On success the sheet closes, the runs list refetches, and the list switches to the Live tab. The request code is createLiveEvaluation in @agenta/entities/evaluationRun. It uses a new getQueriesClient accessor in @agenta/sdk/resources.

The provider key check of the wizard moved to new/providerKey.ts so both sheets share it. The wizard behaves as before.

Not in this PR

The annotation row of the trace filter (pick an evaluator and its feedback) is not offered yet. It comes with AnnotatedTracesFilter, which another open branch builds. LiveTraceFilter.tsx is the one place to swap in when it lands. The other filter fields work.

/w bugs fixed in the /m version

• A failure after the query was created left an orphan query. /m archives it (POST /simple/queries/{id}/archive, best effort).
• Failures showed the raw API error text. /m shows "Couldn't create the live evaluation" and logs the detail.
• The Query summary counted the locked invocation row, so an empty form said "1 filter". /m counts only the user's rows.
• An LLM-as-a-judge evaluator in OSS without a provider key was created and then failed on every trace. /m blocks it with the wizard's message.
• The suggested name could fall back to "Evaluation-xxxx" because the option label was JSX. /m takes the evaluator name.
• /w shows the revision name of an evaluator, which is often "default". /m shows the evaluator name.

New user-facing text

Please check these; they are new or changed from /w:

• Menu item "Live evaluation" with "Evaluate live traffic with an automatic evaluator."
• Sheet title "Live evaluation" (/w: "Online evaluation configuration").
• Button "Create live evaluation" (/w: "Create online evaluation").
• Toast "Live evaluation created" (/w: "Online evaluation created").
• Error toast "Couldn't create the live evaluation".
• Field errors "Enter a name" and "Select an evaluator".
• "Could not load evaluators" with a retry.
• "Output metrics" above the metric names.

Behavior changes from /w to flag

• After create, the list switches to the Live tab (/w stays on the current tab). This follows the /m wizard.
• The disabled /w controls "Add tags", "Run on historical data" and the date range are dropped.

Tests

• @agenta/entities: evaluation-run-create-live.test.ts checks the exact three request bodies, the sampling rate, a query without data, references rows without a key, the slug, the evaluator rule, and the archive after a failure.
• /m: liveEvaluationSheet.render.test.tsx renders the real sheet with the real evaluator stores, the real createLiveEvaluation and the real vault store. Only the HTTP layer (axios and the generated clients) is mocked. It asserts the exact request payloads, validation, the suggested name, a double click, a failure (input kept, query archived), the provider key block, the empty and error states, and the success path.
• evaluationsScreen, evaluationsRoute and liveEvaluatorSummary tests cover the entries, the app scope rule and the Live tab switch.
• Found in the Storybook shots and fixed: the filter dialog trigger had no button type, so inside the form a click submitted the live evaluation. A test now opens the filter and checks that nothing is sent.
• Ran: /m types, /w types, entities and sdk types, the full /m suite (only the known failures in agentPermissions, membersTabRoleChange and testsetCommit; on the last run addToQueueSlotRegistration also timed out under load, and it passes alone), the entities suite (only the known mcp-policy-ladder failure on the remote copy), lint on the changed files, speed:check, and the antd route check on the evaluations route.
• Storybook: @agenta/mobile/Developer/Evaluations/LiveEvaluationSheet (Empty, Filled, ValidationErrors, NoSupportedEvaluators, Creating, Phone).

Known gaps:

• The shared filter dialog's badge counts the locked invocation row, so the trigger shows "1" while the Query summary says "No filters". The /w drawer shows the same badge. Changing it means changing the shared FilterDialog.
• When a latest-revision request fails, the shared workflowLatestRevisionQueryAtomFamily returns null, so the sheet says no supported evaluator exists instead of showing a load error. The /w drawer has the same gap. Fixing it changes a shared atom, so it is a follow-up.

What to QA

Turn on the evaluations developer view first.

• Evaluations page, All tab, "Start evaluation" menu. "Live evaluation" opens the sheet.
• Live tab with no runs. "Create Evaluation" opens the sheet. With runs, "New evaluation" opens it.
• Pick an evaluator with an empty name. The name becomes "{evaluator}-xxxx". Type a name first, and the name stays.
• Add a filter, set the sampling rate to 25, create. The sheet closes, the toast says "Live evaluation created", and the Live tab lists the new run as running.
• Click Create with no name and no evaluator. The fields say "Enter a name" and "Select an evaluator", and nothing is sent.
• Regression: start an auto evaluation from the wizard. It still works, and an LLM evaluator without a provider key is still blocked.

[mmabrouk]

🤖 The AI agent says:

Review loop result for head 4386ca772b (base cddc61bed9), 3 rounds. Lenses: slop and simplify (Opus and Codex), correctness (Codex). The security lens was not run.

Lens	Final rating
Slop	Low
Simplify	Low
Correctness	Low

Fixed:

• The live create uses the generated query and evaluation clients instead of raw axios (new getQueriesClient accessor).
• Evaluator options, the section summary and the suggested name use the evaluator name (or its slug), not the revision name, which is often "default".
• The rule that an app scoped list offers no Live item lives in the route only.
• picked and noneSupported are computed once; the payload helper types are private.
• The sheet tests wait on state instead of a fixed number of ticks (one test was flaky under load).

Left on purpose:

• A failed latest-revision request shows "no supported evaluators" instead of a load error. The shared revision atom swallows the error for every caller, the /w drawer too. Follow-up.
• A lost create response can leave a running run while the sheet reports a failure. Rare network case; the wizard has the same trade-off.
• selectLiveEvaluators keeps its own rule (the /w online rule on revisions); the existing helper serves another list shape.
• The name fallback is read in three places (Low, last round).

[coderabbitai]

Caution

Review failed

The pull request is closed.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[github-actions]

Website preview

Preview URL: https://pr-8068-agenta-website-preview.mahmoud-637.workers.dev

Built from 1a24973e381a59a0de87fbb163e6c5a88646d733. This comment updates in place on every push.

[mmabrouk]

🤖 The AI agent says:

Review loop delta for head b5a41ef420 (base 4386ca772b, the head of the full loop above), 1 round, Codex correctness.

Lens	Final rating
Correctness	Low

Fixed: the trace filter trigger sat inside the form without a button type, so a click submitted the live evaluation. It is now a plain button, with a test. The Storybook shots showed the bug.

[github-actions]

Railway Preview Environment

Item	Value
Status	stopped
Commit	1a24973e381a59a0de87fbb163e6c5a88646d733
Manual expiry (UTC)	Not granted
CI deadline (UTC)	Not granted
Workflow	https://github.com/Agenta-AI/agenta/actions/runs/41684814141

Preview use finished or expired. Environment absence verified; retained storage requires separate provider accounting.

Post /preview as a new comment to request a one-hour preview. Repeating it while active does not extend the hour.
Preview data is disposable. Expiry is enforced by the next successful cleanup sweep, not an exact timer.
Environment deletion stops its compute. Provider-retained storage and permanent services may still be billed.