Skip to content

feat(frontend): shared evaluation logic for /m, and the live card shows the run value - #8164

Merged
mmabrouk merged 2 commits into
release/v0.128.9from
feat/m-parity-evaluation-logic
Nov 6, 2026
Merged

mmabrouk merged 2 commits into
release/v0.128.9from
feat/m-parity-evaluation-logic

Conversation

@mmabrouk

@mmabrouk mmabrouk commented Nov 6, 2026

Copy link
Copy Markdown
Member

Context

The /w vs /m comparison found two numbers that disagree on the same run:

  • Live overview. The "Answer correctness" chart card said "Capital finder live checks: 0.00%" while the summary table said 95.45%. The card printed the newest one-minute bucket (two failures in that minute), not the whole run.
  • Auto run summary. /m showed "Duration (Avg) 522 ms" and /w showed "Duration (Total) 261 ms". The API row for this run has the app step at a mean of 522.35 ms over 10 scenarios. /m is right. /w gets 261 ms because its flatten sets the app step's value and then averages the next step in the response (an evaluator step at 0.2 ms) into it: (5223.472 + 2.091) / 20 = 261.3. Its "(Total)" label is also wrong: it prints a mean.

The evaluations plan (Decision 2) also asks for the pure /w logic that /m lacks (metrics, filters, export, compare) to live once in @agenta/entities, so the next /m evaluation PRs (list columns, filters, run header, scenarios, compare, live) build on it instead of writing it again.

Changes

/m fix. The live overview card now shows the whole run's value, the same number as the summary table. The chart lines are unchanged.

Shared logic in @agenta/entities/evaluationRun/core, copied from web/oss as pure functions (no antd, no state):

File From /w What it does
runStatsMap.ts EvalRunDetails/atoms/runMetrics.ts Flattens a run's run-level metric row into {stepKey}.{metricKey} and {metricKey} entries, merges stat blobs, reads the mean and total of an invocation metric. Invocation metrics come from the invocation step only (the /w bug above is not copied).
metricFormat.ts @agenta/ui/cell-renderers metric formatters The text the /w list cells and CSV export print for a metric. Copied because that barrel loads antd.
runFilters.ts list context, referencePayload.ts, previewRunsRequest.ts, filter summary Flags per tab (which hide annotation queue runs), reference filters, the POST /evaluations/runs/query body, the filter count.
runExport.ts useTableExport and the list export resolvers CSV cells and rows, file name, run name, status, kind, created at, metric header and value.
runCompare.ts atoms/compare.ts, state/urlCompare.ts, CompareRunsMenu.tsx The ?compare= value, the 4-run cap, whether a run can be compared and why not, and which runs can be compared with it.

Nothing in /m reads the new functions yet except the tests; the follow-up PRs (7.1, 7.2, 7.6, 7.7, 7.11) wire them in. web/oss is not changed, so /w keeps its 261 ms until it retires.

Tests

  • tests/unit/evaluation-run-shared-logic.test.ts (entities, 17 tests) runs every function on real API responses captured from a test stack: two finished auto runs on the same test set and a live run. It pins 522 ms for the app duration, reproduces the 261 ms /w printed and why, and checks the /w cell texts, the runs query body, the flags, the CSV escaping and the compare candidates.
  • runOverviewCharts.render.test.tsx (/m): a new test that the live card shows 95.45% next to a newest minute at 0%. It fails without the fix.
  • Types pass for @agenta/entities, @agenta/mobile and @agenta/oss (the package changed, so /w was checked too). Lint passes on the changed files.
  • Checked in the browser on /m against the test stack: the live card and the table both show 96.75% now; the auto run shows 522 ms.

/m without developer mode does not change: the only screen change is inside the Evaluations results page, which is a developer view (stage hidden) and only opens with beta developer mode or a deployment that forces it on.

What to QA

  • Open a live evaluation's Overview in /m (beta developer mode on). Each chart card's ": N%" equals the evaluator's row in the summary table.
  • Regression: open a finished auto evaluation's Overview. Duration (Avg) shows the app's mean (522 ms on the parity data), and the metric cards are unchanged.

…live card value

Copy the old app's run-level metric aggregation, metric cell text, runs
list filters, CSV export helpers and compare rules into
@agenta/entities/evaluationRun as pure functions, with tests on real API
responses. The invocation metrics come from the invocation step only,
which fixes the old app's 261 ms (an evaluator step averaged in).

The live overview card now shows the whole run's value, the same number
as the summary table, instead of the newest minute.

Entire-Checkpoint: 01M6R7H5Q39S1AW5SSYZ9Q1MBS
@coderabbitai

coderabbitai Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Website preview

Preview URL: https://pr-8164-agenta-website-preview.mahmoud-637.workers.dev

Built from 394fc2d0b2daf278a02af7cd94bc8e2dc26624cc. This comment updates in place on every push.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Railway Preview Environment

Item Value
Status stopped
Commit 394fc2d0b2daf278a02af7cd94bc8e2dc26624cc
Manual expiry (UTC) Not granted
CI deadline (UTC) Not granted
Workflow https://github.com/Agenta-AI/agenta/actions/runs/42222082711

Preview use finished or expired. Environment absence verified; retained storage requires separate provider accounting.

Post /preview as a new comment to request a one-hour preview. Repeating it while active does not extend the hour.
Preview data is disposable. Expiry is enforced by the next successful cleanup sweep, not an exact timer.
Environment deletion stops its compute. Provider-retained storage and permanent services may still be billed.

…ure user

formatMetricNumber dropped the zeros of whole numbers (100 -> "1"); it now
strips trailing zeros only after a decimal point. csvCell prefixes text that
starts with =, +, -, @, tab or carriage return with a quote, so a
spreadsheet does not run it as a formula. The fixtures use a made-up user id.

Entire-Checkpoint: 01M6RCPXCQ4AF87KFAZ32NNNHH
@mmabrouk

mmabrouk commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

🤖 The AI agent says: Fixed the batch review items in 394fc2d: formatMetricNumber keeps the zeros of whole numbers (100, 120 and 1000 no longer print as 1, 12 and 1); csvCell prefixes text that starts with =, +, -, @, tab or carriage return with a quote; the fixtures use a made-up user id. Tests cover each case.

@mmabrouk
mmabrouk changed the base branch from main to release/v0.128.9 November 6, 2026 14:32
@mmabrouk
mmabrouk merged commit 6879681 into release/v0.128.9 Nov 6, 2026
87 of 92 checks passed
@mmabrouk mmabrouk added the lgtm This PR has been approved by a maintainer label Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm This PR has been approved by a maintainer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant