You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[feat] Run a scenario's invocation from the /m evaluation run page - #8103
Pre-preview: run a scenario's invocation from the hidden /m evaluation run page.
Context
A human evaluation created in /m could not be annotated. Every scenario said "Run the invocation to generate output before annotating.", and /m had no Run button, so the user had to open the run in /w and press Run there for each scenario. The page is behind the evaluations developer view.
Changes
runScenarioInvocation({projectId, runId, scenarioId, stepKey}) is new in @agenta/playground. It does what the /w run action does for one scenario: it reads the app revision from the run's invocation step, loads the scenario's test case, calls the app with executeWorkflowRevision, stores the step result (POST /evaluations/results/ with the trace and span ids, the references and the outputs), marks the scenario failure when the app fails, refreshes the scenario and run metrics, and drops the cached scenario steps and run metrics. A second call for the same scenario while one runs joins the first one and sends nothing. It returns {ok: true} or {ok: false, reason, detail}; it never shows the raw error.
In the Annotate tab and the annotate sheet of a human run, a scenario without output now shows "Generate output to annotate" with a Run button where the form will be. While it runs: "Generating output..." and "Running...". On success: the toast "Invocation completed", then the output and the form. On failure: "Couldn't run this scenario" with "Run again" (the detail goes to the console). Cmd or Ctrl+Enter runs the app while the scenario has no output, and saves the form after. Nothing runs when a scenario opens, so browsing spends no model credits. Run needs edit_evaluation_results and is hidden on a closed run. /w has no "Run all", so there is none here.
The runner code loads only when Run is clicked. Size (first-load JS, gzip, against an origin/main build): app shell 955.2 KB to 955.3 KB, run details 1097.5 KB to 1097.6 KB. To get there, executeWorkflowRevision now calls the runner through executionItemController.helpers.executeStepForSession instead of importing executionRunner directly. The function is the same; /w behaves as before. The reason: the /m shell already holds the runner, merged into one module. A direct import from an on-demand chunk made Turbopack split that module, and the shell grew by 1.2 KB. A control build without the import confirmed this.
/w bugs found (not changed in /w; fixed in the shared function)
A successful "Run again" after a failure leaves the scenario failure. Now the scenario goes back to pending before the success is stored.
Run calls the app on a closed run, and the API then refuses the result. Now a closed run does not call the app.
An app answer without a trace id is stored as success, which leaves a scenario with no output to annotate and no Run button. Now it is stored as a failure.
When the test case does not load, /w calls the app with empty inputs. Now the call stops with request-failed.
/w shows the raw API error text in a toast for 8 seconds.
Tests
@agenta/playground: evaluationRunInvocation.test.ts runs the real executeWorkflowRevision and the entity code with only axios and fetch mocked. It asserts the app request inputs, the exact result upsert for success and failure, the scenario status writes, the metric refreshes, the in-flight guard, and the missing-reference, closed-run, no-trace and request-failure paths.
/m: runAnnotateTab.render.test.tsx adds 8 render tests with the real controllers and HTTP mocked: the Run panel, no request before Run, Run then annotate (exact upsert body), failure then Run again, permission, closed run, the keyboard, and the sheet.
Passed: /m and /w type checks, @agenta/playground tests (302), /m lint (0 errors), full /m suite except playgroundRevisionActions.render.test.tsx, which fails the same way on origin/main and passes alone on this branch. speed:check passes.
Live check on a fresh OSS dev stack with mistral-small-latest: created a human evaluation in /m, ran scenario 1 from the Annotate tab and scenario 2 from the row sheet, annotated both, reloaded. Outputs and annotations stayed, scenario 3 still showed Run, and the stored results have success and a trace id. No console errors.
Review: slop, simplify and correctness lenses, then a correctness pass on the final head (Low).
What to QA
Enable the evaluations developer view. Create a human evaluation in /m and open its Annotate tab. The first scenario shows "Generate output to annotate" and Run. Click Run: "Running...", then "Invocation completed", the output and the form. Annotate and save.
On the Scenarios tab, open a row's Annotate sheet for another scenario. Run works there too.
Reload. Annotated scenarios keep their output and values; the others still show Run.
Regression: a run that /w already invoked opens on the form, with no Run button.
…n page
Add runScenarioInvocation to @agenta/playground and a Run button in the
Annotate tab and the annotate sheet of a human run, so a run made in /m
can be annotated without opening /w.
Reviewing files that changed from the base of the PR and between 0550f9c and 0550f9c.
You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.
Use the checkbox below for a quick retry:
Trigger review
Summary
Summary by CodeRabbit
New Features
Run an app directly from an evaluation scenario when its output is pending, then annotate the generated result.
View progress while the app runs and retry after a failed run.
Use the annotation keyboard shortcut to run a pending app or submit annotations when output is ready.
Run controls respect permissions and are unavailable for closed evaluation runs.
The change adds on-demand execution for human invocation steps in evaluation runs. It stores scenario results and updates related metrics. The mobile annotate interface displays invocation status and gates annotation submission until output is available.
Changes: Human scenario invocation
Layer / File(s)
Summary
Execute and store invocation results web/packages/agenta-playground/src/state/evaluationRunInvocation.ts, web/packages/agenta-playground/src/state/controllers/executionItemController.ts, web/packages/agenta-playground/src/executeWorkflowRevision.ts, web/packages/agenta-playground/src/index.ts, web/packages/agenta-playground/tests/unit/evaluationRunInvocation.test.ts
The playground loads invocation references and testcase data, executes the workflow revision, and stores the result. It handles failed and closed runs, coalesces concurrent calls for the same scenario and step, refreshes metrics, and invalidates related queries. The tests cover these outcomes and result ID conversion.
Run action in the annotate flow web/mobile/src/features/evaluation-run/focus/RunAnnotateBody.tsx, web/mobile/src/features/evaluation-run/focus/RunAnnotateSheet.tsx, web/mobile/src/features/evaluation-run/focus/RunAnnotateTab.tsx, web/mobile/src/features/evaluation-run/focus/RunInvocationPanel.tsx, web/mobile/src/features/evaluation-run/focus/runScenarioInvocation.ts, web/mobile/src/features/evaluation-run/focus/useScenarioInvocation.ts, web/mobile/tests/unit/runAnnotateTab.render.test.tsx
The annotate interface uses invocation state to show a Run action, running status, or failure alert. It gates annotation submission while invocation is pending and supports running through the keyboard shortcut and annotation sheet. Tests cover permissions, closed runs, retries, and subsequent annotation.
sequenceDiagram
participant AnnotateTab as RunAnnotateTab
participant InvocationHook as useScenarioInvocation
participant InvocationRunner as runScenarioInvocation
participant EvaluationAPI
participant WorkflowRevision as executeWorkflowRevision
AnnotateTab->>InvocationHook: Run action for scenario
InvocationHook->>InvocationRunner: Load and call runner
InvocationRunner->>EvaluationAPI: Load run and testcase references
InvocationRunner->>WorkflowRevision: Execute revision with testcase data
WorkflowRevision-->>InvocationRunner: Return execution result
InvocationRunner->>EvaluationAPI: Store result and refresh metrics
InvocationRunner-->>InvocationHook: Return invocation result
InvocationHook-->>AnnotateTab: Update invocation state
Loading
Possibly related PRs
Agenta-AI/agenta#8042: Adds the evaluation-run details page and human-run Annotate tab that this change extends.
Agenta-AI/agenta#8048: Adds the scenario focus drawer and invocation fields used by the updated annotate flow.
Agenta-AI/agenta#8102: Changes the shared annotation controller used after invocation output becomes available.
Merge Risk: Low · up to 0550f
The invocation flow has no established user-facing failure, but its new backend calls should use the required Fern client before merge.
Pre-merge checks: 5
Check name
Status
Explanation
Title check
The title names the main change: running one scenario's invocation from the /m evaluation run page. The changeset adds RunInvocationPanel, useScenarioInvocation, and runScenarioInvocation, and wires a…
Description check
The description states the purpose (running a scenario's invocation from the hidden /m run page), the new runScenarioInvocation function, the UI states, the shared-runner fixes, the bundle-size reason…
Docstring Coverage
Docstring coverage is 66.67% which is sufficient. The required threshold is 60.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentati…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Title check
Explanation
The title names the main change: running one scenario's invocation from the /m evaluation run page. The changeset adds RunInvocationPanel, useScenarioInvocation, and runScenarioInvocation, and wires a Run button into the Annotate tab and annotate sheet, which matches the title.
Full details: Description check
Explanation
The description states the purpose (running a scenario's invocation from the hidden /m run page), the new runScenarioInvocation function, the UI states, the shared-runner fixes, the bundle-size reasoning, the tests, and QA steps. These all relate to the changeset.
Full details: Docstring Coverage
Explanation
Docstring coverage is 66.67% which is sufficient. The required threshold is 60.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 9 functions across 12 files.
Finishing Touches
Generate docstrings
Commit to this branch
Create a new PR
Generate unit tests (beta)
Commit to this branch
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
156-186: Trivial · Maintainability & Code Quality · Heavy lift
Use the Fern evaluations client for the new backend calls.
setScenarioStatus, upsertInvocationResult, and refreshMetrics call the shared axios instance with {params: {project_id}}. Frontend conventions require the Fern client for new API code. Query params go through the client's own argument shape, not {params}.
getEvaluationsClient() already exists in @agenta/sdk/resources. Import it from that subpath, not from the @agenta/sdk root. Add @agenta/sdk to web/packages/agenta-playground/package.json dependencies, using the same workspace pattern as the other entries. @agenta/entities already imports from @agenta/sdk/resources, so this does not add a new layer dependency.
Replace the three raw calls with the matching EvaluationsClient methods for the scenario patch, results upsert, and metrics refresh endpoints. Keep the try/catch around metrics refresh so a failure is still only logged. Update the axios mocks in evaluationRunInvocation.test.ts to mock the Fern client.
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 4 remain after this review.
Dependent tests finished. Post /preview for manual review. Environment absence verified; retained storage requires separate provider accounting.
Post /preview as a new comment to request a one-hour preview. Repeating it while active does not extend the hour.
Preview data is disposable. Expiry is enforced by the next successful cleanup sweep, not an exact timer.
Environment deletion stops its compute. Provider-retained storage and permanent services may still be billed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Pre-preview: run a scenario's invocation from the hidden /m evaluation run page.
Context
A human evaluation created in /m could not be annotated. Every scenario said "Run the invocation to generate output before annotating.", and /m had no Run button, so the user had to open the run in /w and press Run there for each scenario. The page is behind the
evaluationsdeveloper view.Changes
runScenarioInvocation({projectId, runId, scenarioId, stepKey})is new in@agenta/playground. It does what the /w run action does for one scenario: it reads the app revision from the run's invocation step, loads the scenario's test case, calls the app withexecuteWorkflowRevision, stores the step result (POST /evaluations/results/with the trace and span ids, the references and the outputs), marks the scenariofailurewhen the app fails, refreshes the scenario and run metrics, and drops the cached scenario steps and run metrics. A second call for the same scenario while one runs joins the first one and sends nothing. It returns{ok: true}or{ok: false, reason, detail}; it never shows the raw error.In the Annotate tab and the annotate sheet of a human run, a scenario without output now shows "Generate output to annotate" with a Run button where the form will be. While it runs: "Generating output..." and "Running...". On success: the toast "Invocation completed", then the output and the form. On failure: "Couldn't run this scenario" with "Run again" (the detail goes to the console). Cmd or Ctrl+Enter runs the app while the scenario has no output, and saves the form after. Nothing runs when a scenario opens, so browsing spends no model credits. Run needs
edit_evaluation_resultsand is hidden on a closed run. /w has no "Run all", so there is none here.The runner code loads only when Run is clicked. Size (first-load JS, gzip, against an origin/main build): app shell 955.2 KB to 955.3 KB, run details 1097.5 KB to 1097.6 KB. To get there,
executeWorkflowRevisionnow calls the runner throughexecutionItemController.helpers.executeStepForSessioninstead of importingexecutionRunnerdirectly. The function is the same; /w behaves as before. The reason: the /m shell already holds the runner, merged into one module. A direct import from an on-demand chunk made Turbopack split that module, and the shell grew by 1.2 KB. A control build without the import confirmed this./w bugs found (not changed in /w; fixed in the shared function)
failure. Now the scenario goes back topendingbefore the success is stored.success, which leaves a scenario with no output to annotate and no Run button. Now it is stored as a failure.request-failed.Tests
@agenta/playground:evaluationRunInvocation.test.tsruns the realexecuteWorkflowRevisionand the entity code with only axios andfetchmocked. It asserts the app request inputs, the exact result upsert for success and failure, the scenario status writes, the metric refreshes, the in-flight guard, and the missing-reference, closed-run, no-trace and request-failure paths.runAnnotateTab.render.test.tsxadds 8 render tests with the real controllers and HTTP mocked: the Run panel, no request before Run, Run then annotate (exact upsert body), failure then Run again, permission, closed run, the keyboard, and the sheet.@agenta/playgroundtests (302), /m lint (0 errors), full /m suite exceptplaygroundRevisionActions.render.test.tsx, which fails the same way on origin/main and passes alone on this branch.speed:checkpasses.mistral-small-latest: created a human evaluation in /m, ran scenario 1 from the Annotate tab and scenario 2 from the row sheet, annotated both, reloaded. Outputs and annotations stayed, scenario 3 still showed Run, and the stored results havesuccessand a trace id. No console errors.What to QA
evaluationsdeveloper view. Create a human evaluation in /m and open its Annotate tab. The first scenario shows "Generate output to annotate" and Run. Click Run: "Running...", then "Invocation completed", the output and the form. Annotate and save.