You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(frontend): open New Evaluation and Add to test set from the /m playground - #8110
#8075 left two playground links as seams, because their target code was not in its base: "New Evaluation" from the playground header, and "Add to test set" for the playground runs. Both targets are on main now: the /m new evaluation wizard (M7) and the /m add-to-test-set sheet (M10). This PR wires the two links. The playground is still a hidden developer view.
Changes
New Evaluation. The playground header has a "New Evaluation" button, as in /w. It opens the auto evaluation wizard with the app fixed and the revisions on screen already picked, in the order of the page. The Test set section opens first. A local draft is left out, because the server does not know its id. The button is disabled when only drafts are on screen, and hidden while the evaluations view is off (the started run opens in that view). The wizard waits until each picked revision and the variant names have loaded, so no chip opens with a placeholder label.
Add to test set. The run pane's test set menu has "Add to test set", as in /w. /w takes every successful run, finds the root span of its trace, and opens its trace drawer with those spans. /m does the same and opens the traces page sheet. The root span holds the run's inputs and outputs, so the sheet maps them like any trace. The item is disabled until a run succeeds, and hidden without edit_testsets. Two points differ from /w:
A run with repeats adds every repeat, not only the first one.
If the trace of one run cannot be read, no sheet opens and an error shows. /w opens the sheet with the runs it could read.
Both sheets load on demand. The run traces are read in the playground route itself: reading them inside the on-demand chunk made Turbopack split the app shell differently, which cost 1.2 KB on every page.
Size
First-load gzip against the #8075 head, same build machine: the app shell and the playground route grow 0.1 KB, every other page 0.0 to 0.2 KB. speed:check passes.
Tests
New playgroundNewEvaluation.render.test.tsx: the real playground selection, workflow stores and wizard, with HTTP mocked and the wizard's pickers as stubs (their popovers need pointer events). The wizard opens with exactly the shown revision ids, in order; a local draft is left out; the button is hidden while the view is off.
New playgroundAddToTestset.render.test.tsx: a real playground run, the real trace stores and the real sheet, with HTTP mocked and the sheet's search selects as native selects. The commit carries exactly the run's row; a run with two repeats adds two rows; without trace access no sheet opens; the item is disabled before a run and hidden without permission.
/m types pass, lint has 0 errors. Full /m suite: 2,230 to 2,234 of 2,240 pass over three runs. The 6 to 10 failures are all in playgroundRevisionActions (delete and discard) and one in playgroundCompare. The same tests fail on the feat(frontend): Playground share links, last selection and links in from other /m pages #8075 head without this change (checked locally: 10 of 37 in those two files fail with and without it).
Review
One Codex correctness round. It rated the first version Medium, with three findings, all fixed: a failed trace dropped a run without a word, repeats were left out, and a compare revision still loading could give a wrong chip label. Codex also noted that /w takes the span id from the run response first and the trace root only as a fallback. /m always takes the root span. I left this as is: the root span of a prompt app run holds its inputs and outputs, as the live check shows.
Live check
A fresh OSS dev stack with the four views on. From the playground: "New Evaluation", pick a test set and Exact Match, Start. The created run's application step references exactly the playground revision, and the run finishes. Then run one test case and use "Add to test set" on an existing test set. The new test set revision holds the row with the input and the output. The provider QA keys had no credit, so the runs used the stack's mock model.
What to QA
Turn on playground, evaluations and testsets.
Open an app's playground, compare two revisions, "New Evaluation". Both revisions show as picks. Start a run and open it from the toast.
Run two test cases, then Test set menu, "Add to test set". The preview shows both rows. Commit to a test set and open it.
Set repeats to 2 and run one test case. "Add to test set" previews two rows.
Added options to send successful playground runs, including repeated runs, to a test set.
Added a “New Evaluation” action to the playground for saved revisions when evaluations are available.
Bug Fixes
The add-to-queue picker no longer appears when there are no items to add.
Tests
Added coverage for adding runs to test sets, evaluation setup, and empty queue behavior.
The playground adds a New Evaluation action that initializes the evaluation wizard with saved revisions. It also adds a permission-gated action to add successful run traces to test sets. The Add to queue renderer now returns nothing when its item list is empty.
Changes: Playground actions
Layer / File(s)
Summary
Start evaluations from the playground web/mobile/src/features/playground/PlaygroundNewEvaluation.tsx, web/mobile/src/features/playground/PlaygroundNewEvaluationSheet.tsx, web/mobile/src/features/playground/PlaygroundWorkspace.tsx, web/mobile/tests/unit/playgroundNewEvaluation.render.test.tsx
The workspace header adds a New Evaluation action when the evaluations developer view is enabled. The sheet waits for workflow data and initializes the wizard with selected saved revisions. Tests cover revision selection, draft exclusion, and button visibility.
Add playground runs to test sets web/mobile/src/features/playground/playgroundRunTraces.ts, web/mobile/src/features/playground/PlaygroundTestsetMenu.tsx, web/mobile/src/features/playground/PlaygroundAddToTestset.tsx, web/mobile/tests/unit/playgroundAddToTestset.render.test.tsx
The menu collects trace IDs from successful runs when the project has edit permission. The sheet waits for trace summaries and root span IDs before opening. Tests cover commits, repeated runs, and permission states.
Hide the queue picker for empty item lists web/mobile/src/features/playground/usePlaygroundAddToQueue.tsx, web/mobile/tests/unit/playgroundAddToQueue.test.tsx
The queue renderer returns nothing when the item ID list is empty. A unit test covers this case.
sequenceDiagram
participant PlaygroundWorkspace
participant PlaygroundNewEvaluation
participant PlaygroundNewEvaluationSheet
participant NewEvaluationSheet
PlaygroundWorkspace->>PlaygroundNewEvaluation: Render with saved revision IDs
PlaygroundNewEvaluation->>PlaygroundNewEvaluationSheet: Open with selected revision IDs
PlaygroundNewEvaluationSheet->>NewEvaluationSheet: Pass initial picks after workflow data loads
Loading
sequenceDiagram
participant PlaygroundTestsetMenu
participant runTraceIds
participant PlaygroundAddToTestset
participant AddToTestsetSheet
PlaygroundTestsetMenu->>runTraceIds: Collect successful run trace IDs
PlaygroundTestsetMenu->>PlaygroundAddToTestset: Open with selected trace IDs
PlaygroundAddToTestset->>AddToTestsetSheet: Pass root span IDs after summaries load
Loading
Possibly related PRs
Agenta-AI/agenta#8056: Adds the sheet and span-loading flow that the playground action opens.
Agenta-AI/agenta#8062: Adds the test-set menu and optional action extended by this change.
Agenta-AI/agenta#8075: Adds the playground route, workspace, and existing queue integration extended here.
Merge Risk: Moderate · up to 54756
Runs without trace IDs can be omitted from a test set or leave the action disabled. Resolve trace handling before merging; loading feedback and incorrect labels remain bounded concerns.
Pre-merge checks: 5
Check name
Status
Explanation
Title check
The title clearly and concisely identifies both primary changes: opening New Evaluation and Add to test set from the playground.
Description check
The description directly explains the playground wiring, behavior, permissions, loading, tests, and QA steps for both features.
Docstring Coverage
No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage
Explanation
No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 10 files.
Finishing Touches
Generate docstrings
Commit to this branch
Create a new PR
Generate unit tests (beta)
Commit to this branch
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
9-14: Trivial · Maintainability & Code Quality · Low value
Shorten the multi-line block comment to one line.
The repository rule limits each comment to one short line. This block spans six lines of prose about why the component behaves as it does. The line break in the middle of a sentence also hurts readability.
♻️ Proposed fix
-/**- * "Add to test set" from the playground, as in /w: every finished run goes to the sheet as the- * root span of its trace, which holds the run's inputs and outputs. The sheet opens when every- * run's span is known. If one is missing,- * nothing opens: a sheet with some of the runs would add fewer rows than the page shows.- */+/** Opens the sheet only when every run's root span is known; a partial sheet would add fewer rows. */
As per coding guidelines: "At most ONE short line per comment. No multi-line blocks narrating why in prose."
Show loading feedback while the trace summaries are pending.
While any summary is pending, spanIds is null and the component renders nothing. The user picks "Add to test set" and sees no change until every trace loads. A slow trace read looks like a dead menu item, and the user may click again.
Render a loading state in this period. A toast or a skeleton sheet both work. Keep the existing error path for the failed case.
As per coding guidelines: "Every screen and every data-bearing component defines loading, empty, error ... states as first-class sibling components".
2-7: Trivial · Maintainability & Code Quality · Low value
Shorten the file header comment.
The repository rule limits each comment to one short line. This header is a six-line block. Reduce it to one line, for example: "Add to test set from the playground, over real run and trace stores; only HTTP is mocked."
As per coding guidelines: "At most ONE short line per comment."
The frontend conventions set a hard limit of one short line per comment. The new code in this file breaks that limit. Examples are the comment blocks at Lines 118-120, 141-142, 189-190 and 205-206. The new doc blocks in playgroundSnapshot.ts and tracePlayground.ts break it in the same way.
Keep a longer comment only where it explains a surprising constraint, such as a race or an ordering requirement. The restore token comment at Lines 141-142 and the "write mid-read" comment in syncSnapshotDrafts are the likely candidates. Reduce the other comments to one line each.
As per coding guidelines: "At most ONE short line per comment. No multi-line blocks narrating why, no restating what the code shows."
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 4 remain after this review.
Provisioning failed; no manual review time granted. Environment absence verified; retained storage requires separate provider accounting.
Post /preview as a new comment to request a one-hour preview. Repeating it while active does not extend the hour.
Preview data is disposable. Expiry is enforced by the next successful cleanup sweep, not an exact timer.
Environment deletion stops its compute. Provider-retained storage and permanent services may still be billed.
The reason will be displayed to describe this comment to others. Learn more.
Caution
Some comments are outside the diff and can’t be posted inline due to GitHub limitations.
⚠️ Outside diff range comments (2)
Major · Do not drop runs without a trace ID before the completeness check. · playgroundRunTraces.ts:15-27
web/mobile/src/features/playground/playgroundRunTraces.ts:15-27 Major · Data Integrity & Integration · Quick win
Do not drop runs without a trace ID before the completeness check.
runTraceIds removes runs whose traceId is missing. PlaygroundAddToTestset then compares the remaining span count with the remaining ID count, so it cannot detect the gap. A successful run or repeat can have no trace ID when the invoke response lacks trace_id and tree_id. The sheet can then open with only the other runs. The component's documented contract says it must show an error instead.
Keep the missing entries in the list. The component then sees a span count below the run count and shows the error.
Minor · Enable "Add to test set" when a run succeeds, without… · PlaygroundTestsetMenu.tsx:100-113
web/mobile/src/features/playground/PlaygroundTestsetMenu.tsx:100-113 Minor · Functional Correctness · Quick win
Enable "Add to test set" when a run succeeds, without requiring a trace ID.
PlaygroundTestsetMenu passes onAddToTestset whenever the user has edit_testsets. The item is disabled by model.hasSuccessfulResults. That predicate requires a truthy traceId, so a successful run with a null trace ID leaves the item disabled. The empty-list error path in PlaygroundAddToTestset is never reached. The PR intends the item to enable on success.
The predicate has no other consumer in the inspected code. The change affects only this menu item.
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 2 remain after this review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Release after v0.128.7. Merge #8075 first.
Context
#8075 left two playground links as seams, because their target code was not in its base: "New Evaluation" from the playground header, and "Add to test set" for the playground runs. Both targets are on main now: the /m new evaluation wizard (M7) and the /m add-to-test-set sheet (M10). This PR wires the two links. The playground is still a hidden developer view.
Changes
New Evaluation. The playground header has a "New Evaluation" button, as in /w. It opens the auto evaluation wizard with the app fixed and the revisions on screen already picked, in the order of the page. The Test set section opens first. A local draft is left out, because the server does not know its id. The button is disabled when only drafts are on screen, and hidden while the
evaluationsview is off (the started run opens in that view). The wizard waits until each picked revision and the variant names have loaded, so no chip opens with a placeholder label.Add to test set. The run pane's test set menu has "Add to test set", as in /w. /w takes every successful run, finds the root span of its trace, and opens its trace drawer with those spans. /m does the same and opens the traces page sheet. The root span holds the run's inputs and outputs, so the sheet maps them like any trace. The item is disabled until a run succeeds, and hidden without
edit_testsets. Two points differ from /w:Both sheets load on demand. The run traces are read in the playground route itself: reading them inside the on-demand chunk made Turbopack split the app shell differently, which cost 1.2 KB on every page.
Size
First-load gzip against the #8075 head, same build machine: the app shell and the playground route grow 0.1 KB, every other page 0.0 to 0.2 KB.
speed:checkpasses.Tests
playgroundNewEvaluation.render.test.tsx: the real playground selection, workflow stores and wizard, with HTTP mocked and the wizard's pickers as stubs (their popovers need pointer events). The wizard opens with exactly the shown revision ids, in order; a local draft is left out; the button is hidden while the view is off.playgroundAddToTestset.render.test.tsx: a real playground run, the real trace stores and the real sheet, with HTTP mocked and the sheet's search selects as native selects. The commit carries exactly the run's row; a run with two repeats adds two rows; without trace access no sheet opens; the item is disabled before a run and hidden without permission.playgroundRevisionActions(delete and discard) and one inplaygroundCompare. The same tests fail on the feat(frontend): Playground share links, last selection and links in from other /m pages #8075 head without this change (checked locally: 10 of 37 in those two files fail with and without it).Review
One Codex correctness round. It rated the first version Medium, with three findings, all fixed: a failed trace dropped a run without a word, repeats were left out, and a compare revision still loading could give a wrong chip label. Codex also noted that /w takes the span id from the run response first and the trace root only as a fallback. /m always takes the root span. I left this as is: the root span of a prompt app run holds its inputs and outputs, as the live check shows.
Live check
A fresh OSS dev stack with the four views on. From the playground: "New Evaluation", pick a test set and Exact Match, Start. The created run's application step references exactly the playground revision, and the run finishes. Then run one test case and use "Add to test set" on an existing test set. The new test set revision holds the row with the input and the output. The provider QA keys had no credit, so the runs used the stack's mock model.
What to QA
Turn on
playground,evaluationsandtestsets.