You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(frontend): create evaluators from New evaluation, a new queue and trace Annotate in /m - #8197
In /m you could only pick evaluators that already existed. New evaluation had "Add evaluator" with no create, a new annotation queue had no "Create evaluator", and the trace Annotate sheet only offered "Open evaluators" when the project had no human evaluator. In /w each of these places can create an evaluator inline and comes back with it selected.
Changes
With beta developer mode on, each entry point can now create an evaluator and select it:
New annotation queue: the Feedback picker offers "Create evaluator", which opens the human evaluator form and adds the new evaluator to the queue.
Trace Annotate: the Select evaluators step gets a "Create evaluator" button, also in its empty state. The new evaluator is selected for the annotation.
How it fits together:
CreateEvaluatorFlow in features/evaluators/create runs either flow and hands back {workflowId, revisionId, name, slug, version}. It stops submit events from reaching the host's own form: React events bubble through portals, so without this the human form's Create also submitted the new queue form around it.
createHumanEvaluatorAtom in @agenta/entities now returns the created evaluator and its first revision instead of the slug only. /w ignores the return value, and its type check passes.
The automatic flow puts its evaluator in the playground state. The New evaluation sheet allows it only where the host says so: the Evaluations page does, the playground's New evaluation sheet does not (it offers human create only).
Without beta developer mode nothing changes at any entry point.
This PR is stacked on #8192 (playground evaluator picker), and merges #8146 (human evaluator form) in. Based on main.
Tests
createEvaluatorFlow.render.test.tsx: human and automatic flows hand back the new evaluator, the name comes from its workflow, and a submit inside the flow does not submit the host form (fails before the fix).
newEvaluationEvaluators.render.test.tsx: no create without beta mode; automatic create only where the host allows it; a created human evaluator is picked for the run.
annotationQueueEvaluatorField.render.test.tsx: no create without beta mode; a created evaluator is added to the queue.
selectEvaluatorsStep.render.test.tsx: no create without beta mode; create selects the new evaluator; the empty state offers create.
humanEvaluatorSheet.render.test.tsx: the real create atom, with HTTP faked, hands back the created evaluator.
Types for @agenta/mobile and @agenta/oss, lint on the changed files.
Checked in the browser on a dev build against a test stack: new queue, Create evaluator, the new evaluator added to Feedback; New Auto Evaluation, Create new evaluator, Exact Match in the drawer, Create, the new evaluator picked with its name.
What to QA
Beta developer mode on. Evaluations > Start evaluation > Auto evaluation, pick an app, open Evaluators > Add evaluator > "Create new evaluator". Pick a template, Create, name it. The drawer closes and the new evaluator shows as picked.
Same with Human evaluation: the human form opens; after Create the evaluator is picked.
Annotation queues > New queue > Add evaluator > "Create evaluator": after Create the evaluator shows under Feedback, and the queue form is not submitted.
Observability > a trace > Annotate > Select evaluators > "Create evaluator": the new evaluator is checked.
Regression: with beta developer mode off none of these create actions show.
Added a step-based evaluation setup experience with searchable tables for apps, revisions, test sets, and evaluators, including expandable revision history.
Added SDK evaluation setup with installation guidance, documentation links, and API key generation.
Added prompt creation from Chat, Completion, and Agent templates, plus prompt overview navigation.
Added revision selection when adding traces to an existing test set.
Added live-run query details and improved run metrics, status labels, and CSV export support.
Added trace-reference links and clearer filtered, empty, and rate-limited states in observability.
Bug Fixes
Improved app-renaming validation while names are loading, and prevented deployment before revision data is available.
Improved workspace invitation defaults and added a retry option when role loading fails.
This pull request revises evaluation creation, run summaries, observability, prompt workflows, and developer layouts. It also adds shared evaluation and observability utilities, testing guidance, and supporting tests.
Changes: Evaluation authoring and evaluator creation
Layer / File(s)
Summary
Evaluation sheet and selection tables web/mobile/src/features/evaluations/new/*, web/mobile/tests/unit/newEvaluation*, web/mobile/tests/support/newEvaluationMocks.tsx, web/storybook/stories/developer-views/NewEvaluationSheet.stories.tsx
The evaluation form changes from accordion sections and pickers to steps and searchable tables for apps, revisions, test sets, and evaluators. The sheet tracks draft input and asks for discard confirmation when it closes with changes.
Creation flows return evaluator details and connect new evaluators to annotation and evaluation selections. SDK evaluation selection opens a setup sheet with instructions, code samples, and API-key generation.
Evaluation run data and summaries web/packages/agenta-entities/src/evaluationRun/*, web/mobile/src/features/evaluation-run/*, web/mobile/src/features/evaluations/list/*, web/mobile/src/features/evaluations/live/*
Shared helpers cover metric aggregation and formatting, query display, run filtering, CSV export, and comparison. Run lists pass flags, live run summaries show query details and stopped status, and temporal metric cards use run-level stats.
Prompts, observability, and supporting UI
Layer / File(s)
Summary
Prompt creation and navigation web/mobile/src/features/prompts/*, web/mobile/tests/unit/createPrompt.render.test.tsx, web/mobile/tests/unit/prompts*
Prompt creation starts from Chat, Completion, or Agent templates. Prompt rows can open an app overview, and prompt renaming uses the shared app rename dialog.
Observability tables and references web/packages/agenta-observability-ui/src/*, web/mobile/src/features/observability/*, web/packages/agenta-ui/src/CellRenderers/*
Observability adds reference labels, filter icons, permission-gated trace export, URL-backed tabs, rate-limit and filtered-empty states, and theme-provided role colors.
Other workspace changes and guidance .agents/skills/test-design/SKILL.md, .claude/skills/test-design, .gitignore, web/AGENTS.md, web/mobile/AGENTS.md, web/mobile/src/features/developer/*, web/mobile/src/features/*/*Screen.tsx, web/mobile/src/styles/globals.css, web/oss/src/components/pages/settings/WorkspaceManage/Modals/*, web/packages/agenta-playground-ui/src/panels/drawer/*
The changes add test-design guidance, developer-mode page frame hooks, playground-scoped editor styles, invite-role loading behavior, and revision drawer state updates. Other edits adjust labels, loading states, and tests.
Live evaluation run metrics can display misleading values, and the new evaluator-creation flow has edge cases that can apply a creation twice or reopen the picker. Fix these before merging.
Pre-merge checks: 5
Check name
Status
Explanation
Title check
The title describes evaluator creation in the three changed entry points.
Description check
The description covers the evaluator creation flows and related tests in the changeset.
Docstring Coverage
Docstring coverage is 100.00% which is sufficient. The required threshold is 60.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage
Explanation
Docstring coverage is 100.00% which is sufficient. The required threshold is 60.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 2 functions across 50 files. (121 skipped: 10 unsupported, 111 over the file limit.)
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Preview use finished or expired. Environment absence verified; retained storage requires separate provider accounting.
Post /preview as a new comment to request a one-hour preview. Repeating it while active does not extend the hour.
Preview data is disposable. Expiry is enforced by the next successful cleanup sweep, not an exact timer.
Environment deletion stops its compute. Provider-retained storage and permanent services may still be billed.
Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.
Add a story for the automatic flow and the human flow. Cover the template picker's empty and error states so the new user-facing flow has the required state coverage.
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 1 remain after this review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Context
In /m you could only pick evaluators that already existed. New evaluation had "Add evaluator" with no create, a new annotation queue had no "Create evaluator", and the trace Annotate sheet only offered "Open evaluators" when the project had no human evaluator. In /w each of these places can create an evaluator inline and comes back with it selected.
Changes
With beta developer mode on, each entry point can now create an evaluator and select it:
How it fits together:
CreateEvaluatorFlowinfeatures/evaluators/createruns either flow and hands back{workflowId, revisionId, name, slug, version}. It stops submit events from reaching the host's own form: React events bubble through portals, so without this the human form's Create also submitted the new queue form around it.createHumanEvaluatorAtomin@agenta/entitiesnow returns the created evaluator and its first revision instead of the slug only. /w ignores the return value, and its type check passes.This PR is stacked on #8192 (playground evaluator picker), and merges #8146 (human evaluator form) in. Based on main.
Tests
createEvaluatorFlow.render.test.tsx: human and automatic flows hand back the new evaluator, the name comes from its workflow, and a submit inside the flow does not submit the host form (fails before the fix).newEvaluationEvaluators.render.test.tsx: no create without beta mode; automatic create only where the host allows it; a created human evaluator is picked for the run.annotationQueueEvaluatorField.render.test.tsx: no create without beta mode; a created evaluator is added to the queue.selectEvaluatorsStep.render.test.tsx: no create without beta mode; create selects the new evaluator; the empty state offers create.humanEvaluatorSheet.render.test.tsx: the real create atom, with HTTP faked, hands back the created evaluator.@agenta/mobileand@agenta/oss, lint on the changed files.What to QA