You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(frontend): create and edit human evaluators in /m - #8146
In /m you could not create a human evaluator, and you could not edit one either. The Human tab of Evaluators had no create button, and a human row opened a read-only drawer that showed the raw output schema instead of the feedback fields. So a /m user could not set up a human evaluation or an annotation queue with new feedback. /w has a "Create new evaluator" drawer and an "Edit evaluator" drawer for this.
Changes
On the Human tab of Evaluators (a developer view), "Create new" now opens a form sheet. The empty tab offers "Create evaluator" too. The form has:
Evaluator name, slug (filled from the name until you edit it) and description.
One card per feedback field: name, type (Integer, Decimal, Boolean, Categorical single, Categorical multi, String), a minimum and maximum for numbers, an option list for choices (Enter adds the next option), and an Optional switch.
Errors next to each field after the first submit, and a taken slug under the slug field.
A human row, and the new "Edit" item in its row menu, open the same form filled from the evaluator. As in /w, the name and slug stay fixed, and Update waits for a change. Update saves a new revision with the new outputs schema. A reader without edit_evaluators still gets the read-only drawer.
The save uses the existing createHumanEvaluatorAtom and updateHumanEvaluatorAtom from @agenta/entities. Reading the feedback fields back from a stored schema used to live only in a /w component. It is now humanEvaluatorMetricsFromSchema in @agenta/entities/workflow, next to buildHumanEvaluatorOutputsSchema that writes them, with the feedback type list. /w keeps its own copy for now (this PR does not edit /w).
Automatic evaluators get no create button in this PR. Their create flow opens in the editable revision drawer, which comes in a separate PR.
This branch is based on main.
Tests
humanEvaluatorSheet.render.test.tsx drives the real create and update atoms with only HTTP faked, and checks the exact request bodies (workflow create, outputs schema commit; workflow update with meta and tags kept, revision commit).
humanEvaluatorForm.test.ts covers validation (names, slugs, ranges, options) and the payload.
humanEvaluatorMetrics.test.ts checks that every feedback type survives a write and a read.
evaluatorsScreen.render.test.tsx: Create new only on the Human tab and only with edit_evaluators, the empty-state create, a human row opening the edit form, and the read-only drawer without the permission.
Types for @agenta/mobile, @agenta/entities and @agenta/oss, and lint on the changed files.
Checked in the browser on a dev build against a test stack: create, edit and update a human evaluator at 1440 px and 390 px.
What to QA
Developer mode on, Evaluators, Human tab: "Create new" opens "Create human evaluator". Fill a name, add an Integer feedback with 1 to 5 and a Categorical (Single-choice) feedback with two options, Create. The row shows both feedback names.
Click that row: "Edit evaluator" opens filled in. Add a String feedback, Update. The row shows v2 with three feedback names.
Automatic tab: no "Create new" button yet.
Regression: a user without edit rights opens a human row and sees the read-only drawer, with no Create new.
No actionable comments were generated in the recent review.
Summary
Summary by CodeRabbit
New Features
Create and edit human evaluators from the evaluator list, including their name, description, and feedback fields.
Configure text, numeric, and choice-based feedback fields, with validation for required details and invalid values.
Find the create action in the empty state when evaluator creation is available.
Human evaluator rows offer an Edit action to users with editing access; others see Open details.
Bug Fixes
Feedback field errors are shown during entry, and duplicate slugs receive a specific error message.
The change adds human evaluator creation and editing. It introduces feedback metric schema parsing, a validated form and save flow, and entry points from the evaluator list for users who can edit evaluators.
Adds a shared feedback metric type and a parser that maps output-schema properties to metrics. Re-exports the parser and constants, and adds parser tests.
Draft mapping and validation web/mobile/src/features/evaluators/human/humanEvaluatorForm.ts, web/mobile/tests/unit/humanEvaluatorForm.test.ts
Adds draft initialization and schema reconstruction, field-type conversion, validation, metric conversion, and draft signatures. Tests cover validation and mapping behavior.
Feedback field and form UI web/mobile/src/features/evaluators/human/FeedbackFieldCard.tsx, web/mobile/src/features/evaluators/human/FeedbackOptionsField.tsx, web/mobile/src/features/evaluators/human/HumanEvaluatorForm.tsx
Adds controls for feedback names, types, numeric bounds, choice options, and optionality. The form handles validation, field changes, and submission.
Create and edit sheet web/mobile/src/features/evaluators/human/useSaveHumanEvaluator.ts, web/mobile/src/features/evaluators/human/HumanEvaluatorSheet.tsx, web/mobile/tests/unit/humanEvaluatorSheet.render.test.tsx
Adds create/update saving and a sheet that manages the form lifecycle, save results, success messages, and dismissal. Tests cover create, validation failure, duplicate slug, and edit flows.
Adds create and edit entry points for editable human evaluators. Human row actions open the edit sheet; other row selections continue to route to the revision view. Tests cover create visibility and row behavior.
sequenceDiagram
actor User
participant EvaluatorsScreen
participant EvaluatorSheets
participant HumanEvaluatorSheet
participant HumanEvaluatorForm
participant useSaveHumanEvaluator
User->>EvaluatorsScreen: Open create or edit
EvaluatorsScreen->>EvaluatorSheets: Set human sheet state
EvaluatorSheets->>HumanEvaluatorSheet: Pass open and editing state
HumanEvaluatorSheet->>HumanEvaluatorForm: Render form with draft
HumanEvaluatorForm->>HumanEvaluatorSheet: Submit valid draft
HumanEvaluatorSheet->>useSaveHumanEvaluator: Save draft
useSaveHumanEvaluator->>EvaluatorsScreen: Invalidate workflows query on success
useSaveHumanEvaluator->>HumanEvaluatorSheet: Return save result
HumanEvaluatorSheet->>User: Show success and close sheet
Loading
Possibly related PRs
Agenta-AI/agenta#8007: Adds the evaluator list, row menu, empty state, and read-only drawer extended by this change.
Agenta-AI/agenta#8149: Adds a create-evaluator link from the New Evaluation sheet to the category where this change adds human evaluator creation.
Merge Risk: Minimal · previous review f24ef
The previously reported edit blockers appear addressed, with no remaining issue established that should prevent merging after normal checks.
Pre-merge checks: 5
Check name
Status
Explanation
Title check
The title clearly describes the addition of human evaluator creation and editing in /m.
Description check
The description covers the human evaluator create and edit flows, permissions, shared schema parser, and tests.
Docstring Coverage
Docstring coverage is 100.00% which is sufficient. The required threshold is 60.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage
Explanation
Docstring coverage is 100.00% which is sufficient. The required threshold is 60.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 20 files.
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 0 remain after this review.
This block comment narrates behavior across five lines. The same pattern appears in useSaveHumanEvaluator.ts, HumanEvaluatorSheet.tsx, FeedbackOptionsField.tsx and humanEvaluatorMetrics.ts. Keep one short line per comment. Keep a longer comment only for a surprising constraint.
♻️ Proposed change
-/**- * The human evaluator form: name, slug and description, then the feedback fields. The slug- * follows the name until the reader edits it. Errors show after the first submit and then follow- * each change. In edit mode the name and the slug stay fixed, as in /w, and Update waits for a- * change.- */+/** The human evaluator create and edit form. */
As per coding guidelines: "Hard rule. At most ONE short line per comment. No multi-line blocks narrating why".
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 1 remain after this review.
Preview use finished or expired. Environment absence verified; retained storage requires separate provider accounting.
Post /preview as a new comment to request a one-hour preview. Repeating it while active does not extend the hour.
Preview data is disposable. Expiry is enforced by the next successful cleanup sweep, not an exact timer.
Environment deletion stops its compute. Provider-retained storage and permanent services may still be billed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Context
In /m you could not create a human evaluator, and you could not edit one either. The Human tab of Evaluators had no create button, and a human row opened a read-only drawer that showed the raw output schema instead of the feedback fields. So a /m user could not set up a human evaluation or an annotation queue with new feedback. /w has a "Create new evaluator" drawer and an "Edit evaluator" drawer for this.
Changes
On the Human tab of Evaluators (a developer view), "Create new" now opens a form sheet. The empty tab offers "Create evaluator" too. The form has:
A human row, and the new "Edit" item in its row menu, open the same form filled from the evaluator. As in /w, the name and slug stay fixed, and Update waits for a change. Update saves a new revision with the new outputs schema. A reader without
edit_evaluatorsstill gets the read-only drawer.The save uses the existing
createHumanEvaluatorAtomandupdateHumanEvaluatorAtomfrom@agenta/entities. Reading the feedback fields back from a stored schema used to live only in a /w component. It is nowhumanEvaluatorMetricsFromSchemain@agenta/entities/workflow, next tobuildHumanEvaluatorOutputsSchemathat writes them, with the feedback type list. /w keeps its own copy for now (this PR does not edit /w).Automatic evaluators get no create button in this PR. Their create flow opens in the editable revision drawer, which comes in a separate PR.
This branch is based on main.
Tests
humanEvaluatorSheet.render.test.tsxdrives the real create and update atoms with only HTTP faked, and checks the exact request bodies (workflow create, outputs schema commit; workflow update with meta and tags kept, revision commit).humanEvaluatorForm.test.tscovers validation (names, slugs, ranges, options) and the payload.humanEvaluatorMetrics.test.tschecks that every feedback type survives a write and a read.evaluatorsScreen.render.test.tsx: Create new only on the Human tab and only withedit_evaluators, the empty-state create, a human row opening the edit form, and the read-only drawer without the permission.@agenta/mobile,@agenta/entitiesand@agenta/oss, and lint on the changed files.What to QA