Repository navigation
refactor(rubric): rename rubric criterion IDs by area and check method - #206
Draft
aleckhoury wants to merge 1 commit into
Draft
aleckhoury wants to merge 1 commit into
aleckhoury wants to merge 1 commit into
Conversation
Rename the provenance and task contract criteria to area-based IDs whose check IDs end with their method, for example task-contract.scoring-authority.llm-judge. Keep the source rubric IDs in source and source_id fields, and link reference sections by ID instead of position. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
aleckhoury
added a commit
that referenced
this pull request
Nov 13, 2026
## What this does Catches evaluation tasks that grade unfairly. If a task's grader checks for something the instructions never asked for, a correct answer fails and the score is wrong. This PR helps find those tasks. 1. **Audits check every task's grader against its instructions.** An audit now reports: - a check the instructions never asked for, as "rejects valid work" - an expected answer that contradicts its source, also as "rejects valid work" - an expected answer with no stated basis, as an evidence gap - a stated requirement that nothing checks, as a weak grader 2. **New optional scorecard for task quality.** Only when a user explicitly asks for it, the tool scores each task on three questions: - Are the instructions clear enough to grade fairly? - Is everything that gets graded stated somewhere the tested agent can see? - Is each expected answer backed by something real, such as the instructions or the input data, rather than "the sample solution produced it"? 3. **Ethos documents are checked before they're trusted.** A new read-only check runs the existing Ethos validator (the one that ships with the Ethos writing skill, not a copy) on the selected `ETHOS.md`. It records which document and which validator version were checked and whether the document passed. The scorecard won't cite an Ethos that fails the check or has changed. ## How it works, and its limits - **The AI agent does the task review.** It reads the task files and decides, following written instructions. The only deterministic piece is the Ethos structure check; everything else is the model's judgment. - **Every finding must cite evidence.** The grader line and the instruction text, or proof the text is missing, so a person can verify it quickly. - **It can find problems but not approve.** A "this task is good" score needs a person to sign off. Without that, the result is "not assessed". - **Its accuracy hasn't been measured.** The tests in this PR check that the instruction files are well formed, not that the model judges well. ## Test scenarios - **Audit:** one new scenario with a small sample task that has a deliberately planted unfair check (it demands an unrequested `summary.json`). The tool should flag that check and nothing else. - **Scorecard:** three new scenarios: - the same planted problem - a clean task the tool must not approve without a person's sign-off - a request for a different area, which must not trigger the scorecard - **Case limit raised:** the live pipeline allowed at most 6 scenarios per feature. It now allows 9. The scorecard grows from 6 to 9 scenarios and the audit from 4 to 5. Time budget, concurrency and attempts are unchanged. - **Not yet run:** these scenarios run only in the live pipeline after merge. If the time budget proves tight, drop the "different area" scenario first, then the "clean task" one. ## For reviewers - **Audit rule:** `skills/eval-author-audit/references/coverage-report.md` - **Scorecard:** - `skills/eval-author-rubric/references/task-contract-authority.md` (instructions) - `task-contract-authority.json` (definitions) - The three criteria (CREATE-03, CREATE-04, CREATE-05-A) are copied unchanged from the rubric JSON embedded in the "EA(U) de Rubrica" draft. - Renaming the criterion and check IDs by area and method is a separate PR, #206, stacked on this one. - Shared scoring policies and the result format are reused from the existing provenance area, not copied. - **Ethos check:** `skills/eval-author/scripts/rubric_ethos.py`, wired into `skills/eval-author-rubric/scripts/measure.py` as an optional `ethos` selection plus `--ethos-validator <path>`. A valid result means the structure is valid, not that the content was confirmed or says the right things. The validator runs in-process with its stdout captured, and a validator that exits or crashes leaves validity unknown. - **Sample task:** `skills/eval-author-audit/evals/files/tasks/order-total/`, with byte hashes in `docs/harness-dataset-provenance.json`. ## Validation - **Tests:** `uv run --locked python -m pytest -q -n auto tests/test_rubric_*.py tests/test_skill_contract.py tests/test_live_skill_evaluations.py tests/test_audit_*.py` reports 2628 passed. - **Demos:** `tools/run_rubric_measurements.py` passes. - **Smoke run:** `measure.py` with an `ethos` selection on a repository `ETHOS.md` and the real validator reports `valid: true` with both digests. - **Pipeline plan without credentials:** 9 scorecard scenarios, 5 audit scenarios, limit 9. - **Hashes:** all sample-task hashes match the file bytes. - **Lint and headers:** `ruff check`, `ruff format --check`, `git diff --check` and `make check-copyright-headers` pass. - **Not run:** `make check-licenses` (needs `osv-scanner`; no dependencies changed). Refs ASE-1376.
Base automatically changed from
akhoury/ase-1376-w03-validate-ethos-and-task-contracts-with
to
main
November 13, 2026 20:00
aleckhoury
marked this pull request as draft
November 13, 2026 20:02
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
It renames every criterion and check ID in two experimental rubric areas, artifact, configuration, and provenance and task contract and scoring authority. Scoring rules, anchors, and checks are unchanged. Only the IDs and the way reference docs point at entries change.
Stacked on #199. Review that first; this PR's base switches to
mainwhen #199 merges.Why
Requested in #199 review: the old IDs came from the source rubric's workflow numbering.
CREATE-03andskill_id: first_eval_createsit on criteria that apply to every task, whoever created it.C01was the judge check for one criterion andC02for another, so the instructions had to list checks one by one.#/items/0), so reordering the JSON would silently break them.What changes
task-contract.scoring-authority,provenance.grading-path..deterministic,.human-review,.llm-judge,.probe, with-1,-2only when an item has more than one check of that method.source: {id, skill_id}and each check hassource_id, so earlier results and the planning docs still map. Result rows carrysource;skill_idis no longer filled from the source rubric..llm-judgechecks" instead of three listed IDs).Criterion ID map
DISC-01-Aprovenance.task-artifact-accessDISC-01-Bprovenance.grading-pathDISC-01-Cprovenance.grading-dependency-accessDISC-02-Aprovenance.declared-configurationDISC-02-Bprovenance.observed-configurationDISC-03-Aprovenance.runtime-routeAUD-02provenance.comparison-fairnessENV-01provenance.solver-tool-accessENV-02-Aprovenance.dependency-substitutionENV-03provenance.external-stateCREATE-07provenance.evidence-revision-consistencyCREATE-03task-contract.clear-contractCREATE-04task-contract.discoverable-requirementsCREATE-05-Atask-contract.scoring-authorityCheck IDs follow the same pattern, for example
CREATE-05-A-C02→task-contract.scoring-authority.llm-judgeandDISC-01-B-C03→provenance.grading-path.human-review-1. The full map is in thesource_idfields.Impact
docs/results.md).endswith("C01")) and now compares the full ID.Validation
68d7b90, which passed CI on feat(rubric): assess task contracts and scoring authority #199 before the rename moved here.uv run --locked python -m pytest -q -n auto tests/test_rubric_*.py tests/test_skill_contract.py tests/test_live_skill_evaluations.py tests/test_audit_*.py tests/test_reference_ownership.py tests/test_task_evidence.py tests/test_trace_loading.pyreports 2725 passed.tools/run_rubric_measurements.pyandtools/run_rubric_provenance_demo.pypass with unchanged outcomes under the new IDs.Refs ASE-1376.