Skip to content

refactor(rubric): rename rubric criterion IDs by area and check method - #206

Draft
aleckhoury wants to merge 1 commit into
mainfrom
akhoury/rubric-area-method-ids
Draft

aleckhoury wants to merge 1 commit into
mainfrom
akhoury/rubric-area-method-ids

Conversation

@aleckhoury

@aleckhoury aleckhoury commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

⚠️ This PR is a rename

It renames every criterion and check ID in two experimental rubric areas, artifact, configuration, and provenance and task contract and scoring authority. Scoring rules, anchors, and checks are unchanged. Only the IDs and the way reference docs point at entries change.

Stacked on #199. Review that first; this PR's base switches to main when #199 merges.

Why

Requested in #199 review: the old IDs came from the source rubric's workflow numbering.

  • Criterion IDs didn't fit. CREATE-03 and skill_id: first_eval_create sit on criteria that apply to every task, whoever created it.
  • Check IDs didn't say what they were. C01 was the judge check for one criterion and C02 for another, so the instructions had to list checks one by one.
  • Reference docs linked by position (#/items/0), so reordering the JSON would silently break them.

What changes

  • Criterion IDs name their area: task-contract.scoring-authority, provenance.grading-path.
  • Check IDs add their method: .deterministic, .human-review, .llm-judge, .probe, with -1, -2 only when an item has more than one check of that method.
  • Source IDs are kept: each criterion has source: {id, skill_id} and each check has source_id, so earlier results and the planning docs still map. Result rows carry source; skill_id is no longer filled from the source rubric.
  • Docs link by ID: each reference section says which entry it describes instead of using a JSON position.
  • The task contract scoring rules got shorter now that check IDs carry the method ("the .llm-judge checks" instead of three listed IDs).

Criterion ID map

Old New
DISC-01-A provenance.task-artifact-access
DISC-01-B provenance.grading-path
DISC-01-C provenance.grading-dependency-access
DISC-02-A provenance.declared-configuration
DISC-02-B provenance.observed-configuration
DISC-03-A provenance.runtime-route
AUD-02 provenance.comparison-fairness
ENV-01 provenance.solver-tool-access
ENV-02-A provenance.dependency-substitution
ENV-03 provenance.external-state
CREATE-07 provenance.evidence-revision-consistency
CREATE-03 task-contract.clear-contract
CREATE-04 task-contract.discoverable-requirements
CREATE-05-A task-contract.scoring-authority

Check IDs follow the same pattern, for example CREATE-05-A-C02 → task-contract.scoring-authority.llm-judge and DISC-01-B-C03 → provenance.grading-path.human-review-1. The full map is in the source_id fields.

Impact

  • Breaking for retained experimental outputs: rubric demo and measurement outputs now use the new IDs. As with earlier rubric renames, rebuild old outputs; no aliases are kept (docs/results.md).
  • Mostly mechanical: of about 1,040 changed lines, ~430 are tests and ~320 are rubric JSON (the two definition files, panel expectations, and skill eval cases). Code changes are ID string swaps, plus one place that parsed a check ID suffix (endswith("C01")) and now compares the full ID.

Validation

  • CI: this tree is identical to 68d7b90, which passed CI on feat(rubric): assess task contracts and scoring authority #199 before the rename moved here.
  • Tests: uv run --locked python -m pytest -q -n auto tests/test_rubric_*.py tests/test_skill_contract.py tests/test_live_skill_evaluations.py tests/test_audit_*.py tests/test_reference_ownership.py tests/test_task_evidence.py tests/test_trace_loading.py reports 2725 passed.
  • Demos: tools/run_rubric_measurements.py and tools/run_rubric_provenance_demo.py pass with unchanged outcomes under the new IDs.
  • Structure tests: each check ID must name its item and method, source IDs must be unique, and reference docs may not use positional links.

Refs ASE-1376.

Rename the provenance and task contract criteria to area-based IDs whose
check IDs end with their method, for example
task-contract.scoring-authority.llm-judge. Keep the source rubric IDs in
source and source_id fields, and link reference sections by ID instead of
position.

Signed-off-by: Alec Khoury <akhoury@nvidia.com>
@aleckhoury
aleckhoury requested a review from a team as a code owner November 13, 2026 19:46
aleckhoury added a commit that referenced this pull request Nov 13, 2026
## What this does

Catches evaluation tasks that grade unfairly. If a task's grader checks
for something the instructions never asked for, a correct answer fails
and the score is wrong. This PR helps find those tasks.

1. **Audits check every task's grader against its instructions.** An
audit now reports:
   - a check the instructions never asked for, as "rejects valid work"
- an expected answer that contradicts its source, also as "rejects valid
work"
   - an expected answer with no stated basis, as an evidence gap
   - a stated requirement that nothing checks, as a weak grader

2. **New optional scorecard for task quality.** Only when a user
explicitly asks for it, the tool scores each task on three questions:
   - Are the instructions clear enough to grade fairly?
- Is everything that gets graded stated somewhere the tested agent can
see?
- Is each expected answer backed by something real, such as the
instructions or the input data, rather than "the sample solution
produced it"?

3. **Ethos documents are checked before they're trusted.** A new
read-only check runs the existing Ethos validator (the one that ships
with the Ethos writing skill, not a copy) on the selected `ETHOS.md`. It
records which document and which validator version were checked and
whether the document passed. The scorecard won't cite an Ethos that
fails the check or has changed.

## How it works, and its limits

- **The AI agent does the task review.** It reads the task files and
decides, following written instructions. The only deterministic piece is
the Ethos structure check; everything else is the model's judgment.
- **Every finding must cite evidence.** The grader line and the
instruction text, or proof the text is missing, so a person can verify
it quickly.
- **It can find problems but not approve.** A "this task is good" score
needs a person to sign off. Without that, the result is "not assessed".
- **Its accuracy hasn't been measured.** The tests in this PR check that
the instruction files are well formed, not that the model judges well.

## Test scenarios

- **Audit:** one new scenario with a small sample task that has a
deliberately planted unfair check (it demands an unrequested
`summary.json`). The tool should flag that check and nothing else.
- **Scorecard:** three new scenarios:
  - the same planted problem
  - a clean task the tool must not approve without a person's sign-off
  - a request for a different area, which must not trigger the scorecard
- **Case limit raised:** the live pipeline allowed at most 6 scenarios
per feature. It now allows 9. The scorecard grows from 6 to 9 scenarios
and the audit from 4 to 5. Time budget, concurrency and attempts are
unchanged.
- **Not yet run:** these scenarios run only in the live pipeline after
merge. If the time budget proves tight, drop the "different area"
scenario first, then the "clean task" one.

## For reviewers

- **Audit rule:**
`skills/eval-author-audit/references/coverage-report.md`
- **Scorecard:**
- `skills/eval-author-rubric/references/task-contract-authority.md`
(instructions)
  - `task-contract-authority.json` (definitions)
- The three criteria (CREATE-03, CREATE-04, CREATE-05-A) are copied
unchanged from the rubric JSON embedded in the "EA(U) de Rubrica" draft.
- Renaming the criterion and check IDs by area and method is a separate
PR, #206, stacked on this one.
- Shared scoring policies and the result format are reused from the
existing provenance area, not copied.
- **Ethos check:** `skills/eval-author/scripts/rubric_ethos.py`, wired
into `skills/eval-author-rubric/scripts/measure.py` as an optional
`ethos` selection plus `--ethos-validator <path>`. A valid result means
the structure is valid, not that the content was confirmed or says the
right things. The validator runs in-process with its stdout captured,
and a validator that exits or crashes leaves validity unknown.
- **Sample task:**
`skills/eval-author-audit/evals/files/tasks/order-total/`, with byte
hashes in `docs/harness-dataset-provenance.json`.

## Validation

- **Tests:** `uv run --locked python -m pytest -q -n auto
tests/test_rubric_*.py tests/test_skill_contract.py
tests/test_live_skill_evaluations.py tests/test_audit_*.py` reports 2628
passed.
- **Demos:** `tools/run_rubric_measurements.py` passes.
- **Smoke run:** `measure.py` with an `ethos` selection on a repository
`ETHOS.md` and the real validator reports `valid: true` with both
digests.
- **Pipeline plan without credentials:** 9 scorecard scenarios, 5 audit
scenarios, limit 9.
- **Hashes:** all sample-task hashes match the file bytes.
- **Lint and headers:** `ruff check`, `ruff format --check`, `git diff
--check` and `make check-copyright-headers` pass.
- **Not run:** `make check-licenses` (needs `osv-scanner`; no
dependencies changed).

Refs ASE-1376.
Base automatically changed from akhoury/ase-1376-w03-validate-ethos-and-task-contracts-with to main November 13, 2026 20:00
@aleckhoury
aleckhoury marked this pull request as draft November 13, 2026 20:02

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant