Skip to content

refactor(rubric): rename rubric criterion IDs by area and check method #206

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
aleckhoury wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

refactor(rubric): rename rubric criterion IDs by area and check method #206

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/determinism-checks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,9 @@ that presentation guarantee does not establish deterministic workflow execution.

## Rubric identity measurements

The experimental rubric demo assesses source-bound task access (`DISC-01-A`),
explicit grading-path identification (`DISC-01-B`), and grading-dependency access
with a retained completeness review (`DISC-01-C`). Raw identity, route,
The experimental rubric demo assesses source-bound task access (`provenance.task-artifact-access`),
explicit grading-path identification (`provenance.grading-path`), and grading-dependency access
with a retained completeness review (`provenance.grading-dependency-access`). Raw identity, route,
configuration, relationship, and approval measurements remain separate from
canonical criterion scores. See the
[rubric measurement contract](../skills/eval-author-rubric/references/identity.md)
Expand Down Expand Up @@ -60,7 +60,7 @@ case IDs and omits success claims when any case mismatches. Correctly detecting
planted defect does not cause a nonzero exit.
It also checks that CLI assessment leaves the selected input bytes unchanged.

| Variant | DISC-01-A | DISC-01-B | DISC-01-C |
| Variant | provenance.task-artifact-access | provenance.grading-path | provenance.grading-dependency-access |
| --- | --- | --- | --- |
| Valid declared inventory | 2 | 2 | 2 |
| Wrong scorer | 2 | 0 | 2 |
Expand Down
5 changes: 5 additions & 0 deletions docs/results.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,11 @@ Rubric selections use `rubric_scope` for source-backed inventories. Package demo
observations use `retained_credit`, `retained_credit_controls`, and `coverage` to
identify their consumers. The provenance panel uses `rubric-provenance-demo-v1`
with `rubric-provenance-expectations-v1` expectations; the version remains 1.
Criterion IDs name their area, such as `provenance.grading-path` or
`task-contract.scoring-authority`, and each check ID adds its method, such as
`.deterministic-1` or `.llm-judge`. Result rows and check results keep the source
rubric's IDs in `source` and `source_id`; `skill_id` is no longer filled from the
source rubric.
Update integrations to these current names and rebuild older demo outputs rather
than relying on former field or schema names; no compatibility aliases are kept.
See the [rubric measurements](determinism-checks.md#rubric-identity-measurements)
Expand Down
66 changes: 33 additions & 33 deletions evals/rubric-measurements/cases.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,27 +32,27 @@
{"id":"missing-decision","mutation":"missing-decision","measurement":"approvals","status":"fail","codes":["approvals.decision_missing"]}
],
"check_outcomes": {
"DISC-01-A-C01": "PASS", "DISC-01-A-C02": "PASS", "DISC-01-A-C03": "NOT_RUN",
"DISC-01-B-C01": "PASS", "DISC-01-B-C02": "BLOCKED", "DISC-01-B-C03": "NOT_RUN", "DISC-01-B-C04": "NOT_RUN", "DISC-01-B-C05": "NOT_RUN",
"DISC-01-C-C01": "PASS", "DISC-01-C-C02": "PASS", "DISC-01-C-C03": "NOT_RUN", "DISC-01-C-C04": "NOT_RUN",
"DISC-02-A-C01": "BLOCKED", "DISC-02-A-C02": "PARTIAL", "DISC-02-A-C03": "NOT_RUN", "DISC-02-A-C04": "NOT_RUN",
"DISC-02-B-C01": "PARTIAL", "DISC-02-B-C02": "PARTIAL", "DISC-02-B-C03": "NOT_RUN",
"DISC-03-A-C01": "BLOCKED", "DISC-03-A-C02": "BLOCKED", "DISC-03-A-C03": "NOT_RUN",
"AUD-02-C01": "PARTIAL", "AUD-02-C02": "NOT_RUN", "AUD-02-C03": "NOT_RUN",
"ENV-01-C01": "NOT_RUN", "ENV-01-C02": "BLOCKED", "ENV-01-C03": "NOT_RUN",
"ENV-02-A-C01": "PARTIAL", "ENV-02-A-C02": "NOT_RUN", "ENV-02-A-C03": "PARTIAL",
"ENV-03-C01": "PARTIAL", "ENV-03-C02": "NOT_RUN", "ENV-03-C03": "NOT_RUN",
"CREATE-07-C01": "PARTIAL", "CREATE-07-C02": "PARTIAL", "CREATE-07-C03": "BLOCKED", "CREATE-07-C04": "BLOCKED"
"provenance.task-artifact-access.deterministic-1": "PASS", "provenance.task-artifact-access.deterministic-2": "PASS", "provenance.task-artifact-access.human-review": "NOT_RUN",
"provenance.grading-path.deterministic-1": "PASS", "provenance.grading-path.deterministic-2": "BLOCKED", "provenance.grading-path.human-review-1": "NOT_RUN", "provenance.grading-path.llm-judge": "NOT_RUN", "provenance.grading-path.human-review-2": "NOT_RUN",
"provenance.grading-dependency-access.deterministic-1": "PASS", "provenance.grading-dependency-access.deterministic-2": "PASS", "provenance.grading-dependency-access.human-review-1": "NOT_RUN", "provenance.grading-dependency-access.human-review-2": "NOT_RUN",
"provenance.declared-configuration.deterministic-1": "BLOCKED", "provenance.declared-configuration.deterministic-2": "PARTIAL", "provenance.declared-configuration.human-review-1": "NOT_RUN", "provenance.declared-configuration.human-review-2": "NOT_RUN",
"provenance.observed-configuration.deterministic-1": "PARTIAL", "provenance.observed-configuration.deterministic-2": "PARTIAL", "provenance.observed-configuration.probe": "NOT_RUN",
"provenance.runtime-route.deterministic-1": "BLOCKED", "provenance.runtime-route.deterministic-2": "BLOCKED", "provenance.runtime-route.probe": "NOT_RUN",
"provenance.comparison-fairness.deterministic": "PARTIAL", "provenance.comparison-fairness.human-review": "NOT_RUN", "provenance.comparison-fairness.probe": "NOT_RUN",
"provenance.solver-tool-access.human-review": "NOT_RUN", "provenance.solver-tool-access.deterministic": "BLOCKED", "provenance.solver-tool-access.probe": "NOT_RUN",
"provenance.dependency-substitution.deterministic-1": "PARTIAL", "provenance.dependency-substitution.human-review": "NOT_RUN", "provenance.dependency-substitution.deterministic-2": "PARTIAL",
"provenance.external-state.deterministic": "PARTIAL", "provenance.external-state.human-review": "NOT_RUN", "provenance.external-state.probe": "NOT_RUN",
"provenance.evidence-revision-consistency.deterministic-1": "PARTIAL", "provenance.evidence-revision-consistency.deterministic-2": "PARTIAL", "provenance.evidence-revision-consistency.deterministic-3": "BLOCKED", "provenance.evidence-revision-consistency.deterministic-4": "BLOCKED"
},
"check_overrides": {
"wrong-scorer": {"DISC-01-B-C01": "FAIL"},
"ambiguous-route": {"DISC-01-B-C01": "PARTIAL"},
"confirmed-missing": {"DISC-01-A-C02": "FAIL"},
"partial": {"DISC-01-B-C01": "PARTIAL", "DISC-01-C-C01": "PARTIAL", "DISC-01-C-C02": "PARTIAL"}
"wrong-scorer": {"provenance.grading-path.deterministic-1": "FAIL"},
"ambiguous-route": {"provenance.grading-path.deterministic-1": "PARTIAL"},
"confirmed-missing": {"provenance.task-artifact-access.deterministic-2": "FAIL"},
"partial": {"provenance.grading-path.deterministic-1": "PARTIAL", "provenance.grading-dependency-access.deterministic-1": "PARTIAL", "provenance.grading-dependency-access.deterministic-2": "PARTIAL"}
},
"evidence_check_overrides": {
"sourced_config": {"DISC-02-A-C01": "PARTIAL"},
"freshness": {"CREATE-07-C04": "PARTIAL"}
"sourced_config": {"provenance.declared-configuration.deterministic-1": "PARTIAL"},
"freshness": {"provenance.evidence-revision-consistency.deterministic-4": "PARTIAL"}
},
"evidence_cases": [
{
Expand Down Expand Up @@ -99,55 +99,55 @@
"measurement": "freshness", "status": "unknown", "codes": ["freshness.unresolved"], "affected_status": "unresolved", "changed_dependencies": ["ethos"]
}
],
"not_assessed": ["DISC-02-A", "DISC-02-B", "DISC-03-A", "AUD-02", "ENV-01", "ENV-02-A", "ENV-03", "CREATE-07"],
"not_assessed": ["provenance.declared-configuration", "provenance.observed-configuration", "provenance.runtime-route", "provenance.comparison-fairness", "provenance.solver-tool-access", "provenance.dependency-substitution", "provenance.external-state", "provenance.evidence-revision-consistency"],
"assessments": [
{
"id": "valid", "mutation": null, "cli_exit_code": 0,
"description": "Readable source-bound inventories, explicit strict route, and retained non-independent completeness review.",
"items": {
"DISC-01-A": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"},
"DISC-01-B": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"},
"DISC-01-C": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"}
"provenance.task-artifact-access": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"},
"provenance.grading-path": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"},
"provenance.grading-dependency-access": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"}
},
"diagnostics": {"identity": [], "route": []}
},
{
"id": "wrong-scorer", "mutation": "wrong-scorer", "cli_exit_code": 1,
"description": "Dispatch selects the weak claimed-success route instead of the intended scorer: a weak-route defect, not demonstrated exploitation.",
"items": {
"DISC-01-A": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"},
"DISC-01-B": {"score": 0, "status": "ASSESSED", "evidence_strength": "E2"},
"DISC-01-C": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"}
"provenance.task-artifact-access": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"},
"provenance.grading-path": {"score": 0, "status": "ASSESSED", "evidence_strength": "E2"},
"provenance.grading-dependency-access": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"}
},
"diagnostics": {"identity": [], "route": ["route.scorer_mismatch"]}
},
{
"id": "ambiguous-route", "mutation": "ambiguous-route", "cli_exit_code": 1,
"description": "The selector is absent; no scorer is guessed. Inventoried known grading branches remain readable.",
"items": {
"DISC-01-A": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"},
"DISC-01-B": {"score": null, "status": "NOT_ASSESSED", "evidence_strength": "E0"},
"DISC-01-C": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"}
"provenance.task-artifact-access": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"},
"provenance.grading-path": {"score": null, "status": "NOT_ASSESSED", "evidence_strength": "E0"},
"provenance.grading-dependency-access": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"}
},
"diagnostics": {"identity": [], "route": ["route.dynamic_unresolved"]}
},
{
"id": "confirmed-missing", "mutation": "confirmed-missing", "cli_exit_code": 1,
"description": "The selected task file is deleted after identity capture; an actual read confirms it is missing, unlike a missing-access declaration.",
"items": {
"DISC-01-A": {"score": 0, "status": "ASSESSED", "evidence_strength": "E2"},
"DISC-01-B": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"},
"DISC-01-C": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"}
"provenance.task-artifact-access": {"score": 0, "status": "ASSESSED", "evidence_strength": "E2"},
"provenance.grading-path": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"},
"provenance.grading-dependency-access": {"score": 2, "status": "ASSESSED", "evidence_strength": "E2"}
},
"diagnostics": {"identity": ["identity.artifact_missing"], "route": []}
},
{
"id": "partial", "mutation": "partial", "cli_exit_code": 1,
"description": "A readable task has an unresolved material input reference; the identified scorer has a declared but unresolved dependency. The retained review preserves both limits.",
"items": {
"DISC-01-A": {"score": 1, "status": "ASSESSED", "evidence_strength": "E2"},
"DISC-01-B": {"score": 1, "status": "ASSESSED", "evidence_strength": "E2"},
"DISC-01-C": {"score": 1, "status": "ASSESSED", "evidence_strength": "E2"}
"provenance.task-artifact-access": {"score": 1, "status": "ASSESSED", "evidence_strength": "E2"},
"provenance.grading-path": {"score": 1, "status": "ASSESSED", "evidence_strength": "E2"},
"provenance.grading-dependency-access": {"score": 1, "status": "ASSESSED", "evidence_strength": "E2"}
},
"diagnostics": {"identity": [], "route": ["route.dependency_missing"]}
}
Expand Down
6 changes: 3 additions & 3 deletions skills/eval-author-rubric/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,11 +71,11 @@ the area or criterion IDs explicitly requested by the user:
artifact, configuration, and provenance.
- **Artifact, configuration, and provenance:** use
[the canonical assessment](references/artifact-configuration-provenance.md)
for its original criterion IDs and item-specific anchors. Artifact, configuration,
for its `provenance.*` criteria and item-specific anchors. Artifact, configuration,
and provenance opt-in does not select anti-cheat.
- **Task contract and scoring authority:** use
[the task contract assessment](references/task-contract-authority.md) for
CREATE-03, CREATE-04, and CREATE-05-A. Selecting this area selects neither of
its `task-contract.*` criteria. Selecting this area selects neither of
the others, and selecting another area does not select it.

If the named-rubric request leaves the area ambiguous, clarify the scope once
Expand Down Expand Up @@ -151,7 +151,7 @@ controls. Preserve legitimate inaction and partial-credit semantics.

### 4. Present the scoped assessment

For each selected criterion, report its original ID, applicability, score or
For each selected criterion, report its ID, applicability, score or
explicit unassessed status, evidence strength, measurement, evidence references,
limitation, and next action.
Use that criterion's scoring rules. Do not transfer anti-cheat evidence floors to
Expand Down
Loading
Loading