Repository navigation
feat(rubric): check construction authority references - #207
Open
aleckhoury wants to merge 3 commits into
Open
aleckhoury wants to merge 3 commits into
aleckhoury wants to merge 3 commits into
Conversation
Compare Eval Author's saved Ethos confirmation with the selected document's current bytes, and resolve construction-plan authority references written as locators against selected artifacts. Prose references stay unknown, never broken. Add handcrafted malformed, stale, and broken-reference cases to the rubric mutation panel, each with an unaffected neighbor. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Drop the rubric's own Ethos confirmation comparison; Eval Author's journey state already rechecks the saved confirmation against the current bytes. The task contract scorecard reads it through the read-only journey_state.py show. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Reject malformed check references as plan_malformed instead of crashing, match headings as rendered (CRLF, BOM, setext, inline markup, repeated anchors, no fenced code or front matter), fail missing and symlinked paths, and document every authority code. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
aleckhoury
marked this pull request as ready for review
November 13, 2026 22:26
asutermo
reviewed
Nov 13, 2026
| if not mapped: | ||
| diagnostics.append(diagnostic("authority.requirement_unmapped", row["id"], "unknown")) | ||
| rows = result["requirements"] | ||
| result["counts"] = { |
asutermo
reviewed
Nov 13, 2026
| rechecks it against the current bytes. Only an Ethos that passes, whose | ||
| `state.ethos.path` is the selected Ethos path, and whose `state.ethos.stage` is | ||
| `complete` is owner intent. Otherwise, including when `show` reports no saved | ||
| state, it is a claim to compare. |
Collaborator
There was a problem hiding this comment.
nit: last sentence reads a little odd to me. might consider rephrasing?
asutermo
approved these changes
Nov 13, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Authority-reference checker. Task construction plans record, for every graded requirement, where its correct answer comes from. Today that is free text, so nothing checks it. A new read-only
authorityselection in the rubric'smeasure.pyresolves each reference written as a locator against the selected files:path,path#Heading(or its anchor),path#L3-L5,path#/json/pointer. A missing heading, line, or JSON value fails, and so does a path that doesn't exist, a symlinked or escaping path, or a file that changed since it was selected. A path that exists but wasn't selected is reported as unknown.-1/-2anchors for repeated headings. Front matter and fenced code don't count as headings./or.is read as a path.authority.plan_malformedrather than invalidating the whole measurement run.The construction guide now asks authors to write locators. The plan schema is unchanged.
Ethos confirmation comes from journey state. Eval Author's journey state already rechecks a saved Ethos confirmation against the current file and reopens review if it changed. The task contract scorecard now reads that through the read-only
journey_state.py showinstead of keeping a second copy of the check. Only an Ethos that passes the structure check from feat(rubric): assess task contracts and scoring authority #199, whose savedstate.ethos.pathis the selected Ethos, and whosestate.ethos.stageiscompletecounts as owner intent. Without saved journey state, the Ethos is a claim to compare.Broken-case test fixtures. Five new cases in the rubric mutation panel:
ethos-writevalidator.Why
These close the W03 acceptance check: "malformed or stale documents and broken authority links fail the intended checks, while honest unknowns remain valid."
The authority checker also gives the scorecard deterministic counts for its two counting checks when a construction plan exists.
Limits
For reviewers
skills/eval-author/scripts/rubric_authority.py: the resolver.tools/run_rubric_measurements.pyandevals/rubric-measurements/cases.json: the new panel cases.skills/eval-author-rubric/references/identity.mdandtask-contract-authority.md: the agent-facing contract.skills/eval-author/references/construction-plan.md: the locator guidance.docs/determinism-checks.md: user docs.Validation
uv run --locked python -m pytest -q -n auto tests/test_rubric_*.py tests/test_skill_contract.py tests/test_live_skill_evaluations.py tests/test_audit_*.py tests/test_reference_ownership.pyreports 2661 passed. The full suite reports 6877 passed; its only 3 errors aretests/test_live_shard_apt.pycases that need native APT.uv run --locked python tools/run_rubric_measurements.py --format textpasses, with 13/13 evidence cases matched.journey_state.py showreported Ethos reviewcompleteafterconfirm-ethos. After the file was edited, it reportedreviewwith "ETHOS.md changed or is unavailable", and the state file was left unchanged.ruff check,ruff format --check,ty check,make check-copyright-headers, andgit diff --checkpass.Refs ASE-1376.