Skip to content

feat(rubric): check construction authority references - #207

Open
aleckhoury wants to merge 3 commits into
mainfrom
akhoury/ase-1376-ethos-confirmation-and-authority-checks
Open

aleckhoury wants to merge 3 commits into
mainfrom
akhoury/ase-1376-ethos-confirmation-and-authority-checks

Conversation

@aleckhoury

@aleckhoury aleckhoury commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

What this does

  1. Authority-reference checker. Task construction plans record, for every graded requirement, where its correct answer comes from. Today that is free text, so nothing checks it. A new read-only authority selection in the rubric's measure.py resolves each reference written as a locator against the selected files:

    • path, path#Heading (or its anchor), path#L3-L5, path#/json/pointer. A missing heading, line, or JSON value fails, and so does a path that doesn't exist, a symlinked or escaping path, or a file that changed since it was selected. A path that exists but wasn't selected is reported as unknown.
    • Headings match as GitHub renders them: any newline style or BOM, setext headings, links and code spans reduced to their text, and -1/-2 anchors for repeated headings. Front matter and fenced code don't count as headings.
    • Prose references stay valid but are reported as "not checkable", never as broken. A one-word reference containing / or . is read as a path.
    • A plan whose requirements or checks don't have the shape this check reads is reported as authority.plan_malformed rather than invalidating the whole measurement run.
    • It also counts requirements by authority basis and status, counts mapped checks, and flags checks that cite requirements that don't exist.

    The construction guide now asks authors to write locators. The plan schema is unchanged.

  2. Ethos confirmation comes from journey state. Eval Author's journey state already rechecks a saved Ethos confirmation against the current file and reopens review if it changed. The task contract scorecard now reads that through the read-only journey_state.py show instead of keeping a second copy of the check. Only an Ethos that passes the structure check from feat(rubric): assess task contracts and scoring authority #199, whose saved state.ethos.path is the selected Ethos, and whose state.ethos.stage is complete counts as owner intent. Without saved journey state, the Ethos is a claim to compare.

  3. Broken-case test fixtures. Five new cases in the rubric mutation panel:

    • Ethos valid and Ethos malformed, both using the real ethos-write validator.
    • Authority reference resolved, broken, and prose. Each of the last two keeps a second requirement that must still resolve.

Why

These close the W03 acceptance check: "malformed or stale documents and broken authority links fail the intended checks, while honest unknowns remain valid."

The authority checker also gives the scorecard deterministic counts for its two counting checks when a construction plan exists.

Limits

  • A resolved reference shows the cited content exists, not that it supports the requirement.
  • Tasks without a construction plan still rely on agent review for the crosswalk.

For reviewers

  • skills/eval-author/scripts/rubric_authority.py: the resolver.
  • tools/run_rubric_measurements.py and evals/rubric-measurements/cases.json: the new panel cases.
  • skills/eval-author-rubric/references/identity.md and task-contract-authority.md: the agent-facing contract.
  • skills/eval-author/references/construction-plan.md: the locator guidance.
  • docs/determinism-checks.md: user docs.
  • This PR doesn't include the ID rename from refactor(rubric): rename rubric criterion IDs by area and check method #206; it uses the current IDs.

Validation

  • Tests: uv run --locked python -m pytest -q -n auto tests/test_rubric_*.py tests/test_skill_contract.py tests/test_live_skill_evaluations.py tests/test_audit_*.py tests/test_reference_ownership.py reports 2661 passed. The full suite reports 6877 passed; its only 3 errors are tests/test_live_shard_apt.py cases that need native APT.
  • Panel: uv run --locked python tools/run_rubric_measurements.py --format text passes, with 13/13 evidence cases matched.
  • Journey state: in a scratch repository, journey_state.py show reported Ethos review complete after confirm-ethos. After the file was edited, it reported review with "ETHOS.md changed or is unavailable", and the state file was left unchanged.
  • Lint and types: ruff check, ruff format --check, ty check, make check-copyright-headers, and git diff --check pass.

Refs ASE-1376.

Compare Eval Author's saved Ethos confirmation with the selected document's
current bytes, and resolve construction-plan authority references written as
locators against selected artifacts. Prose references stay unknown, never
broken. Add handcrafted malformed, stale, and broken-reference cases to the
rubric mutation panel, each with an unaffected neighbor.

Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Drop the rubric's own Ethos confirmation comparison; Eval Author's journey
state already rechecks the saved confirmation against the current bytes. The
task contract scorecard reads it through the read-only journey_state.py show.

Signed-off-by: Alec Khoury <akhoury@nvidia.com>
@aleckhoury aleckhoury changed the title feat(rubric): check Ethos confirmation and authority references feat(rubric): check construction authority references Nov 13, 2026
Reject malformed check references as plan_malformed instead of crashing,
match headings as rendered (CRLF, BOM, setext, inline markup, repeated
anchors, no fenced code or front matter), fail missing and symlinked
paths, and document every authority code.

Signed-off-by: Alec Khoury <akhoury@nvidia.com>
@aleckhoury
aleckhoury marked this pull request as ready for review November 13, 2026 22:26
@aleckhoury
aleckhoury requested a review from a team as a code owner November 13, 2026 22:26
if not mapped:
diagnostics.append(diagnostic("authority.requirement_unmapped", row["id"], "unknown"))
rows = result["requirements"]
result["counts"] = {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's type this.

rechecks it against the current bytes. Only an Ethos that passes, whose
`state.ethos.path` is the selected Ethos path, and whose `state.ethos.stage` is
`complete` is owner intent. Otherwise, including when `show` reports no saved
state, it is a claim to compare.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: last sentence reads a little odd to me. might consider rephrasing?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants