Skip to content

fix(rubric): standardize criterion identifiers and preserve source order - #205

Draft
asutermo wants to merge 1 commit into
mainfrom
fix/rubric-score-identifiers
Draft

asutermo wants to merge 1 commit into
mainfrom
fix/rubric-score-identifiers

Conversation

@asutermo

Copy link
Copy Markdown
Collaborator

Summary

  • Expand rubric prefixes to DISCOVERY, AUDIT, ENVIRONMENT, and ANTI-CHEAT. Keep CREATE and all numeric and letter suffixes.
  • Align canonical definitions, check IDs, scorer/report references, fixtures, and documentation. Preserve source order in CLI, demo, and retained reports.
  • Preserve all 11 provenance criteria, 38 checks, 44 score anchors, and numeric source references. Keep the four anti-cheat criteria agent-guided.
  • Version the renamed contract as rubric 4.1-draft, measurement output v3, and mutation-panel output v4. Scores remain 0 through 3 in the rubric; the bounded scorer still emits only 0 through 2 for the first three Discovery criteria.

Validation

  • 256 affected tests passed on the final tree. Both independent reviews passed.
  • Ruff lint and formatting, copyright headers, license checks, and Git whitespace checks passed.
  • Canonical JSON matched the baseline exactly after identifier/version mapping. Source numbering, score anchors, and order were unchanged.
  • Full local suite: 6 failed, 6860 passed, 126 skipped, and 76 subtests passed. All failures were in unchanged tests/test_verifier_cheat_evidence.py, which uses a 0.1-second subprocess timeout. Its 28 tests passed when rerun alone on both candidate and baseline main. The full suite is not reported as green.
  • No live model or native-runtime acceptance was performed. Existing unassessed criteria remain unassessed.

Signed-off-by: Andrew Suter-Morris <asutermorris@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant