Skip to content

feat(audit): audit-limits sanctions the failure types the register declares (R5) - #19

Open
afflom wants to merge 2 commits into
mainfrom
claude/template-r5-declared-failures
Open

afflom wants to merge 2 commits into
mainfrom
claude/template-r5-declared-failures

Conversation

@afflom

@afflom afflom commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Closes #18.

Change

cargo xtask audit-limits no longer carries a list of error types. A shipped crate may name an error type in a Result only when the register declares it.

A row of model/ids.toml declares a failure type with two optional fields, so the declaration is a field of an existing register row and no fourth register or schema is added (R1):

[[id]]
id = "OV-01"
level = "build"
suite = "overflow"
statement = "A value past the bound is refused as Overflow."
failure = "Overflow"     # the error type, as written in Result<_, E>
negative = "OV-02"       # the build row whose test provokes the refusal; may be this row's own ID

Model::check (so check-model, the meta-gate and audit-limits) refuses a malformed declaration: a failure without a negative or the reverse, a name that is not a bare type name, a type declared by two rows, a negative that is not a registered ID, a negative that is not a build row (an open claim is never asserted; a some-true claim is not established here). audit-limits then refuses:

  • a Result<_, E> whose E no row declares (the last path segment is compared, so crate::Overflow is Overflow; Box<dyn Error> is no declared name);
  • a Result that names no error type (io::Result<T>, Result<T>), since an alias hides the type;
  • a declared type with no test whose name ends in its negative ID, lowercased (the rule the meta-gate already uses).

Result<..> is read across lines and its arguments are split at the top level, so a signature rustfmt has broken, and Result<Vec<(u8, u8)>, E>, are read correctly. A type alias such as pub type R<T> = Result<T, Overflow>; is the one place Overflow is named; uses of R<T> carry no Result<.

NotAProduct, ObservedBound, KappaError: I searched the repository (grep -rn over every file but target/ and .git/). The only occurrence was the list in xtask/src/audit.rs. No register row, scenario or test uses them, and the template declares no failure of its own, so they are removed and not turned into rows.

The test-name scanner moves from crates/conformance/tests/bdd.rs to repo_conformance::workspace_test_names so that the meta-gate and the audit ask the same question of the same list. xtask gains the path dependency repo-conformance (both publish = false; no new external dependency; Cargo.lock gains one dependency edge).

R3 order

Two commits, in order:

  1. 5ebbb08 test(audit): the fixtures (12). Against the old audit 9 of the 12 fail; the 3 that pass are the cases the old audit already got right (a non-shipped crate, a comment, the template itself).
  2. 5d99a17 feat(audit): the implementation; all 12 pass. It also adds one more fixture, only_a_bare_type_name_is_a_declared_name, which passes on arrival.

The register rows of the fixtures live in the fixtures (each builds a throwaway repository root with its own model/, one shipped crate and, where needed, the negative test). The template's own model/ids.toml stays empty: a row in it would be copied into every repository cut from the template, and its scenario directory is likewise empty. So the "row, scenario, failing test" of R3 is a fixture register and failing fixtures, not rows in the template's register.

Fixtures of the audit (xtask/src/audit.rs, mod tests)

Case Test
a declared type with its negative test passes (also crate::Overflow and the alias form) a_declared_failure_type_passes
an undeclared type fails an_undeclared_failure_type_fails
a declared type without its negative test fails a_declared_type_without_its_negative_test_fails
negative test not a registered ID fails a_declared_type_whose_negative_test_is_not_registered_fails
negative test not a build row fails a_negative_test_is_a_build_claim
failure without negative, negative without failure, a type declared twice a_declaration_is_a_pair_and_is_made_once
the three former names are not sanctioned by the audit no_error_type_is_sanctioned_without_a_declaration
alias hiding the error type fails an_alias_that_hides_the_error_type_fails
multi-line Result, ok and bad a_result_broken_across_lines_is_read_whole
Box<dyn ..> and a fn-pointer arrow only_a_bare_type_name_is_a_declared_name
publish = false crate, comments and #[cfg(test)] stay out of scope a_crate_that_does_not_ship_is_not_audited, a_result_in_a_comment_or_a_test_module_is_not_audited
the template passes the_template_itself_passes

Validated

In /home/claude/agents/template-r5, with RUSTUP_TOOLCHAIN=stable (1.97.0; the pinned 1.97.1 cannot be downloaded here):

  • cargo test --workspace --locked: all pass (27 in xtask, of which 13 are new; 4 + 3 in repo-conformance, 1 in repo-model).
  • cargo fmt --all -- --check and cargo clippy --workspace --all-targets -- -D warnings: clean.
  • cargo run -p xtask -- check-model, audit-limits, audit-deferral, audit-dependency-updates: pass.
  • cargo run -p xtask -- audit-bootstrap and validate stop at prismpm.lock is required (no SDK here); validate runs check-model, audit-limits and audit-deferral before it. prismpm template check, prismpm lock check and cargo deny were not run (no SDK; cargo-deny is not installed). The bootstrap job fails at the locked SDK on every PR, as on main.

Planted defect

A shipped crate (crates/planted, no publish = false) in the real tree, planted alone, run, then removed:

Planted Command Diagnostic
pub fn f(x: u8) -> Result<u8, Overflow> with no register row cargo run -q -p xtask -- audit-limits (exit 1) gate failed: R5: every public failure is typed, modeled, documented, and covered by a negative test. ...
crates/planted/src/lib.rs:2: error type Overflow is not declared in model/ids.toml
rows OV-01 (failure = "Overflow", negative = "OV-02") and OV-02 added, no test same (exit 1) OV-01: failure Overflowhas negative test OV-02, but no test name ends inov_02``
negative = "OV-09" same (exit 1) model is inconsistent: OV-01: failure Overflow names negative test OV-09, which is not a registered ID (R5)
crates/planted/tests/negative.rs with #[test] fn refuses_past_the_bound_ov_02() and negative = "OV-02" same (exit 0) audit-limits: every error type a shipped crate names is declared with a negative test (1 declared) (R5)

The last row is the restored result. The planted crate and rows were removed afterwards; Cargo.lock carries only the new xtask edge. Recorded in TEMPLATE-VERIFICATION.md.

Not met

Owner decisions

  • Whether negative may name the declaring row itself (allowed here; the row's own test is then its negative test).
  • Whether a boundary-check exemption is wanted (above).
  • audit-limits now needs a loadable model/; a repository with no model/ fails it, as check-model does.

https://claude.ai/code/session_01F4DJnZorC2Jd4MJfN9DbZk


Generated by Claude Code

Alex Flom and others added 2 commits October 22, 2026 07:08
…the model declares

R3: the fixtures come before the implementation, and they fail against it.

Each fixture is a throwaway repository root with its own register, one
shipped crate and, where the case needs it, the negative test:

- a declared type with its negative test passes;
- an undeclared type fails, including NotAProduct, ObservedBound and
  KappaError, which the audit sanctioned by name and no register declares;
- a declared type without its negative test fails;
- a declared type whose negative test is not a registered ID, or is not a
  build claim, fails;
- a declaration is a (failure, negative) pair, made once;
- an alias that hides the error type fails; a Result broken across lines is
  read whole.

The rows live in the fixtures because the template's register is empty and a
row in it would be copied into every repository cut from the template.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F4DJnZorC2Jd4MJfN9DbZk
…clares

R5 says every public failure is typed, modeled, documented and covered by a
negative test; the audit enforced a fixed list of three names that are the
error types of one library. A row of model/ids.toml now declares a failure
type with two optional fields:

    failure   the error type as written in Result<_, E>
    negative  the ID of the build row whose test is its negative test

A shipped crate may name an error type in a Result only when a row declares
it, its negative test is a registered build row, and a test whose name ends in
that ID exists. A type is declared once. An undeclared type, an alias that
hides the error type, a declared type without its test, and a negative test
that is not registered or not a build claim are all refused, with the rule in
the diagnostic.

The three fixed names are removed: no register in this repository declares
them, and the template declares no failure of its own.

The audit reads a Result across lines, so a signature that rustfmt has broken
is read whole, and splits its arguments at the top level, so
Result<Vec<(u8, u8)>, E> has the error type E. The test-name scanner moves
from the BDD test to repo-conformance so that the meta-gate and the audit ask
the same question of the same list.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F4DJnZorC2Jd4MJfN9DbZk

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

audit-limits: sanction the failure types the model declares, not a fixed list (R5)

1 participant