An open-source integrity layer for AI-agent evaluations.
Every claim, backed by evidence.
Status: pre-release (0.0.x). The integrity core, the Inspect runner, signing, the comparison gate and one live, signed evidence chain are built; no benchmark result or performance claim is asserted yet. What remains before v0.1.0, and who it waits on, is in the release acceptance matrix; the design is the implementation plan.
On top of the runners teams already use (Inspect AI first; Harbor next), Aureproof certifies three things about an evaluation:
- The instrument — a per-task health record proves the grader works (its reference-positive, repaired-negative, no-op, and anti-cheat controls pass) before a task is allowed to count.
- The record — a signed, re-scorable evidence bundle: content-addressed artifacts, a hash-chained journal, an in-toto statement, and a Sigstore signature. Anyone can verify and re-score it offline.
- The decision — a pre-registered pass / fail / inconclusive regression gate with cluster-aware statistics, so a result that the data cannot support is reported as inconclusive rather than as a win.
Its flagship benchmark measures whether a security agent's findings are real — false alarms included — the problem the 2026 "AI slop" wave made concrete.
- In scope (v0.1): security-defect discovery on self-authored synthetic Python fixtures; structured findings graded by a deterministic label matcher in the trusted controller; offline, signed bundles; an A/B decision gate.
- Not yet: executing agent-submitted proof-of-concept code (v0.2), real-CVE tasks, patch grading, chat/RAG/voice, multi-agent systems, training.
- Synthetic fixtures measure controlled skills, not real-world capability.
Aureproof is designed to complement these projects, not replace them, and
to contribute fixes upstream. To our knowledge, the combination below is not
offered as one tool today; see docs/plan-review.md for
the cited landscape.
| Task-health records | Signed, re-scorable results | Pre-registered 3-way decision | |
|---|---|---|---|
| Inspect AI | admission checks | re-score yes, signing no | clustered SE only |
| Harbor / Terminal-Bench | admission checks | regrade yes, signing no | — |
| SEC-bench / CyberGym | — | — | — |
| Every Eval Ever | — | interchange, no signing | — |
| Aureproof (v0.1, pre-release) | persistent + exclusion | yes | yes |
Validate, re-score and report a sealed example bundle. No credentials, no network:
uv sync --frozen
uv run aureproof validate examples/bundles/tutorial-v0.1
uv run aureproof score examples/bundles/tutorial-v0.1 --offline
uv run aureproof report examples/bundles/tutorial-v0.1 --out report.htmlWalkthrough: docs/guides/tutorial.md.
Guides: authoring a task (task-health controls, licensing), reading results (verdicts, denominators, exclusions, refusals) and the comparison gate (pass, fail, inconclusive).
Requires uv and Python 3.12+.
uv sync # create the environment from the locked dependencies
uv run aureproof --version
uv run pytest # tests + coverage
uv run ruff check . # lint
uv run mypy # types
uv run lint-imports # architecture boundaries (plan v2 §3.2)Install the pre-commit hooks once: uvx pre-commit install.
docs/aureproof-plan-v2.md— the full plan (the spec this repo implements)docs/plan-review.md— the review and verified landscapedocs/scope.md— personas, goals, non-goalsdocs/adr/— architecture decision recordsROADMAP.md— milestones M0–M5docs/project-status.md— audited progress, effort assumptions, and execution order
See CONTRIBUTING.md. Commits are signed off under the
Developer Certificate of Origin. By
participating you agree to the Code of Conduct. Please
read RESPONSIBLE_USE.md before adding fixtures or tasks.
Report vulnerabilities privately per SECURITY.md.
Code is Apache-2.0. Self-authored fixtures are CC-BY-4.0. The Code of
Conduct is adapted from the Contributor Covenant 3.0 (CC-BY-SA-4.0). The
repository is REUSE-compliant; per-file terms are in
each file's SPDX header or in REUSE.toml.