diff --git a/CHANGELOG.md b/CHANGELOG.md index 9284e13f..607fcc10 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,7 +13,7 @@ Format: [Semantic Versioning](https://semver.org/). Spec versions follow `MAJOR. ### Added -- **`build_provenance` now declares verification depth.** A new optional `provenance_depth` (`surface`, `builder`, `transitive`) says how far down the supply chain the issuer claims to have walked, and a new optional `appraisal.provenance_depth_verified` records how far the verifier actually walked. Spec section 3.3 step 7 previously left three stopping points equally conformant, so two verifiers could reach opposite conclusions on the same record with no way to say why. Both fields are optional and a record omitting `provenance_depth` is read as `surface`, so existing records keep their meaning. Resolves [#50](https://github.com/agentrust-io/trace-spec/issues/50). +- **`build_provenance` now declares verification depth.** A new optional `provenance_depth` (`surface`, `builder`, `transitive`) says how far down the supply chain the issuer claims to have walked, and a new optional `appraisal.provenance_depth_verified` records how far the verifier actually walked. Spec section 3.3 step 7 previously left three stopping points equally conformant, so two verifiers could reach opposite conclusions on the same record with no way to say why. Both fields are optional and a record omitting `provenance_depth` is read as `surface`, so existing records keep their meaning. Evidence that does not resolve and evidence that resolves and contradicts the record are separate outcomes: the first downgrades the recorded depth and names what was missing, the second fails the appraisal and cannot be downgraded away. Resolves [#50](https://github.com/agentrust-io/trace-spec/issues/50). ### Security diff --git a/docs/build-provenance-depth.md b/docs/build-provenance-depth.md index b88caa6b..94df8436 100644 --- a/docs/build-provenance-depth.md +++ b/docs/build-provenance-depth.md @@ -66,9 +66,9 @@ Nor does it assure the artifact was built from the source you think. A trusted b |---|---|---| | Surface | The artifact digest and a trusted-builder list | Nothing. It always runs. | | Builder-chain | Attestation retrieval at verification time, or a cached bundle | A record whose `provenance_uri` is absent or unresolvable cannot be verified past surface | -| Dependency-chain | A publisher attestation per input, and ecosystem coverage for those inputs | Inputs from ecosystems without attestation coverage fail, benign ones included | +| Dependency-chain | A publisher attestation per input, and ecosystem coverage for those inputs | Inputs from ecosystems without attestation coverage cannot be verified past builder, benign ones included | -Deeper is not free, and the honest reason deployments stop early is usually not laziness. Dependency-chain verification fails closed on unattested inputs, and attestation coverage across package ecosystems is uneven, so a strict verifier rejects a lot of software that is fine. +Deeper is not free, and the honest reason deployments stop early is usually not laziness. An unattested input is unverifiable rather than incriminating, so [verification.md](verification.md) has the verifier record `builder` and name what it could not fetch; a deployment whose floor is `transitive` then treats that record as `contraindicated`. Attestation coverage across package ecosystems is uneven, so such a deployment turns away a lot of software that is fine — the strictness lives in the floor, not in a finding against the record. That trade-off is a legitimate choice. Picking Surface because it was the cheapest to implement, without recording that "trusted builder" is then a claim nobody checked, is not the same choice. diff --git a/docs/verification.md b/docs/verification.md index cacf06d3..60cf544c 100644 --- a/docs/verification.md +++ b/docs/verification.md @@ -140,15 +140,33 @@ have walked. A verifier MUST record what it actually checked in `appraisal.provenance_depth_verified`, which is a statement about the verifier, not about the record. -| Claimed depth | Verifier MUST | Verifier MAY downgrade to | -|---|---|---| -| `surface` (or absent) | Confirm `digest` matches the workload artifact and `builder` resolves to the configured trusted-builder set. | Already the floor | -| `builder` | All of surface, plus fetch `provenance_uri`, verify the SLSA attestation signature, check the attestation `subject` matches `digest`, and check the attestation `builder.id` matches `builder`. | `surface`, when `provenance_uri` is unreachable or its signature does not resolve. The appraisal MUST then record the lower depth rather than a failure of the higher one. | -| `transitive` | All of builder, plus enumerate the SLSA `materials` / `resolvedDependencies` and confirm every entry has a verifiable publisher attestation (npm OIDC, PyPI Trusted Publisher, Sigstore Rekor entry, or platform equivalent). | `builder`, when transitive coverage is unavailable for any input, or `surface` per the row above. | +| Claimed depth | Verifier MUST | MAY downgrade to — evidence does not resolve | MUST fail — evidence resolves and contradicts | +|---|---|---|---| +| `surface` (or absent) | Confirm `digest` matches the workload artifact and `builder` resolves to the configured trusted-builder set. | Already the floor. | `digest` does not match the artifact the verifier independently holds, or `builder` is outside the trusted-builder set. | +| `builder` | All of surface, plus fetch `provenance_uri`, verify the SLSA attestation signature, check the attestation `subject` matches `digest`, and check the attestation `builder.id` matches `builder`. | `surface`, when `provenance_uri` is absent or unreachable, or its signature does not resolve. | The attestation resolves and its `subject` does not match `digest`, or its `builder.id` does not match `builder`. | +| `transitive` | All of builder, plus enumerate the SLSA `materials` / `resolvedDependencies` and confirm every entry has a verifiable publisher attestation (npm OIDC, PyPI Trusted Publisher, Sigstore Rekor entry, or platform equivalent). | `builder`, when an input carries no publisher attestation or the attestation declares no inputs at all; or `surface` per the row above. | An input's publisher attestation resolves and was signed under an issuer outside the configured trusted set. | + +The two right-hand columns are disjoint, and which one applies turns on whether the evidence +resolved — not on how serious the finding is. + +**Evidence that does not resolve** leaves a check unrun. The verifier MAY stop at the depth +below, MUST then record that lower depth in `appraisal.provenance_depth_verified`, and MUST +NOT report the missing evidence as a failure of the record. A record is not defective because +someone else's transparency log is unreachable, and a verifier that rejects on this is failing +records for the weather. + +**Evidence that resolves and contradicts the record** MUST fail the appraisal. A verifier MUST +NOT downgrade to escape it. Downgrading there would record a narrower claim that is true while +suppressing a wider one that is false — the record would pass as `builder` on evidence that +positively refutes it at `transitive`, and the appraisal would say nothing about why. A verifier MUST NOT record `provenance_depth_verified` at a depth higher than it executed. Downgrading is how a verifier stays honest when evidence does not resolve; claiming depth it did -not run is what the field exists to prevent. +not run is what the field exists to prevent. This last rule cannot be expressed in JSON Schema: +the record is byte-identical whether the verifier walked the chain or merely says it did. The +conformance vectors in +[`examples/build-provenance-depth/`](https://github.com/agentrust-io/trace-spec/tree/main/examples/build-provenance-depth) +hold it instead, against a verifier's own output, and encode the split above vector by vector. Records that omit `provenance_depth` MUST be treated as `surface`. This keeps every record issued before this field existed valid and correctly interpreted. @@ -176,6 +194,22 @@ attestation does not chain back to the legitimate maintainer. Without a recorded conformant verifiers reach opposite conclusions on the same record and neither says why, which is the federation gap [section 1](../spec/trace-v0.2.md) names. +That case is a failure and not a downgrade, and it is the sharpest reason the two are kept +apart. The poisoned input's attestation resolved: the verifier holds it and can see the issuer +is outside the trusted set. A verifier permitted to call that "transitive coverage unavailable" +would record `builder`, accept, and report exactly what a verifier that never looked reports — +which would make the depth field cover for the attack it was added to expose. + +### `transitive` is a floor on effort, not a comparable claim + +Until evidence resolution is standardized, two verifiers can both honestly record `transitive` +over different material sets. Nothing above specifies which inputs must be enumerated or where a +publisher attestation must be looked up, so the value states how far a verifier walked, not what +ground it covered. `builder` has no such gap, because `provenance_uri` names its own evidence. +A consumer comparing `transitive` across verifiers is therefore comparing effort; it does not +license the inference that the same dependencies were checked. Specifying a transitive coverage +URI is left to a follow-up. + [Build provenance depth](build-provenance-depth.md) is the informative companion to this section: it states what each depth does not assure. diff --git a/examples/build-provenance-depth/01-all-depths-accept.json b/examples/build-provenance-depth/01-all-depths-accept.json index b2c7b293..9bdc783a 100644 --- a/examples/build-provenance-depth/01-all-depths-accept.json +++ b/examples/build-provenance-depth/01-all-depths-accept.json @@ -118,15 +118,21 @@ "expected": { "surface": { "outcome": "accept", - "failures": [] + "verified_depth": "surface", + "failures": [], + "unresolved": [] }, - "builder_chain": { + "builder": { "outcome": "accept", - "failures": [] + "verified_depth": "builder", + "failures": [], + "unresolved": [] }, - "dependency_chain": { + "transitive": { "outcome": "accept", - "failures": [] + "verified_depth": "transitive", + "failures": [], + "unresolved": [] } } } diff --git a/examples/build-provenance-depth/02-surface-accepts-attestation-subject-mismatch.json b/examples/build-provenance-depth/02-surface-accepts-attestation-subject-mismatch.json index f0b37be3..d117e615 100644 --- a/examples/build-provenance-depth/02-surface-accepts-attestation-subject-mismatch.json +++ b/examples/build-provenance-depth/02-surface-accepts-attestation-subject-mismatch.json @@ -118,19 +118,25 @@ "expected": { "surface": { "outcome": "accept", - "failures": [] + "verified_depth": "surface", + "failures": [], + "unresolved": [] }, - "builder_chain": { + "builder": { "outcome": "reject", + "verified_depth": "builder", "failures": [ "attestation_subject_mismatch" - ] + ], + "unresolved": [] }, - "dependency_chain": { + "transitive": { "outcome": "reject", + "verified_depth": "transitive", "failures": [ "attestation_subject_mismatch" - ] + ], + "unresolved": [] } } } diff --git a/examples/build-provenance-depth/03-surface-accepts-attestation-builder-mismatch.json b/examples/build-provenance-depth/03-surface-accepts-attestation-builder-mismatch.json index bdce19ea..52bb1ce5 100644 --- a/examples/build-provenance-depth/03-surface-accepts-attestation-builder-mismatch.json +++ b/examples/build-provenance-depth/03-surface-accepts-attestation-builder-mismatch.json @@ -118,19 +118,25 @@ "expected": { "surface": { "outcome": "accept", - "failures": [] + "verified_depth": "surface", + "failures": [], + "unresolved": [] }, - "builder_chain": { + "builder": { "outcome": "reject", + "verified_depth": "builder", "failures": [ "attestation_builder_mismatch" - ] + ], + "unresolved": [] }, - "dependency_chain": { + "transitive": { "outcome": "reject", + "verified_depth": "transitive", "failures": [ "attestation_builder_mismatch" - ] + ], + "unresolved": [] } } } diff --git a/examples/build-provenance-depth/04-builder-chain-accepts-dependency-unattested.json b/examples/build-provenance-depth/04-builder-accepts-dependency-unattested.json similarity index 88% rename from examples/build-provenance-depth/04-builder-chain-accepts-dependency-unattested.json rename to examples/build-provenance-depth/04-builder-accepts-dependency-unattested.json index de0fbf4b..4b28afd4 100644 --- a/examples/build-provenance-depth/04-builder-chain-accepts-dependency-unattested.json +++ b/examples/build-provenance-depth/04-builder-accepts-dependency-unattested.json @@ -1,6 +1,6 @@ { - "name": "builder-chain-accepts-dependency-unattested", - "description": "Subject and builder both bind, so Builder-chain accepts. One resolved build input has no publisher attestation at all, which is the shape a compromised-dependency build presents: the top-level attestation is honest about a poisoned input.", + "name": "builder-accepts-dependency-unattested", + "description": "Subject and builder both bind, so `builder` accepts. One resolved build input has no publisher attestation at all, which is the shape a compromised-dependency build presents: the top-level attestation is honest about a poisoned input.", "profile": "trace.build_provenance.depth.v0", "context": { "artifact_digest": "sha256:361044e7e99a17fef9f0d4d454286b1c888a28819ad0fb756d05deb0b1a272e8", @@ -124,15 +124,21 @@ "expected": { "surface": { "outcome": "accept", - "failures": [] + "verified_depth": "surface", + "failures": [], + "unresolved": [] }, - "builder_chain": { + "builder": { "outcome": "accept", - "failures": [] + "verified_depth": "builder", + "failures": [], + "unresolved": [] }, - "dependency_chain": { - "outcome": "reject", - "failures": [ + "transitive": { + "outcome": "accept", + "verified_depth": "builder", + "failures": [], + "unresolved": [ "dependency_attestation_missing" ] } diff --git a/examples/build-provenance-depth/05-builder-chain-accepts-dependency-publisher-untrusted.json b/examples/build-provenance-depth/05-builder-accepts-dependency-publisher-untrusted.json similarity index 94% rename from examples/build-provenance-depth/05-builder-chain-accepts-dependency-publisher-untrusted.json rename to examples/build-provenance-depth/05-builder-accepts-dependency-publisher-untrusted.json index 30b694c7..8be55f20 100644 --- a/examples/build-provenance-depth/05-builder-chain-accepts-dependency-publisher-untrusted.json +++ b/examples/build-provenance-depth/05-builder-accepts-dependency-publisher-untrusted.json @@ -1,5 +1,5 @@ { - "name": "builder-chain-accepts-dependency-publisher-untrusted", + "name": "builder-accepts-dependency-publisher-untrusted", "description": "Every resolved input carries an attestation, so a verifier checking only for presence accepts. One was signed under an OIDC issuer the verifier does not trust, so the input's publisher is unverified rather than verified-and-wrong.", "profile": "trace.build_provenance.depth.v0", "context": { @@ -31,21 +31,21 @@ }, "resolvedDependencies": [ { - "uri": "pkg:npm/%40example-org/agent-core@1.8.2", + "uri": "pkg:npm/%40example-org/telemetry@0.4.7", "digest": { - "sha256": "fb6bfb2c7ffab3de12cbac5639014ae049aafb4dbb36390f251c82f63cf2d2cb" + "sha256": "3b6f7d63eeee3e1a792c0fa54e5dbb581ba3a651cfc20672fb6ca2af3fea0fb3" } }, { - "uri": "pkg:pypi/requests@2.32.3", + "uri": "pkg:npm/%40example-org/agent-core@1.8.2", "digest": { - "sha256": "a30964f520316ef09eb682027001f6f142547f350fadaeef8be4bb1b48f6d49c" + "sha256": "fb6bfb2c7ffab3de12cbac5639014ae049aafb4dbb36390f251c82f63cf2d2cb" } }, { - "uri": "pkg:npm/%40example-org/telemetry@0.4.7", + "uri": "pkg:pypi/requests@2.32.3", "digest": { - "sha256": "3b6f7d63eeee3e1a792c0fa54e5dbb581ba3a651cfc20672fb6ca2af3fea0fb3" + "sha256": "a30964f520316ef09eb682027001f6f142547f350fadaeef8be4bb1b48f6d49c" } } ] @@ -149,17 +149,23 @@ "expected": { "surface": { "outcome": "accept", - "failures": [] + "verified_depth": "surface", + "failures": [], + "unresolved": [] }, - "builder_chain": { + "builder": { "outcome": "accept", - "failures": [] + "verified_depth": "builder", + "failures": [], + "unresolved": [] }, - "dependency_chain": { + "transitive": { "outcome": "reject", + "verified_depth": "transitive", "failures": [ "dependency_publisher_untrusted" - ] + ], + "unresolved": [] } } } diff --git a/examples/build-provenance-depth/06-builder-chain-accepts-resolved-dependencies-absent.json b/examples/build-provenance-depth/06-builder-accepts-resolved-dependencies-absent.json similarity index 78% rename from examples/build-provenance-depth/06-builder-chain-accepts-resolved-dependencies-absent.json rename to examples/build-provenance-depth/06-builder-accepts-resolved-dependencies-absent.json index 9bd48151..8e83253f 100644 --- a/examples/build-provenance-depth/06-builder-chain-accepts-resolved-dependencies-absent.json +++ b/examples/build-provenance-depth/06-builder-accepts-resolved-dependencies-absent.json @@ -1,6 +1,6 @@ { - "name": "builder-chain-accepts-resolved-dependencies-absent", - "description": "The attestation binds subject and builder but declares no build inputs. A verifier that walks resolvedDependencies accepts vacuously: the loop has nothing to check. Dependency-chain assurance cannot be met by an attestation that enumerates nothing.", + "name": "builder-accepts-resolved-dependencies-absent", + "description": "The attestation binds subject and builder but declares no build inputs. A verifier that walks resolvedDependencies accepts vacuously: the loop has nothing to check. `transitive` assurance cannot be met by an attestation that enumerates nothing, so the evidence never resolves and the verifier records `builder`.", "profile": "trace.build_provenance.depth.v0", "context": { "artifact_digest": "sha256:361044e7e99a17fef9f0d4d454286b1c888a28819ad0fb756d05deb0b1a272e8", @@ -53,15 +53,21 @@ "expected": { "surface": { "outcome": "accept", - "failures": [] + "verified_depth": "surface", + "failures": [], + "unresolved": [] }, - "builder_chain": { + "builder": { "outcome": "accept", - "failures": [] + "verified_depth": "builder", + "failures": [], + "unresolved": [] }, - "dependency_chain": { - "outcome": "reject", - "failures": [ + "transitive": { + "outcome": "accept", + "verified_depth": "builder", + "failures": [], + "unresolved": [ "resolved_dependencies_absent" ] } diff --git a/examples/build-provenance-depth/README.md b/examples/build-provenance-depth/README.md index 9784a983..f0046ec5 100644 --- a/examples/build-provenance-depth/README.md +++ b/examples/build-provenance-depth/README.md @@ -4,20 +4,36 @@ Spec section 3.3 step 7 requires that "SLSA provenance resolves to a trusted bui It does not say how far the verifier walks, and three stopping points satisfy that sentence today: -1. **Surface** — `build_provenance.digest` matches the artifact, `builder` is in a +1. **`surface`** — `build_provenance.digest` matches the artifact, `builder` is in a trusted set. -2. **Builder-chain** — also resolve `provenance_uri` and check the attestation binds to +2. **`builder`** — also resolve `provenance_uri` and check the attestation binds to that digest and names that builder. -3. **Dependency-chain** — also walk the build inputs in `resolvedDependencies` to a +3. **`transitive`** — also walk the build inputs in `resolvedDependencies` to a publisher attestation for each one. -**Status.** These vectors are informative and encode no accepted requirement. The depth -question is [#50](https://github.com/agentrust-io/trace-spec/issues/50), labelled -`policy: take-internal`; nothing here anticipates its outcome. The names `surface`, -`builder_chain` and `dependency_chain` describe where verifiers stop today, not a -proposed enum — if the internal outcome names the levels differently, the names in this -directory change and the separations they encode do not. No text here carries an -uppercase RFC 2119 keyword. +**Status.** These vectors are informative and encode no accepted requirement; no text +here carries an uppercase RFC 2119 keyword. +[#50](https://github.com/agentrust-io/trace-spec/issues/50) has since resolved, so the +names above are now the `build_provenance.provenance_depth` wire values rather than the +descriptive ones this directory first used. That was a rename; the separations it +encodes did not change. The normative rules are in +[docs/verification.md](../../docs/verification.md), and this directory is what a +conformance runner executes against them. + +**Two outcomes, kept apart.** A depth can be unreachable for two different reasons and +they are the same event only to a verifier that never fetched the evidence: + +- **Evidence does not resolve** — no `provenance_uri`, a URI that 404s, a build input + with no publisher attestation. The check never ran. The verifier records the depth it + *did* reach in `appraisal.provenance_depth_verified` and names the missing evidence in + `unresolved`. The record is not at fault and is not rejected for it. +- **Evidence resolves and contradicts** — an attestation naming a different subject or a + different builder, a build input signed under an untrusted issuer. The check ran and + the record is wrong. It rejects, and no shallower reading makes that go away. + +Whether the appraisal is then *accepted* is a separate question from depth: a deployment +profile with a floor of `transitive` rejects a `verified_depth` of `builder` on the floor, +not on the evidence. Depth records what happened; the floor decides what is enough. ## What these add @@ -25,31 +41,41 @@ A vector that all three depths reject separates nothing: any verifier strict eno reject it passes, whatever depth it stopped at. Existing negative vectors are that kind by construction, and they are the kind an implementer writes unprompted. -Each vector here is **accepted by the depth below it and rejected by the depth in its -filename**. Run the set and a verifier's stopping point is visible in its verdicts -alone, which is the property that makes two verifiers' disagreement diagnosable rather -than just observable. +Each vector here **passes the depth below it with nothing to report, and the depth in its +filename is the first one that says something** — a rejection, or a `verified_depth` +lower than the one attempted. Run the set and a verifier's stopping point is visible in +its own output, which is the property that makes two verifiers' disagreement diagnosable +rather than just observable. -| Vector | Surface | Builder-chain | Dependency-chain | Separates on | +| Vector | `surface` | `builder` | `transitive` | Separates on | |---|---|---|---|---| | `01-all-depths-accept` | accept | accept | accept | — (control) | | `02-…-attestation-subject-mismatch` | accept | reject | reject | attestation names a different subject | | `03-…-attestation-builder-mismatch` | accept | reject | reject | attestation names a different builder | -| `04-…-dependency-unattested` | accept | accept | reject | a build input has no publisher attestation | +| `04-…-dependency-unattested` | accept | accept | accept, verified `builder` | a build input has no publisher attestation | | `05-…-dependency-publisher-untrusted` | accept | accept | reject | an input's attestation was signed under an untrusted issuer | -| `06-…-resolved-dependencies-absent` | accept | accept | reject | the attestation declares no build inputs at all | +| `06-…-resolved-dependencies-absent` | accept | accept | accept, verified `builder` | the attestation declares no build inputs at all | + +`04` and `06` are the vectors that separate a boundary without rejecting anything, and +they are why separation is defined over everything the verifier reports rather than over +rejections. Their evidence never resolves, so a verifier that walks the dependencies +learns only that it cannot finish — and says so, by recording `builder` and naming the +missing evidence. A verifier that stopped at `builder` records `builder` too, but with an +empty `unresolved`: it never looked. That difference is the separation. `05` is the +neighbouring case where evidence *does* resolve and refutes the record, so it rejects. `01` is load-bearing in the way a positive control is: without it the set is satisfied by a verifier that rejects unconditionally, and every separation below it means nothing. Two vectors per boundary, on two different defects, is deliberate. One vector per boundary has no margin — a single implementation shortcut that happens to catch that one -defect passes the whole boundary. `02` and `03` both sit at Surface → Builder-chain: a +defect passes the whole boundary. `02` and `03` both sit at `surface` → `builder`: a verifier that checks the subject but never compares the builder inside the attestation -passes `02` and fails `03`. `04`, `05` and `06` sit at Builder-chain → -Dependency-chain: presence of an attestation, trust in who signed it, and whether there -was anything to walk are three different implementations, and `06` is the one that -catches a verifier whose dependency loop accepts vacuously over an empty list. +passes `02` and fails `03`. `04`, `05` and `06` sit at `builder` → `transitive`: +presence of an attestation, trust in who signed it, and whether there was anything to +walk are three different implementations, and `06` is the one that catches a verifier +whose dependency loop accepts vacuously over an empty list — that verifier records +`transitive` where the honest answer is `builder`. `02` is also the vector for the shortcut of comparing truncated digests. Its mismatching subject digest shares the leading 12 hex characters with the real one — the prefix @@ -72,8 +98,16 @@ artifact digest the six vectors share is a fixed constant and derives from nothi runs offline. - `context.dependency_attestations` — publisher attestations keyed by package URL, plus the `verified_identity` a Sigstore bundle verification would have established. -- `expected` — the verdict at each depth. Depths stack, so a deeper depth carries every - failure the shallower depths found. +- `expected` — the verdict at each depth *attempted*, keyed by the `provenance_depth` + wire values. Depths stack, so a deeper depth carries every finding the shallower depths + made. Each block holds four fields: + - `outcome` — `accept` or `reject`. Set by `failures` alone. + - `verified_depth` — what belongs in `appraisal.provenance_depth_verified`: the depth + the verifier reached. Never deeper than the one attempted, and one level below any + depth whose evidence did not resolve. + - `failures` — evidence that resolved and contradicts the record. + - `unresolved` — evidence that never resolved, so the check could not run. This is what + caps `verified_depth`; it is not a finding against the record. ## What there is deliberately no vector for @@ -83,10 +117,13 @@ carries the result rather than the bundle. What varies here is *binding* — sub builder identity, per-input publisher — which is where a shallower verifier accepts silently rather than loudly. -**The absent and unresolvable `provenance_uri`.** Both reject at Builder-chain and both -are cases an implementer writes without prompting. `tests/test_build_provenance_depth_vectors.py` -exercises them by mutating the accepting control, so they are covered without diluting -the fixture set with the obvious. +**The absent and unresolvable `provenance_uri`.** Both downgrade to `surface` rather than +rejecting, and both are cases an implementer writes without prompting. +`tests/test_build_provenance_depth_vectors.py` exercises them by mutating the accepting +control, so they are covered without diluting the fixture set with the obvious. The +mutation pair is worth reading even so: the same control, mutated two ways, rejects when +the evidence contradicts and downgrades when the evidence is merely absent. That contrast +is the distinction in one screen. ## Running them @@ -96,9 +133,16 @@ pytest tests/test_build_provenance_depth_vectors.py The test module carries a reference verifier parameterised by depth. It asserts each vector's verdict at each depth, and separately asserts the properties that keep the set -honest as it changes: verdicts are monotone over depth, every boundary keeps two -vectors separating on distinct defects, every rule is load-bearing (delete it and some -fixture stops matching), and no rule is exercised by declaration alone. +honest as it changes: verdicts are monotone over depth, no verifier records a depth +deeper than it attempted, a downgrade always names the evidence that caused it, every +boundary keeps two vectors separating on distinct defects, every rule is load-bearing +(delete it and some fixture stops matching), and no rule is exercised by declaration +alone. + +The second and third of those are the machine-checkable half of a rule `docs/verification.md` +has to state in prose: a record is byte-identical whether the verifier walked the chain or +merely said it did, so no JSON Schema can hold *"MUST NOT record a depth higher than +executed"*. A conformance runner can hold it, against its own output. An implementation in another language runs the vectors directly: read `build_provenance` and `context`, verify at each depth, compare against `expected`. diff --git a/tests/test_build_provenance_depth_vectors.py b/tests/test_build_provenance_depth_vectors.py index 3d13ec36..8e36f617 100644 --- a/tests/test_build_provenance_depth_vectors.py +++ b/tests/test_build_provenance_depth_vectors.py @@ -1,18 +1,30 @@ """Depth-separating vectors for `build_provenance` verification (#50). §3.3 step 7 requires that "SLSA provenance resolves to a trusted builder" without saying -how far a verifier walks. Three stopping points are all conformant today: +how far a verifier walks. Three stopping points are all conformant today, and they are +the `build_provenance.provenance_depth` values: -1. **Surface** — `build_provenance.digest` matches the artifact, `builder` is trusted. -2. **Builder-chain** — also resolve `provenance_uri` and check the attestation binds to +1. **`surface`** — `build_provenance.digest` matches the artifact, `builder` is trusted. +2. **`builder`** — also resolve `provenance_uri` and check the attestation binds to that digest and names that builder. -3. **Dependency-chain** — also walk `resolvedDependencies` to a publisher attestation +3. **`transitive`** — also walk `resolvedDependencies` to a publisher attestation per build input. +Two outcomes are kept apart, because they are the same event only if you never fetched +the evidence. Evidence that **does not resolve** — no `provenance_uri`, a URI that 404s, +an input with no publisher attestation — leaves the check unrun, so the verifier records +the depth it did reach and says which evidence was missing. Evidence that **resolves and +contradicts** the record — an attestation naming another subject or another builder, an +input signed under an untrusted issuer — is a rejection, and no shallower reading of the +same record makes it go away. `verify` returns both: `unresolved` caps `verified_depth`, +`failures` sets the outcome. + A vector that all three depths reject separates nothing: it is satisfied by any verifier strict enough to reject it, whatever depth it stopped at. The set here is the other kind -— each vector is **accepted by the depth below and rejected by the depth named in its -filename**, so a verifier's stopping point is observable from its verdicts alone. +— each vector passes the depth below with nothing to report, and the depth named in its +filename is the first one that says something: a rejection, or a recorded depth lower +than the one it attempted. Either way a verifier's stopping point is observable from its +own output, which is the property the set exists for. The one vector that rejects nowhere is `01-all-depths-accept`. Without it, the set is satisfiable by a verifier that rejects unconditionally, and the separations mean nothing. @@ -43,21 +55,33 @@ FIXTURE_DIR = Path(__file__).parent.parent / "examples" / "build-provenance-depth" -# Ordered shallowest first. Depths stack: a verifier at depth N runs every check at -# depths 0..N, so a failure found shallow is still a failure deep. -DEPTHS: tuple[str, ...] = ("surface", "builder_chain", "dependency_chain") +# Ordered shallowest first, on the `build_provenance.provenance_depth` wire values. +# Depths stack: a verifier at depth N runs every check at depths 0..N, so a failure +# found shallow is still a failure deep. +DEPTHS: tuple[str, ...] = ("surface", "builder", "transitive") DEPTH_INDEX = {depth: index for index, depth in enumerate(DEPTHS)} +# What a fired rule does to the appraisal. The two are disjoint, and what separates +# them is whether the evidence resolved — not how serious the finding is. +FAILS = "fails" # evidence resolved and contradicts the record +DOWNGRADES = "downgrades" # evidence did not resolve, so the check could not run + @dataclass(frozen=True) class Rule: """One check, owned by the shallowest depth that can run it. ``check`` returns True when the defect is present — i.e. when the code is emitted. + + ``effect`` separates the two things a defect can mean. A rule that ``FAILS`` has + evidence in hand that contradicts the record, and no shallower reading makes that + go away. A rule that ``DOWNGRADES`` never got its evidence: the verifier caps + `provenance_depth_verified` at the depth below and stands on what it did check. """ code: str depth: str + effect: str check: Callable[[dict[str, Any]], bool] = field(compare=False) @@ -98,7 +122,7 @@ def _builder_untrusted(vector: dict[str, Any]) -> bool: return builder is None or builder not in vector["context"]["trusted_builders"] -# -- builder chain ----------------------------------------------------------------- +# -- builder ---------------------------------------------------------------------- def _provenance_uri_missing(vector: dict[str, Any]) -> bool: @@ -128,7 +152,7 @@ def _attestation_builder_mismatch(vector: dict[str, Any]) -> bool: return bool(attested_builder != vector["build_provenance"].get("builder")) -# -- dependency chain -------------------------------------------------------------- +# -- transitive ------------------------------------------------------------------- def _resolved_dependencies_absent(vector: dict[str, Any]) -> bool: @@ -155,15 +179,17 @@ def _dependency_publisher_untrusted(vector: dict[str, Any]) -> bool: RULES: tuple[Rule, ...] = ( - Rule("artifact_digest_mismatch", "surface", _artifact_digest_mismatch), - Rule("builder_untrusted", "surface", _builder_untrusted), - Rule("provenance_uri_missing", "builder_chain", _provenance_uri_missing), - Rule("attestation_unresolvable", "builder_chain", _attestation_unresolvable), - Rule("attestation_subject_mismatch", "builder_chain", _attestation_subject_mismatch), - Rule("attestation_builder_mismatch", "builder_chain", _attestation_builder_mismatch), - Rule("resolved_dependencies_absent", "dependency_chain", _resolved_dependencies_absent), - Rule("dependency_attestation_missing", "dependency_chain", _dependency_attestation_missing), - Rule("dependency_publisher_untrusted", "dependency_chain", _dependency_publisher_untrusted), + Rule("artifact_digest_mismatch", "surface", FAILS, _artifact_digest_mismatch), + Rule("builder_untrusted", "surface", FAILS, _builder_untrusted), + Rule("provenance_uri_missing", "builder", DOWNGRADES, _provenance_uri_missing), + Rule("attestation_unresolvable", "builder", DOWNGRADES, _attestation_unresolvable), + Rule("attestation_subject_mismatch", "builder", FAILS, _attestation_subject_mismatch), + Rule("attestation_builder_mismatch", "builder", FAILS, _attestation_builder_mismatch), + Rule("resolved_dependencies_absent", "transitive", DOWNGRADES, _resolved_dependencies_absent), + Rule( + "dependency_attestation_missing", "transitive", DOWNGRADES, _dependency_attestation_missing + ), + Rule("dependency_publisher_untrusted", "transitive", FAILS, _dependency_publisher_untrusted), ) RULE_CODES = frozenset(rule.code for rule in RULES) @@ -183,12 +209,25 @@ def _dependency_publisher_untrusted(vector: dict[str, Any]) -> bool: def verify(vector: dict[str, Any], depth: str, rules: Sequence[Rule] = RULES) -> dict[str, Any]: - """Verify at `depth`, running every rule owned by that depth or a shallower one.""" + """Verify at `depth`, running every rule owned by that depth or a shallower one. + + `depth` is what the verifier attempts. `verified_depth` is what it reached, which is + the value belonging in `appraisal.provenance_depth_verified`: never deeper than the + attempt, and capped one level below any depth whose evidence did not resolve. + """ limit = DEPTH_INDEX[depth] - failures = [ - rule.code for rule in rules if DEPTH_INDEX[rule.depth] <= limit and rule.check(vector) - ] - return {"outcome": "reject" if failures else "accept", "failures": failures} + fired = [rule for rule in rules if DEPTH_INDEX[rule.depth] <= limit and rule.check(vector)] + failures = [rule.code for rule in fired if rule.effect == FAILS] + unresolved = [rule.code for rule in fired if rule.effect == DOWNGRADES] + reached = min( + [limit] + [DEPTH_INDEX[rule.depth] - 1 for rule in fired if rule.effect == DOWNGRADES] + ) + return { + "outcome": "reject" if failures else "accept", + "verified_depth": DEPTHS[reached], + "failures": failures, + "unresolved": unresolved, + } def _load(path: Path) -> dict[str, Any]: @@ -206,9 +245,9 @@ def test_fixture_set_is_complete() -> None: "01-all-depths-accept.json", "02-surface-accepts-attestation-subject-mismatch.json", "03-surface-accepts-attestation-builder-mismatch.json", - "04-builder-chain-accepts-dependency-unattested.json", - "05-builder-chain-accepts-dependency-publisher-untrusted.json", - "06-builder-chain-accepts-resolved-dependencies-absent.json", + "04-builder-accepts-dependency-unattested.json", + "05-builder-accepts-dependency-publisher-untrusted.json", + "06-builder-accepts-resolved-dependencies-absent.json", ] @@ -238,6 +277,49 @@ def test_verdicts_are_monotone_over_depth(name: str, vector: dict[str, Any]) -> rejected = rejected or result["outcome"] == "reject" +@pytest.mark.parametrize("name,vector", FIXTURES, ids=[name for name, _ in FIXTURES]) +def test_verified_depth_never_exceeds_the_depth_attempted( + name: str, vector: dict[str, Any] +) -> None: + """The rule `docs/verification.md` states and no JSON Schema can: a verifier MUST NOT + record `provenance_depth_verified` deeper than it executed. + + The record is byte-identical whether a verifier walked the chain or claimed it did, so + the schema cannot hold this. A conformance runner can, on its own output. + """ + for depth in DEPTHS: + result = verify(vector, depth) + assert DEPTH_INDEX[result["verified_depth"]] <= DEPTH_INDEX[depth], ( + f"{name} attempted {depth} and recorded {result['verified_depth']}" + ) + + +@pytest.mark.parametrize("name,vector", FIXTURES, ids=[name for name, _ in FIXTURES]) +def test_a_downgrade_is_never_silent(name: str, vector: dict[str, Any]) -> None: + """Recording a shallower depth than attempted is a claim about missing evidence, so + the evidence has to be named. A verifier that downgrades without saying which check it + could not run is indistinguishable from one that never attempted the depth at all — + and that is the whole failure this vector set exists to make visible.""" + for depth in DEPTHS: + result = verify(vector, depth) + downgraded = DEPTH_INDEX[result["verified_depth"]] < DEPTH_INDEX[depth] + assert downgraded == bool(result["unresolved"]), ( + f"{name} at {depth}: verified {result['verified_depth']}, unresolved " + f"{result['unresolved']}" + ) + + +def test_the_floor_depth_has_no_downgrading_rule() -> None: + """There is nothing below `surface` to downgrade to, so every rule owned by it must be + a failure. A downgrading rule there would have to record a depth that does not exist.""" + floor = DEPTHS[0] + assert not [rule for rule in RULES if rule.depth == floor and rule.effect == DOWNGRADES] + + +def test_every_rule_declares_one_of_the_two_effects() -> None: + assert {rule.effect for rule in RULES} <= {FAILS, DOWNGRADES} + + BOUNDARIES = list(zip(DEPTHS[:-1], DEPTHS[1:], strict=True)) @@ -252,31 +334,59 @@ def test_each_boundary_is_separated_by_two_independent_vectors( One vector per boundary has no margin: a single implementation shortcut that happens to catch that one defect passes the whole boundary. Requiring two distinct codes means a verifier has to have implemented the depth, not one check from it. + + Separation is over everything the verifier reports, not over rejection alone. A + vector whose deeper evidence never resolves is still separating: the shallower + verifier reports nothing, the deeper one records a lower depth and names what it + could not fetch. Counting rejections only would have hidden that boundary the moment + a defect moved from `FAILS` to `DOWNGRADES`. """ + + def reported(vector: dict[str, Any], depth: str) -> frozenset[str]: + result = verify(vector, depth) + return frozenset(result["failures"]) | frozenset(result["unresolved"]) + introduced = [ - frozenset(verify(vector, deeper)["failures"]) - - frozenset(verify(vector, shallower)["failures"]) + reported(vector, deeper) - reported(vector, shallower) for _, vector in FIXTURES - if verify(vector, shallower)["outcome"] == "accept" - and verify(vector, deeper)["outcome"] == "reject" + if not reported(vector, shallower) and reported(vector, deeper) ] assert len(introduced) >= 2, f"{shallower} -> {deeper} has {len(introduced)} separating vectors" assert len(set(introduced)) >= 2, f"{shallower} -> {deeper} separates on one defect only" -def test_expected_failure_codes_are_all_registered() -> None: +def _expected_codes(vector: dict[str, Any], depth: str) -> set[str]: + block = vector["expected"][depth] + return set(block["failures"]) | set(block["unresolved"]) + + +def test_expected_codes_are_all_registered() -> None: for name, vector in FIXTURES: for depth in DEPTHS: - unknown = set(vector["expected"][depth]["failures"]) - RULE_CODES + unknown = _expected_codes(vector, depth) - RULE_CODES assert not unknown, f"{name} expects codes no rule can emit: {sorted(unknown)}" +def test_expected_codes_are_filed_under_the_effect_their_rule_declares() -> None: + """A code cannot be expected as a failure in one fixture and an unresolved in another. + The two lists are the wire form of the distinction; letting a code cross between them + is how the conflict this set had with `docs/verification.md` got in.""" + effect_of = {rule.code: rule.effect for rule in RULES} + for name, vector in FIXTURES: + for depth in DEPTHS: + block = vector["expected"][depth] + for code in block["failures"]: + assert effect_of[code] == FAILS, f"{name}: {code} is not a failure" + for code in block["unresolved"]: + assert effect_of[code] == DOWNGRADES, f"{name}: {code} does not downgrade" + + def test_every_rule_is_exercised() -> None: from_fixtures = { code for _, vector in FIXTURES for depth in DEPTHS - for code in vector["expected"][depth]["failures"] + for code in _expected_codes(vector, depth) } assert from_fixtures | EXERCISED_BY_MUTATION == RULE_CODES @@ -288,7 +398,7 @@ def test_no_rule_is_exercised_only_by_declaration() -> None: code for _, vector in FIXTURES for depth in DEPTHS - for code in vector["expected"][depth]["failures"] + for code in _expected_codes(vector, depth) } assert not (from_fixtures & EXERCISED_BY_MUTATION) @@ -329,16 +439,13 @@ def _mutate(**changes: Any) -> dict[str, Any]: "surface", ), ({"builder": "https://ci.example.org/pipelines/agent"}, "builder_untrusted", "surface"), - ({"provenance_uri": None}, "provenance_uri_missing", "builder_chain"), - ( - {"provenance_uri": "https://provenance.example.org/support-agent/2.4.2.intoto.jsonl"}, - "attestation_unresolvable", - "builder_chain", - ), ], ) -def test_control_mutation_is_rejected(mutation: dict[str, Any], code: str, depth: str) -> None: - """The rules with no fixture, checked against the vector they would reject. +def test_contradicting_control_mutation_is_rejected( + mutation: dict[str, Any], code: str, depth: str +) -> None: + """Evidence in hand that contradicts the record. The verifier fetched what it needed + and what it got says the record is wrong, so there is no honest shallower reading. Each mutation also has to leave the unmutated control accepting at the same depth, so a rule that fires on everything cannot pass this as a detection. @@ -349,6 +456,40 @@ def test_control_mutation_is_rejected(mutation: dict[str, Any], code: str, depth assert code in result["failures"] +@pytest.mark.parametrize( + "mutation,code", + [ + ({"provenance_uri": None}, "provenance_uri_missing"), + ( + {"provenance_uri": "https://provenance.example.org/support-agent/2.4.2.intoto.jsonl"}, + "attestation_unresolvable", + ), + ], +) +def test_unresolvable_control_mutation_downgrades_rather_than_rejects( + mutation: dict[str, Any], code: str +) -> None: + """The other half of the same control, and the pair is the point. + + Nothing about the record changed except whether its evidence can be fetched. The + contradicting mutations above reject; these two record `surface` — the depth actually + reached — and name the evidence that never arrived. A verifier that rejects here is + failing a record for the state of someone else's transparency log, which is what + `provenance_depth_verified` exists to avoid. + """ + assert verify(CONTROL, "builder") == { + "outcome": "accept", + "verified_depth": "builder", + "failures": [], + "unresolved": [], + } + result = verify(_mutate(**mutation), "builder") + assert result["outcome"] == "accept" + assert result["verified_depth"] == "surface" + assert result["unresolved"] == [code] + assert result["failures"] == [] + + def test_surface_mutation_rejects_at_every_depth() -> None: """Monotonicity, from the other side: a surface defect is not outgrown by depth.""" mutated = _mutate(digest="sha256:" + "0" * 64)