Repository navigation
Reproducibility claim: deterministic re-execution as positive evidence for software-only producers #249
Description
Spec section affected
Claims model (spec/trace-v0.2.md) — a new optional claim member plus its interaction with appraisal. Normative, non-breaking.
Problem
Per the ruling in #231: software-only is currently defined by what it lacks, and there is no room in the claims model for deterministic re-execution as evidence. This issue proposes the primitive's shape, answering the four questions asked there from a working producer. It deliberately does not define any producer's journal layout — that is the annex, after this.
Proposed change
1. What is the claim, exactly?
A reproducibility claim states: re-executing a named deterministic function of the run — the producer's coordination logic, not the workload's side effects — over a pinned input closure yields a transcript whose canonical digest equals the claimed value. Three parts, all explicit in the record:
- the code identity re-executed: a resolvable artifact digest (for a producer that ships publicly, the same digest
build_provenancealready carries); - the input closure: the complete content-addressed set of everything that function reads — initial configuration and every recorded external interaction, model calls included;
- the claimed transcript digest, over RFC 8785 canonical bytes — the canonicalization test: independent-producer evidence for the parent-record canonicalization boundary #245 just ruled for record hashing, so a verifier carries exactly one.
A claim whose closure omits anything that can change the transcript is malformed. That is the definition of the claim, not a quality bar on it.
2. What must a verifier hold to re-run?
Three things, none supplied by the producer's environment: the implementation at the pinned digest (a public artifact), the input-closure blobs (each {id, digest, resolver}-addressable — the shape references already uses), and the claim itself. The rule follows from your framing: if re-execution requires any state only the producer's environment can supply, the claim must not be called reproduction — it is an assertion about that environment and belongs elsewhere. This is also why model interactions are inputs in the closure rather than something a verifier re-invokes: a model call is not deterministic, so the boundary of the deterministic function is drawn around it, and the recorded interaction is pinned by digest like any other input.
3. What is the honest failure mode?
Three statuses, never conflated:
- reproduced — re-run completed, digests equal;
- diverged — re-run completed, digests differ; the result records the verifier's observed digest, because divergence localizes nothing by itself (producer tampering, verifier drift, and an under-pinned closure all look identical until a third party can compare transcripts);
- not-attempted — any closure blob unresolvable, code identity unavailable, or the re-run not run to completion; carries the reason.
not-attempted must not be reportable as reproduced, and equally must not be reported as diverged: absent is not pass, and absent is not failure. Same discipline as TR-APR.
4. Where does it sit relative to appraisal?
The claim is producer-side. The re-execution result is an appraisal — made by a party who verified by re-running, with that party as the verifier. Proposal: an appraisal method discriminator (e.g. method: "re-execution") with the tri-state outcome scoped under it, leaving the existing appraisal.status enum untouched (aware that #190 holds that enum closed). If the editors prefer the outcome folded into status instead, the requirement is the tri-state; the placement is yours.
Explicitly out of scope: runtime.measurement stays as it is (its software-only scope is #240's question), and nothing here names a journal format.
Backward compatibility
- Non-breaking (new optional field, informative addition)
- Breaking (removes or changes required field, changes wire format, changes algorithm set)
Motivation
The answers above are lifted from a producer rather than designed on paper: our orchestrator keeps the coordination loop deterministic, records every model interaction as a content-addressed input, and signs the run journal; re-executing over the recorded closure reproduces the transcript digest. The part that does not re-execute — the model calls — is exactly what forced the input-closure discipline in (1) and (2). The vendor mapping (journal layout, how a closure is stored and resolved) is held back for the annex once this primitive has a shape.
Related issues or PRs
#231 (parent ruling and sequencing), #245 (canonicalization precedent), #240 (runtime.measurement scope — untouched here), #190 (appraisal enum — left closed).
Activity
lywinged commentedon Aug 31, 2026
@chernistry The four answers land, and they answer what was asked rather than what would have been convenient. The boundary drawn around model calls, recorded interactions pinned by digest inside the closure rather than something a verifier re-invokes, is the piece that makes the claim decidable, and the verifier-holdings rule, nothing supplied by the producer's environment or it is not reproduction, is the same boundary stated from the other side. The tri-state does the right work: not-attempted reportable as neither pass nor failure is the discipline you point at in TR-APR, and the same line the gap-disclosure outcomes proposed in #260 draw for a different claim, so three places now hold the same line, which is worth more than any single member. And the method discriminator keeps #190's enum closed, which was the wall to respect.
One question worth settling while the shape is still soft. diverged bundles three causes you name as indistinguishable: producer tampering, verifier drift, an under-pinned closure. The result already records the verifier's observed digest. Would it also carry the verifier's implementation digest and a digest of the closure manifest as resolved? That gives the third-party comparison you point to a starting point, at the cost of two fields, and without them every diverged ends in a conversation rather than a comparison.
On process, stating it rather than assuming it. The rule as written on #245 is that normative text from an individual sits under the organizational-sponsor rule, and the #245 exemption was for fixing a defect in existing coverage, not adding an obligation. This adds a member rather than fixing a hole, so by my reading it sits on that gate, optional or not. Whether it proceeds as spec text, waits on a sponsor, or is carried is not a ruling I will make alone. Flagging it for @imran-siddique, together with the §4.4 annex-scope question from #231, which is still open and which the annex behind this issue is waiting on.
chernistry commentedon Sep 1, 2026
Both, and they are not doing the same job — which is the reason to carry both rather than pick one.
Digest of the closure manifest as resolved is the one that actually discriminates, and it discriminates from a single result. If what the verifier resolved differs from what the producer declared, the closure did not pin what it claimed: under-pinned, decidable from the record alone, no second verifier required. If it matches and the output still diverges, under-pinning is excluded — which is the useful half, because it is the cause that is cheapest to fix and most likely.
The verifier's implementation digest discriminates nothing on its own: it is the verifier naming its own build, self-asserted, and a tampering producer would be equally happy to see it recorded. It earns its place only as a correlation key across results — two verifiers at different digests disagreeing on a closure whose resolved manifest matched is verifier drift, and that reading is unavailable without the field. Worth stating in the spec that it carries no weight singly, so nobody reads it as attestation.
So the enum stays closed (#190's wall) and the discrimination lives in the comparison, not in new outcomes:
The third row is the only one that accuses, and it accuses on absence. I would have the spec say exactly that — "no reproducing verifier found a benign explanation" — rather than naming tampering, because the record cannot distinguish a tampering producer from a cause nobody has thought of yet.
imran-siddique commentedon Sep 3, 2026
Ruling all of it, and it should go to a PR. This answers the four questions from #231 better than the questions were asked, and @lywinged's review has already done the work of checking it against the neighbouring claims.
Accepted as proposed
The claim's definition (1), including that a closure omitting anything which can change the transcript is malformed rather than low-quality. That is the right severity: a closure that does not pin what it claims is not a weak claim, it is not a claim.
The verifier-holdings rule (2). "If re-execution requires state only the producer's environment can supply, it is not reproduction" is the load-bearing sentence and it should appear in the normative text close to verbatim. The model-call boundary follows from it correctly: a non-deterministic call is drawn outside the deterministic function and its recorded interaction is pinned by digest as an input. That is the part that makes the claim decidable rather than aspirational, and it is the part a reader will be tempted to relax.
The tri-state (3), and
not-attemptedmust be as prominent in the text as the other two, not a trailing sentence. Recording the verifier's observed digest ondivergedis right for the reason you give: divergence localizes nothing on its own.The placement, which you left to me
Take the method discriminator. Do not touch
appraisal.status.appraisal.statusis EAR's closed enum,affirming | warning | contraindicated | none, and #190 holds it closed for a reason that has not changed. Addmethod("re-execution") with the tri-state outcome scoped under it.One thing to write into the text so it is not "simplified" later. EAR's existing
nonelooks like it already coversnot-attempted, and reusing it would be the cheap answer. Do not. #260 faced the same choice for receipt gaps and introduced a namedreceipt_gap_disclosedoutcome instead, because a generic no-claim value gets read as "fine" by the next reader and the reason for the absence is lost. Say in the section whynoneis not the same thing, or someone will collapse them in six months and think they are tidying up.Scope, confirmed
runtime.measurementstays out, it is #240's question. No journal format. The annex follows on #231 once this lands, as ruled there.This is an instance of a pattern, and I have filed it
Yours is the sixth thread in this repository needing a first-class not-established value, alongside #190, #191, #192, #206 and #66, and there are three more in sibling repos. Filed #279 as the tracker, with #260 as the precedent and the naming rule extracted from it. Your tri-state is the cleanest statement of the discipline anyone has written, so #279 quotes the shape rather than inventing one.
Nothing about #279 gates this. It exists so the seventh thread does not re-argue it.
Please open the PR
Normative text needs an organisational sponsor here and you do not have one, so I will carry it: authorship stays yours, and I answer for the
MUSTs on merge. That is the arrangement, not a formality, so write it as you would write it.What lands well: the claim definition and the three parts in §3.1, the verifier-holdings rule stated as a requirement rather than as guidance, the tri-state with
not-attemptedgiven equal weight and areason, and themethoddiscriminator with an explicit note on why it is notappraisal.status.none. Vectors from the Bernstein producer alongside it would be worth more than prose, since you already have a producer that makes the claim and a verifier that can check it.Ping me on the PR and I will review it rather than re-litigate any of the above.
chernistry commentedon Sep 6, 2026
Taking the PR. Thank you for carrying the sponsorship — I will write it as my own text rather than as a draft for you to rewrite, since that is what the arrangement asks for.
What I will put up, in the shape you ruled:
MUST, not as guidance.not-attemptedcarrying equal weight and areason.methoddiscriminator, with an explicit note on why it is notappraisal.status.none— stating the reason for the absence, so the next reader cannot collapse the two while tidying.On
#279: agreed that it does not gate this, and I would rather it quote the shape than have me restate it.On vectors, one correction to what I would otherwise have sent you. I went to produce the second discriminating vector before writing this, and found
examples/canonicalization-boundary/03-utf16-key-order.jsonalready does exactly what I was about to duplicate — U+1F600 against U+FFFD, the two orderings genuinely disagreeing. So a hand-built second vector adds nothing.What is still missing is the thing you actually asked for on #231: a record emitted by a second independent producer, not generated by a script in this repository. The failure that matters is two producers disagreeing on an astral key, and a vector this repo generates for itself cannot demonstrate that no matter how correct it is. Producing one means driving a real signing path to emit a record with a supplementary-plane key, which is a change on my side rather than a file I can write here.
So: the normative PR first, on its own. The producer-emitted vectors follow separately, and I will not hold up §3.1 waiting for them.
Ping me on the PR — I would rather have the review than the agreement.
imran-siddique commentedon Sep 7, 2026
The reproducibility-claim PR is accepted for maintainer carry in the prior ruling. Preserve the verifier-holdings rule, explicit not-attempted reason, and method discriminator. Independent-producer vectors follow separately rather than gating the normative text.
chernistry commentedon Sep 7, 2026
Re-reading the ruling so we are aligned: you carry the PR and answer for the
MUSTs on merge; the writing and the authorship are mine — the same arrangement as on the 6th. The duplicate of that comment which I posted above has been removed.The PR will follow in the shape you ruled — §3.1 claim definition with the malformed-closure rule, verifier holdings as a
MUST, the tri-state with an explicitnot-attemptedreason, and themethoddiscriminator with the note on why it is notappraisal.status.none. Independent-producer vectors stay a separate deliverable and do not gate it. I will ping you on the PR when it is up.imran-siddique commentedon Sep 13, 2026
Checking in rather than adding anything: the ruling from the 6th stands and nothing on this is waiting on me. You write it and hold authorship, I carry it and answer for the
MUSTs on merge.Scope as you restated it, unchanged: the §3.1 claim definition with the malformed-closure rule, verifier holdings as a
MUST, the tri-state with an explicitnot-attemptedreason, and themethoddiscriminator with the note on why it is notappraisal.status.none. Independent-producer vectors stay a separate deliverable and do not gate it.Could you have the PR open by 19 September? If the scope has shifted under you, or one of the four parts is harder than it looked when we agreed it, say which here and we will cut it down rather than let the whole thing wait on the hardest part.
chernistry commentedon Sep 13, 2026
Availability note rather than an answer on scope.
We are into the holidays here and I am only partly online. I am back on Wednesday 16 September and will pick up everything that is on me from that day, in one go: the PR here in the shape you ruled on the 6th, the coverage guard on #247, and the producer-side supplementary-plane vector on #245.
On the 19 September date — I am not asking to cut anything yet. I will tell you on Wednesday whether all four parts hold that date, and if one of them does not I will name which rather than let the rest wait on it. I will tag you as soon as the PR is up.
chernistry commentedon Sep 16, 2026
PR is up: #364, in the shape ruled on the 6th. Section 3.1.4 with the claim definition and the malformed-closure rule, verifier holdings as a MUST, the tri-state with an explicit
not-attemptedreason, andappraisal.method: "re-execution"with the outcome scoped under it and the note on why it is notstatus: none. Schema, model anddocs/schema.mdfollow the text rather than lead it, for the reason in the PR body.On the 19 September date: all four parts hold it, nothing needs cutting. The other two items from the 13 September note are also up today: #365 for the #247 guard, and the producer-side supplementary-plane vector on #231. @imran-siddique
spec(claims): one disposition for a read beyond the closure; per-hold…
spec(claims): verifier_code_identity is carried on any outcome (agent…
spec(claims): not-attempted when closure-alone cannot be established (a…
spec(claims): reproducibility claim for deterministic re-execution (#249
imran-siddique commentedon Sep 24, 2026
Resolved by #364, merged September 18. Section 3.1.4 now defines the reproducibility claim, verifier holdings, malformed-closure handling and explicit re-execution outcomes.
Independent-producer vectors remain separate work under #231/#245, as agreed. Closing this specification proposal.