test(conformance): portable PIC/TRACE bridge corpus and independent verifier #206

Activity

1. github-actions commented on Aug 24, 2026

   

   github-actionsBot

   on Aug 24, 2026 – with GitHub Actions

   Last edited by github-actions

   Contributor

   More actions

   🟡 Contributor Check: MEDIUM

   Check	Result
   Profile	MEDIUM
   Credential	LOW
   Overall	MEDIUM

   Automated check by AgenTrust Contributor Check.
2. github-actions

   added

   needs-review:MEDIUMContributor check flagged MEDIUM riskContributor check flagged MEDIUM risk

   on Aug 24, 2026
3. imran-siddique commented on Aug 26, 2026

   

   imran-siddique

   on Aug 26, 2026

   Member

   More actions

   Accepted. Please send it.

   Checked your premises at pinned main rather than taking them: #179 closed on 2026-08-23, and all four artifacts it delivered are there, docs/integration/pic-trace-bridge-v1.md, schema/pic-trace-bridge-v1.json, src/agentrust_trace/intent_bridge.py, and tests/test_intent_bridge.py carrying exactly eight test functions. Your "eight implementation tests, inline fixtures, not portable" is accurate down to the count, and keeping this separate from PR #126's shared JCS corpus rather than duplicating it is the right boundary.

   Your third question was live, and it was broken

   You asked:

   Does each negative case fail for the intended rule rather than an unrelated earlier error?

   Those eight tests were failing that test as of this morning. @lywinged's #204 merged a few hours ago and found that the bridge cases used bare pytest.raises(IntentBridgeError), and since AuthorizationDenied and AuthorizationMismatch both subclass it, a case passed on any rejection rather than the one it named. A ("tools", [""]) case intended to exercise the item-emptiness guard was passing as AuthorizationMismatch: executed tool is outside the authorized tool scope. Separately, the scalar cases ran at now=0 against a fixture valid from 100, so four of five were tripping "authorization is not yet valid" before reaching the guard they were named for.

   Three of the five runtime guards could be deleted with the whole suite still green.

   So you proposed, from outside and without seeing it, the exact property that turned out not to hold inside. That moves it from a nice-to-have in your corpus to the requirement it is built around.

   Scope I am accepting

   Your case contract as drafted, with one thing promoted from implicit to mandatory:

   Every negative vector pins a stable reason code, and the runner asserts the code rather than the fact of rejection. A vector that only records "this must fail" is worth very little, because a second implementation can fail it for a reason neither of us intended and still look conformant. This is the property #204 had to retrofit; build it in.

   The rest as you wrote it: versioned JSON, bridge artifact and trusted-authorizer JWK, declaration and PIC verifier outputs, tool call and optional transcript, fixed verification time, expected semantic result, and bridge-specific canonical bytes with reproducibility metadata. The fixed verification time matters more than it looks, for the reason #204 demonstrated: a vector set whose time base is the wall clock tests the clock.

   Non-normative and informative, as you scoped it. No changes to docs/integration/pic-trace-bridge-v1.md's normative content in the same PR.

   Two suggestions

   Coordinate with @lywinged if you can. #204 just rewrote the inline tests your corpus is meant to supersede, and the match= expectations it added are effectively a first draft of your reason codes. Starting from those rather than deriving them independently will save you a reconciliation pass.

   And prove the corpus is load-bearing the way #204 proved its own: take each rule the bridge enforces, remove it, and show which vectors go red. A corpus where every vector still passes with a guard removed is the same failure in portable form.

   needs-review:MEDIUM on this issue is our contributor-check bot, not a judgement about the proposal.
4. lywinged commented on Aug 26, 2026

   

   lywinged

   on Aug 26, 2026

   Collaborator

   More actions

   @coreytshaffer, on your third question, which is the one I could answer with a measurement.

   "Does each negative case fail for the intended rule rather than an unrelated earlier
   error?" I ran that against the bridge rather than reasoning about it. Each of the five
   runtime guards in src/agentrust_trace/intent_bridge.py was removed one at a time, and
   tests/test_intent_bridge.py run against two trees: the one before #204, and main today.

   guard removed	before #204	main today
   the three _nonempty_string calls	19 passed	3 failed
   or not value, the empty-array half	19 passed	2 failed
   and item, the empty-string half	19 passed	2 failed
   authorization[field] < 0	1 failed	2 failed
   the duplicate check	2 failed	2 failed

   Three of the five could be deleted with the file still green. The file held 19 cases then and
   holds 21 now, the two added being the empty-array ones. The two causes are already on this
   thread: bare pytest.raises(IntentBridgeError) with two subclasses under it, and a
   now=0 time base against a fixture authorized at 100. #204 is mine and fixed both, changing
   only the test file. The right-hand column is what that bought.

   So your fixed verification time is not hygiene. A wall-clock time base was one of the two
   things that made a green suite meaningless here.

   And one thing about why that file carries the weight it does.
   agentrust_trace.intent_bridge is exported in __all__, has no caller anywhere in this
   repository outside its own test, and the conformance suite has no bridge module at all: I
   grepped trace-tests for one and found nothing. So tests/test_intent_bridge.py is the only
   thing checking the bridge today. That is what made three deletable guards more than untidiness,
   and it is the gap your corpus closes rather than duplicates.

   The draft reason codes, since they are worth having in one place. What #204's match=
   expectations amount to:

   Rule	Current expectation
   scalar emptiness	authorization_id must be a non-empty string, same for authorizer and authorizer_key_id
   scalar sign	authorized_at must be a non-negative integer, same for expires_at
   scope duplicates	must not contain duplicates
   scope emptiness, both halves	must be a non-empty array of non-empty strings
   window	expired, after, now
   transcript	before/after
   unknown fields	unknown fields

   They are message fragments rather than codes, and verify_bridge has no code table to bind
   them to. Turning them into codes is the part your corpus adds.

   Where your contract already has precedent in this repository

   Your field	What exists
   artifact and trusted key	trusted_key beside a signed record in the boundary vectors, context.trusted_root_keys in delegation
   fixed verification time	context.now, in all 23 delegation vectors
   expected result and reason code	expected: {classification, codes}, ten codes across those 23, one per registry rule
   declaration, PIC outputs, tool call, transcript	bridge-specific, no precedent

   Two of your acceptance criteria come free. tests/test_generators_reproduce_fixtures.py
   discovers generators by globbing examples/**/gen_*.py, empties the directory, runs the
   generator, and compares name sets before bytes. And your third question is enforced there
   rather than intended: the delegation walk emits every code from one function, and
   test_delegation_completeness asserts by AST that no other function appends to a failure or
   warning list. It also wants two load-bearing vectors per rule and a declared defect that
   separates them.

   Three more of your criteria have partial answers here. Test-only material: NOT_GENERATED
   records that the action-receipts/conformance fixtures 01-09 pin a key whose private half was
   never published, and that everything else there pins its own deterministic test key. Schema
   before semantics: the delegation walk runs iter_errors from the library first and applies
   its own rules after. Language neutrality: examples/action-receipts/acta/gen.mjs is a
   JavaScript generator sitting beside three Python ones, and its fixtures are re-verified in CI
   by tests/test_acta_fixtures.py. The regeneration harness globs gen_*.py though, so the
   byte-identical guarantee is the one part that does not reach across languages today.

   On canonical bytes, since you are carrying them: no corpus here publishes bytes. The
   boundary vectors prove canonicalization through the signature, because a record signed over
   its JCS bytes will not verify under a serializer that produces different ones, and
   diverges_under names the ones that break it. That is mostly a note about what your
   reproducibility metadata has to do: keep published bytes from drifting away from the code
   that produced them.

   One correction on the 8785 dependency. What you would reuse is already merged:
   examples/canonicalization-boundary/ with tests/test_canonicalization_boundary.py, added in
   #160 and extended in #170. #126 itself is still open, and its head no longer shares history
   with main, so build against main rather than against that PR. It is mine, so I would
   rather say it than have you find it.

   On placement: every corpus published here for other implementations sits in
   examples/<topic>/ with its generator beside it and the enforcing harness in tests/.
   trace-tests carries five fixtures under tests/vectors/, but those are its own inputs for
   scoring a record. So the precedent is examples/pic-trace-bridge-v1/. I tried that
   literally before saying it: a new directory there with one gen_*.py and one fixture beside
   it is collected as its own case with nothing registered anywhere, and changing a single byte
   in the fixture fails that case and only that case.
5. coreytshaffer commented on Aug 29, 2026

   

   coreytshaffer

   on Aug 29, 2026 – with ChatGPT Codex Connector

   ContributorAuthor

   More actions

   I have completed the issue #206 Gate 1 rejection inventory against current main
   (e7e2ecab68cf3534c7d5fcb7e9a6f089fcb7d592; #204 is in its ancestry). Before I
   generate bulk fixtures, I would like confirmation on the public v1 taxonomy.

   Proposed runtime classifications remain accepted, invalid, denied, and
   mismatch. The settled codes are:

   invalid:
     bridge_not_object, bridge_unknown_fields, bridge_profile_invalid,
     signature_representation_invalid, authorization_not_object,
     authorization_unknown_fields, authorization_fields_missing,
     authorization_id_invalid, authorizer_invalid, authorizer_key_id_invalid,
     authorization_unverifiable, authorizer_key_id_mismatch,
     authorized_at_invalid, expires_at_invalid, verification_time_invalid,
     authorization_not_yet_valid, authorization_window_invalid,
     authorization_expired, scope_not_object, scope_unknown_fields,
     scope_tools_invalid, scope_tools_duplicate, scope_impacts_invalid,
     scope_impacts_duplicate, tool_call_not_object, declaration_not_object,
     pic_not_object, pic_unknown_fields, pic_profile_invalid,
     pic_intent_digest_invalid, pic_args_digest_invalid,
     execution_input_not_jcs, declaration_digest_invalid,
     tool_call_digest_invalid, transcript_requirement_invalid
   
   denied:
     decision_not_allow
   
   mismatch:
     tool_outside_scope, impact_outside_scope,
     pic_intent_digest_mismatch, pic_args_digest_mismatch,
     declaration_digest_mismatch, tool_call_digest_mismatch,
     transcript_incomplete, transcript_before_mismatch,
     transcript_after_invalid
   

   The deliberate groupings are: empty scope versus empty item; signed versus supplied
   PIC digest syntax; all failures inside the current broad post-decode catch; non-allow
   decisions; and declaration/tool-call JCS failures.

   signature_representation_invalid covers missing, non-string, or decoder-rejected
   signature representations; its anchored raw-ValueError decoder signal is normalized
   despite the documented exception-class inconsistency. authorization_unverifiable
   covers the later broad catch, including cases where malformed trusted-key material or
   authorization canonicalization prevents Ed25519 verification from beginning.

   I found four points I propose to leave explicitly unsettled rather than normalize:

   1. schema-invalid padded, standard-alphabet, or non-alphabet signature encodings can
      be accepted by the permissive runtime decoder;
   2. safe-integer timestamp violations collapse into the broad signature failure;
   3. JSON Schema treats 100.0 as an integer while the Python runtime does not; and
   4. transcript after currently requires only an object, with no required members.

   I will keep these as documented divergences and will not change
   src/agentrust_trace/intent_bridge.py in this corpus contribution. Does this table
   and treatment of the four divergences match the intended Gate 1 contract?
6. imran-siddique commented on Aug 30, 2026

   

   imran-siddique

   on Aug 30, 2026

   Member

   More actions

   Taxonomy confirmed. Four runtime classifications and the code set as listed. Generate the fixtures.

   The groupings you called deliberate are the right cuts, and two of them are the ones I would have asked for if you had not made them: empty scope versus empty item, and signed versus supplied PIC digest syntax. Both separate a producer error from a verifier error, which is the distinction a corpus exists to pin.

   authorization_unverifiable as the name for the later broad catch is right, and worth keeping even after the catch narrows: "verification could not begin" is a different statement from "verification failed", and a corpus that conflates them teaches implementers to conflate them too.

   On the four divergences: two documented, two reclassified

   Divergence 2 (safe-integer timestamp violations collapse into the broad signature failure) and divergence 4 (transcript after requires only an object): document them as you propose. Both are diagnostic-quality and coverage gaps in the current implementation, neither makes two conformant implementations disagree, and fixing them is out of scope for a corpus contribution.

   Divergences 1 and 3 need different treatment, and it is not "fix them here". Do not change intent_bridge.py in this contribution; that boundary was right. But do not record them as expected behaviour either.

   • 1, the permissive signature decoder. Schema-invalid padded, standard-alphabet or non-alphabet encodings being accepted by the runtime means the runtime is looser than the schema. Two implementations reading the two surfaces disagree on what is a valid record. For contrast, ca2a's transport codec rejects exactly those: standard-base64 + and /, = padding, and invalid lengths, failing closed and naming the field.
   • 3, 100.0 as an integer. JSON Schema treats a number with zero fractional part as an integer; the Python runtime does not. This is the third instance of that class today. cmcp#588 has {"n": 1} and {"n": 1.0} sharing one JCS canonical form while Cedar evaluates them differently and parameter_headers() raises on one. schema: the subject pattern accepts identities the reference model refuses #244 has the schema's subject pattern accepting identities model_validate refuses.

   Both belong to #247, which I opened this morning for exactly this: the normative spec, the JSON Schema, the reference model and the docs state the rules and nothing says which wins.

   Why the label matters more than it sounds. A conformance corpus is a statement about what correct behaviour is. A fixture asserting that the permissive decoder accepts a padded signature does not document a divergence, it defines the permissive behaviour as correct, and the next implementer builds to the fixture rather than to the schema. That is the same failure I flagged on trace-tests#86 today, where a vector named invalid_canonical_integer_out_of_range would have encoded a library defect as a conformance verdict.

   So: carry 1 and 3 in the corpus as known defects with a tracker reference, not as expected results. Whatever shape your metadata takes, a consumer should be able to tell "the implementation does this and that is correct" from "the implementation does this and it is a recorded defect". If your fixture format has no slot for that distinction, that is worth telling me before you generate in bulk, because retrofitting it across a corpus is expensive.

   One thing I am not asking for

   Do not block on 1 and 3 being fixed. The corpus is more useful landed with two rows marked as defects than held until the decoder is tightened. #247 is where the fix gets sequenced.

   Thank you for pinning the SHA and confirming #204 is in its ancestry before inventorying. That is why I could answer this from the taxonomy rather than re-deriving the state first.
7. imran-siddique commented on Sep 3, 2026

   

   imran-siddique

   on Sep 3, 2026

   Member

   More actions

   Linking #279. The SKIP-rather-than-PASS question raised here is one of six instances in this repository of the same primitive, and the corpus surface is the one where getting it wrong is most expensive.

   The rule #279 carries that bears directly on a conformance corpus: no fixture may encode current behaviour as expected where a surface silently passes today. A fixture asserting the current behaviour defines it as correct, and the next implementer builds to the fixture rather than the spec.

   #279
8. coreytshaffer commented on Sep 4, 2026

   

   coreytshaffer

   on Sep 4, 2026 – with ChatGPT Codex Connector

   ContributorAuthor

   More actions

   Thanks for linking #279. I ran a read-only reconciliation audit of the held #206 corpus against 3c2f96375afaea6c615c49dea21d83a7969862e3 before changing any expectations.

   One positive fixture needs clarification: TRACE-PIC-BRIDGE-003 (accepted-empty-after-object). It has transcript_required: true and after: {}, and the corpus expects accepted with transcript_bound: true. That is the corpus's normalized result, not a field returned directly by the reference API. The generator explicitly declares this expectation; it does not learn it by running the verifier.

   With the other inputs held fixed, both verifier paths produce these normalized outcomes:

   Transcript input (transcript_required: true)	Outcome
   after: {}	accepted, transcript_bound: true
   after: {"unrelated": true}	accepted, transcript_bound: true
   after: null	mismatch / transcript_after_invalid
   Missing after, or absent transcript	mismatch / transcript_incomplete

   Your earlier guidance treated the object-shape-only behavior as a documented coverage gap. I am asking how to reconcile that with #279, without presuming it is a defect.

   What positive claim should the corpus's transcript_bound: true represent when after is required? Is structural completeness sufficient, or must the after-record establish a defined execution-result fact? If the latter, which fact, and what bridge-specific not-established outcome/reason should the corpus pin when the evidence is insufficient? A nonempty-object guard alone would not settle this, given the unrelated-object result above.

   Case 003 is held locally in an audit sidecar pending that decision. The fixture, both verifiers, taxonomy, runner, existing expectations, and frozen denominator remain unchanged. Zero-silent-PASS certification is withheld; I will not claim new conformance credit for 003 while its semantics remain unresolved.
9. imran-siddique commented on Sep 7, 2026

   

   imran-siddique

   on Sep 7, 2026

   Member

   More actions

   The corpus is held on the explicit meaning of transcript_bound for an empty or unrelated after object. Keep case 003’s disputed positive claim out of new conformance credit until that maintainer decision is resolved; #279 supplies the absence-reporting constraint.
10. imran-siddique

    assigned

    coreytshaffer

    on Sep 7, 2026
11. madeinplutofabio commented on Sep 15, 2026

    

    madeinplutofabio

    on Sep 15, 2026

    More actions

    As PIC maintainer, I think the #206 hold now has a clean boundary after #338/#340.

    From the PIC side, allow means the pre-execution PIC contract passed for the declared action. It does not mean the tool executed, the action succeeded, or a successor state was established.

    So I would keep the bridge corpus on three separate axes:

    1. pic_authorization_bound: the bridge consumed PIC-defined verifier outputs, including PIC-CJSON/1.0, intent_digest, and args_digest, and those matched the signed authorization.
    2. successor_envelope_bound or after_digest_bound: the supplied transcript.after object matched the digest carried in the signed authorization.
    3. successor_outcome: established, contradicted, or not-established, evaluated separately under observer trust, freshness, independence policy, and the transition predicate.

    With that split, an empty or unrelated after object should not receive a positive successor/outcome claim. At most, a valid object can be digest-bound. Whether it establishes a transition is a separate sufficiency question.

    This also avoids a PIC interoperability problem: the corpus should treat PIC-CJSON/1.0, intent_digest, and args_digest as upstream PIC-defined verifier outputs, not redefine their semantics locally inside TRACE. In current PIC, args_digest is SHA-256 over PIC-CJSON canonical bytes of proposal.action.args, while intent_digest is SHA-256 over the raw UTF-8 bytes of proposal.intent.

    I can help with the PIC-facing part without taking over the accepted #206 work: either review the fixture contract for the PIC fields, or send a small PR adding a PIC-source reference and a short note that the bridge consumes PIC verifier outputs rather than defining them.
12. coreytshaffer commented on Sep 16, 2026

    

    coreytshaffer

    on Sep 16, 2026 – with ChatGPT Codex Connector

    ContributorAuthor

    More actions

    Case 003 reconciliation is now prepared and independently reviewed against the September 15 PIC maintainer decision.

    Scope is intentionally narrow: only case 003 and its direct counterexamples are reconciled. The old transcript_bound: true positive credit remains withdrawn, the frozen conformance denominator remains 100, rows 101/102 remain diagnostics, and full-corpus certification remains held.

    The review patch separates the three axes discussed above:

    • pic_authorization_bound
    • successor_envelope_bound
    • successor_outcome (established / contradicted / not-established)

    The bounded vectors now cover:

    • complete relevant successor envelope -> accepted / established
    • literal after: {} -> exact transcript_after_invalid refusal, no outcome evaluation
    • bound empty observation -> accepted / not-established
    • bound unrelated observation -> accepted / not-established

    Verification after a packaging-only LF normalization:

    • 95/95 scoped checks passed
    • fixture regeneration from an empty destination passed
    • all 101 frozen fixture hashes matched
    • all 105 fixture files remained byte-identical
    • forward/reverse patch roundtrips reproduced the intended trees exactly
    • all 456 checkout files matched byte-for-byte after roundtrip

    Reviewed artifact hashes:

    • patch: c7de8b3621aea7d85155a736005f6ed5fd7558d89176aed603ea8dfe439892bf
    • editable review ZIP: 49c5bd677da1e620315374eed41f4e349aa52c96fad1d676b48bba254e1124cb
    • verification record: 4d21ff8fd13f051a42b9a4963dd5af5eb270c8ced6d1248571c1a70ac743dd29

    The patch was prepared against TRACE main at cd92e16dde6658eedbe1058c903f3fce92bb3a58 and does not claim wider corpus reconciliation or restored certification.

    I’m ready to put this into a reviewable PR next. My current GitHub integration has read-only repository access, so I can’t publish the branch from this session; if a maintainer has a preferred fork/branch path, I’ll use that exact route rather than broadening the change.
13. imran-siddique commented on Sep 18, 2026

    

    imran-siddique

    on Sep 18, 2026

    Member

    More actions

    My hold of 7 September on case 003 is lifted: the PIC maintainer ruled on 15 September, and #363 carries the three-axis split that ruling set out. What remains is the review on #363, not the hold.