Skip to content

test: characterize cA2A advertisement with pinned official A2A TCK - #266

Open
dinakarjs wants to merge 11 commits into
agentrust-io:mainfrom
dinakarjs:test/pinned-a2a-tck-differential
Open

dinakarjs wants to merge 11 commits into
agentrust-io:mainfrom
dinakarjs:test/pinned-a2a-tck-differential

Conversation

@dinakarjs

@dinakarjs dinakarjs commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem and resulting behavior

The TCK comparator reported protected-handler counts without enforcing the ordinary-path isolation claim: 0/7 and 7/7 could compare equal. It now requires an integer-zero protected-handler count on both sides, rejects modified tracked cA2A source, and retains per-test identity/outcome and provenance comparison.

The runner refuses tracked/untracked TCK changes and clears inherited PYTEST_* child settings. The workflow also runs for source, dependency and project metadata changes. Documentation clarifies that both modes enable streaming, that their differential is the extension advertisement, and that the counter observes successful protected lookup after authentication/payload checks rather than every protected-path entry. Existing environment/action pins remain fixed experimental inputs; refreshing them requires separate measurement.

Validation — 2 November 2026

Head: 306f91f; tree: 9cd2c7da11f30f87a4e7e16d4191d75b682e0404.

  • 15 comparator controls pass, including unequal/equal nonzero counts, malformed count types, dirty source and valid 0/0 twins. Restoring the old comparator causes nine assertion failures; the final comparator passes all 15.
  • Python compilation, targeted Ruff E4/E7/E9/F and whitespace checks pass.
  • No fresh local official-TCK run or full runtime suite is claimed for this helper/workflow update. Current-head hosted paired-TCK execution and independent re-review remain required.

Historical initial TCK measurements at their original pinned source: both JSON-RPC MUST runs reported 79 passed, 156 skipped, 30 deselected and equal per-test outcomes; separate bridge/signed-loopback tests reported 55 passed. These historical results are not re-dated by the comparator repair.

Ordinary-path advertisement characterization only; no protected-path TCK certification, full A2A certification, cross-language coverage, hardware acceptance or independent operator run. Prepared with OpenAI Codex; DCO-signed.

Dependency follow-through — 2 November 2026

The first hosted run of this follow-up stopped at pip-audit because the branch still pinned pyasn1 0.6.4. The current head carries the exact reviewed #303 input/lock and SDK/dev-extra floors (pyasn1 >=0.6.5), with both changelog entries preserved. This is a dependency-only follow-up; this PR's runtime/test implementation is unchanged. Hash-required installation, pip check and local pip-audit (editable checkout explicitly skipped by the dependency scanner) pass; the 130 focused revocation/peer/offline/delegation tests pass in that updated environment. Hosted checks are running on the current head. No audit bypass or upstream merge occurred.

Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Contributor Check: MEDIUM

Check Result
Profile MEDIUM
Credential LOW
Overall MEDIUM

Automated check by AgenTrust Contributor Check.

@github-actions github-actions Bot added the needs-review:MEDIUM Contributor check flagged MEDIUM risk label Oct 21, 2026
Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
Signed-off-by: Srinivasa Consultants <dinakarjs@gmail.com>
Preserve existing contribution history and bounded evidence claims; validate the merged tree.

Signed-off-by: Srinivasa Dinakar Jainti <dinakarjs@gmail.com>
Signed-off-by: Srinivasa Consultants <78673153+dinakarjs@users.noreply.github.com>
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@dinakarjs
dinakarjs marked this pull request as ready for review October 31, 2026 03:43
@dinakarjs
dinakarjs requested review from a team, carloshvp and zohebk8s as code owners October 31, 2026 03:43

Copy link
Copy Markdown
Contributor Author

@carloshvp Please review the pinned official A2A TCK advertisement comparison and its comparator controls at current head 877ade76e03878ccf83bb16ef059948a9cb1d40a. This PR is now ready for independent review, and the existing reviewer requests are populated. All returned current-head technical workflows succeed (validation evidence); the PR description records the prior local tests and their revisions.

Ordinary-path characterization only; no protected-handler TCK coverage, full A2A certification or hardware acceptance.

Please identify any remaining change required or disposition this bounded contribution. This request does not claim independent approval, a new full-suite rerun, or merge readiness. Prepared with OpenAI Codex.

@rajnisht7 rajnisht7 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the overall changes looks good, but the evidence gate has a gap that matters for what this PR claims

  • compare.py never asserts that protected_handler_invocations is 0 on either side. It only reports the numbers. I reproduced both [0, 7] and [7, 7] still returning identical_test_outcomes: true when the rest of the provenance and JUnit outcomes match. The README claims 0/0 and the runner records the count specifically for this isolation property, so the comparator should enforce it.

Please add the zero-check in compare() and a negative-control test (or three) covering non-zero / unequal counts. The existing negative controls cover everything else except this.

A couple of related notes while we’re here:

  • The counter only increments inside _lookup() after successful auth + payload checks, so early rejects on the protected path stay at 0. The field name is a bit stronger than the actual metric.
  • Streaming is forced True on both sides (original card is False). The differential is still controlled, but the “only removes the advertisement” wording is slightly imprecise.
  • The new workflow only triggers on the experiment/demo paths; changes under src/ won’t re-run it.
  • TCK cleanliness check uses --untracked-files=no.
  • Child process inherits PYTEST_ADDOPTS etc.
  • Small pin differences (cedarpy 4.12.1 vs current 4.12.2, setup-python still on v7.0.0).
  • ca2a_tracked_changes is recorded but never compared.
    even though none of these break runtime. The missing zero assertion is the one that needs fixing before this can land, since the whole point of the PR is ordinary-path characterization evidence.

Reject nonzero or malformed protected-handler counts and dirty tracked source. Cover unequal and equal nonzero counts with valid twins. Clarify the counter and streaming fixture, reject untracked TCK changes, clear inherited pytest options, and schedule on source/dependency changes.

Prepared with OpenAI Codex.

Signed-off-by: Srinivasa Dinakar Jainti <dinakarjs@gmail.com>

dinakarjs commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author

@rajnisht7 The zero-count finding is addressed at current head 306f91f. Both sides must report genuine integer zero; 0/7, 7/0, 7/7 and malformed counts fail with valid 0/0 controls. Fifteen comparator tests pass; the old comparator fails nine. Tracked-source comparison, untracked TCK cleanliness, inherited PYTEST_* settings, relevant source/dependency triggers and precise streaming/counter wording are addressed. The current head also carries #303's reviewed pyasn1 repair after the earlier hosted audit failure; local audit is clean. The current-head paired TCK workflow succeeds. Historical dependency/action pins remain fixed experimental inputs; no fresh local TCK measurement is claimed. Please re-review this head. Formal reviewer assignment returned 403, so this conversation is the re-review request. Prepared with OpenAI Codex.

Carry the exact reviewed agentrust-io#303 dependency input, lock and SDK/dev-extra floors into this branch to clear the current hosted security audit failure. Preserve this PR's own runtime/tests and changelog entry. No security gate is bypassed.

Prepared with OpenAI Codex.

Signed-off-by: Srinivasa Dinakar Jainti <dinakarjs@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-review:MEDIUM Contributor check flagged MEDIUM risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants