[devdiv07]

Refs #215

Breaking change

ShadowAIScanner.scan_records() and ShadowAIScanner.scan_audit_log() now raise IdentityUnavailableError when any tool-call record has no usable agent_id.

Returning a findings list in that situation would look complete while some tool calls were never classified. A caller could not tell "no violations" from "some records could not be checked". Raising makes the incomplete classification impossible to miss.

For records that do carry a usable agent_id, both methods return the same results as before.

Behavior on f6e8c8b for tool-call records without a usable agent_id:

agent_id	Before	After
missing	classified as the literal ID "unknown": usually unregistered_agent, but undeclared_tool or no finding at all if the registry lists "unknown"	IdentityUnavailableError
"" or whitespace-only	classified as that blank ID: usually unregistered_agent with an empty suggested_manifest_id, but undeclared_tool or no finding if the registry holds the same blank key	IdentityUnavailableError
null, number, boolean	AttributeError	IdentityUnavailableError
array, object	TypeError	IdentityUnavailableError

This is also documented in the Shadow AI README (now shadow-ai/README.md) under "Compatibility change".

Two interfaces

• Report methods (new): scan_records_report() and scan_audit_log_report() return a ScanReport, for callers that need to inspect partial classification:
  • findings: the DiscoveryEvents for records with a usable identity
  • unavailable: one IdentityUnavailable diagnostic per unclassifiable tool-call record, located by 0-based list index or 1-based physical JSONL line
  • complete: a property derived from unavailable
• Strict methods (existing): scan_records() and scan_audit_log() keep their return behavior for valid records and raise on incomplete classification. IdentityUnavailableError is a ValueError subclass. Its .report is the same ScanReport the report method would return, so the valid findings and the diagnostics are not lost when it is raised.

All four methods share the same classification logic.

A usable identity is a str that is not empty or whitespace-only. It is used verbatim: never trimmed, lowercased or converted from another type. IdentityUnavailable carries a reason code (missing, null, blank, non_string), plus the JSON type name for non_string. It has no agent_id or suggested_manifest_id. The diagnostic does not copy the rejected agent_id into an identity field, and the exception message does not print its value. Other record fields are preserved as documented. DiscoveryEvent is unchanged and is only produced when an identity is available.

complete means no successfully processed tool-call record was unclassifiable because of unavailable identity. It does not certify that the audit source itself is complete. Malformed JSON lines and non-object records still raise json.JSONDecodeError and AttributeError as before.

JSONL line splitting

JSONL lines are now split only on \n, \r\n and \r. The previous str.splitlines() also split on characters such as U+2028, which JSON allows unescaped inside strings. With UTF-8 decoding, that broke a valid record in two and raised json.JSONDecodeError; I reproduced this on f6e8c8b. A deterministic regression test feeds the decoded text to the scanner. It covers line splitting only, not how a file is decoded on a given platform. JSONL is still read with Python's default text encoding, which is unchanged.

Compatibility claims

• Removed tested_against:

  • cmcp-runtime: "0.2.0": the scanner does not accept cMCP v0.2.0 output directly.
  • agent-manifest: "1.0.0": not a released version (PyPI's latest is 0.12.0).

  No replacement versions were added, because no such test run has been performed.

• The README and manifest description now document the inputs the scanner actually supports:

  • a separate agent-to-tools registry
  • enriched records carrying a usable agent_id
  • an optional timestamp, copied as-is with no normalization or type validation

  They no longer describe cMCP output or the cMCP catalog.json as direct inputs.

• marketplace/catalog.json was regenerated with scripts/generate_marketplace_catalog.py.

Maintainer follow-up

Imran added commit eda98c4, moving Shadow AI into the standalone shadow-ai/ directory.

The integration manifest and Marketplace listing have been removed. Shadow AI no longer claims working cMCP or Agent Manifest integration. This supersedes the integration.yaml and marketplace/catalog.json items under "Compatibility claims" above.

The identity-diagnostic implementation remains unchanged: the source and test files are pure renames. The Nox paths and lint configuration have been updated for the new location.

Scope

This PR covers the identity-reporting and compatibility-claim scope agreed in #215. It does not include:

• Timestamp normalization: timestamp_utc mapping, which remains separate as agreed.
• The future cMCP adapter: including a trustworthy session-to-agent mapping and genuine cMCP fixtures. This also remains separate.
• changes to cMCP or Agent Manifest
• new diagnostics for malformed input

It references #215 rather than closing it; closing the issue is left to the maintainers.

Testing

Original local verification (60fccb0, before the move)

Run locally on Windows. The commands mirror validate.yml and lint.yml, and the Shadow AI tests use the nox shadow_ai session's PYTHONPATH; nox itself was not run.

• integrations/shadow-ai/tests: 60 passed on Python 3.12.12 and 3.13, with both the default Windows encoding and -X utf8.
• scripts/validate_integrations.py: 41 integrations, 0 failures.
• scripts/validate_compatibility.py: 0 failures.
• scripts/generate_integration_index.py --check and scripts/generate_marketplace_catalog.py --check: current.
• The validate.yml test files all pass.
• ruff check with CI's rule set and git diff --check: clean.

These results are for 60fccb0 and use the old integrations/shadow-ai/ paths. They were not re-run locally after eda98c4.

CI on the updated PR (eda98c4)

• Passing: Lint (ruff, which now covers shadow-ai/), Validate integrations, CodeQL, Check documentation links, and the other triggered workflows.
• Blocking: "Require maintainer approval" (gate) reports failure until a maintainer approves the PR.
• No CI workflow runs the Shadow AI test suite (nox -s shadow_ai), so the 60 tests above are local evidence only.

[imran-siddique]

Matches what #215 settled: strict list methods raise on unusable identity, the report interface carries findings, per-record diagnostics and complete, and missing identity produces no unregistered_agent finding. Since no CI job runs nox -s shadow_ai, I ran it: 60 passed.

Main has moved and marketplace/catalog.json conflicts. Merge main in and run python scripts/generate_marketplace_catalog.py rather than resolving the hunk by hand; that gives 41 integrations and validate_integrations 41/0. I will merge once that is pushed.

[carloshvp]

Approved at 761697e6f1a0e59181426b4af3d836e641ed2440 for the scoped #215 correction.

The standalone move removes Shadow AI from the integration index and Marketplace, and the README no longer claims direct cMCP or Agent Manifest compatibility. The scanner reports missing, null, blank, and non-string agent identities separately from findings; strict list methods raise instead of returning an apparently complete result. Valid findings survive in the report, and a literal unknown identity remains distinct from a missing identity.

In an isolated checkout at this head, 60 Shadow AI tests passed. Integration validation passed (41 integrations, 0 failures), as did compatibility validation, generated index/catalog checks, compileall, and git diff --check. Independent controls covered a cMCP-shaped record without agent_id, mixed valid and unavailable records, strict failure, CRLF line locations, and rejection of a cMCP tool catalog as an agent registry.

The current head's only reported check is the failing maintainer-approval gate; no CI job runs the Shadow AI suite. This approval covers the identity diagnostic and corrected compatibility claims. It does not establish a working cMCP adapter or close the broader follow-up in #215.