Repository navigation
Preserve agent accounting evidence in fallback artifacts #61052
Description
Problem
The daily AIC conclusion can fail with missing agent accounting when the large, best-effort agent artifact upload is unavailable or incomplete. The existing agent-output-fallback artifact preserves safe-output files, but it does not include the compact accounting inputs consumed downstream.
This is distinct from the missing evals accounting fixed by #60869 and #60892. The remaining agent path can lose:
- execution evidence
- agent usage summaries
- firewall token-usage records
Without those files, the conclusion job cannot account for a failed agent job reliably. This was observed while following up on #60870.
Proposed fix
Include the compact agent accounting evidence in agent-output-fallback alongside the existing safe-output files, covering current and legacy firewall usage paths. Add compiler coverage for the generated artifact paths.
Acceptance criteria
- The fallback artifact includes agent execution evidence and engine usage summaries.
- The fallback artifact includes supported firewall token-usage paths.
- Generated workflow coverage verifies these files remain in the fallback artifact.
Activity
dsyme commentedon Sep 15, 2026
Additional evidence from the gh-aw-test E2E run 34810867167, using gh-aw
aa0cabfb259ewith--use-samples:Missing accounting for executed agent component ... conclusion success: agent/token_usage.jsonl is empty; agent_usage.jsonl is missing; agent_usage.json is missing.usageartifact exists and containsagent/token_usage.jsonlas a zero-byte file. Itsagent-output-fallbackartifact contains onlyagent_output.jsonandsafeoutputs.jsonl. Because samples bypass inference, zero AIC is expected for this run.v0.88.7.Important scope distinction for #61053:
check_daily_aic_workflow_guardrail.cjscurrently only selects artifacts namedusage(PRIMARY_GUARDRAIL_ARTIFACT_NAMES = ["usage"]) and selects one artifact rather than merging accounting records fromagent-output-fallback. Therefore adding accounting files toagent-output-fallbackimproves conclusion/downstream recovery when the large agent artifact is missing, but by itself does not fix the observed sampled-run daily-guardrail failures when ausageartifact exists with an empty token file.A regression test should model a successful sampled agent with an existing
usageartifact and empty token usage, and prove it accounts as zero rather than raisingtransient_error. The implementation could either preserve explicit sample/no-inference execution evidence in the primaryusageartifact or teach the guardrail to consume/merge the compact fallback evidence.dsyme commentedon Sep 15, 2026
Addressed in #61053: the fallback now carries
agent_execution.json, conclusion downloads merge it with the agent artifact, and the collector stages it asusage/agent/execution.json. Added regressions proving a successful sampled agent with empty token usage and same-attemptnot_startedevidence accounts as zero rather than producingtransient_error.