Surface AWF steering counters in compact audit usage data by Copilot · Pull Request #63664 · github/gh-aw

Copilot · 2026-09-26T19:20:50Z

Audit required full firewall artifacts to inspect AWF steering behavior. Compact usage artifacts now expose aggregate counters for each steering event type.

Usage payload
- Parse AWF API proxy event logs across supported filenames and schema variants.
- Add normalized steering totals and per-type counters to usage/activity/summary.json.
Audit reporting
- Backfill steering metrics into firewall_token_usage for usage-only audits.
- Preserve detailed firewall analysis when available.
Contract
- Extend the audit schema and artifact documentation.
- Add regression coverage for event aggregation and usage-summary backfill.

{
  "total_steering_events": 3,
  "steering_event_counts": {
    "token_steering": 2,
    "timeout_steering": 1
  }
}

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions · 2026-09-26T19:49:24Z

✅ Ponytail Reviewer completed successfully!

Lean already. Ship.

Generated by Ponytail Reviewer for #63664

github-actions · 2026-09-26T19:49:25Z

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff

🧪 Test quality analysis by Test Quality Sentinel

github-actions · 2026-09-26T19:49:26Z

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

github-actions · 2026-09-26T19:49:29Z

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

🏗️ ADR gate enforced by Design Decision Gate 🏗️

github-actions · 2026-09-26T19:49:29Z

🧠 Matt Pocock Skills Reviewer failed during the skills-based review.

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

Copilot

Copilot review overview

🟡 Changes recommended

Event-file selection and typed payload decoding can silently omit valid steering events.

Review effort: Balanced
Findings: 2

Open (2)

Skip empty event logs when selecting the event file · New
Support non-object payloads when decoding steering entries · New

What changed in this PR

Adds per-type AWF steering counters to compact usage artifacts and exposes them through audit token-usage reporting.

Changes:

Aggregates steering events across supported filenames and schemas.
Backfills usage-only audit results with steering totals and counters.
Extends schemas, documentation, and regression coverage.

File	Description
`.changeset/minor-audit-gateway-steering.md`	Updates the release note.
`actions/setup/js/generate_usage_activity_summary.cjs`	Adds steering aggregation to usage summaries.
`actions/setup/js/generate_usage_activity_summary.test.cjs`	Tests summary steering aggregation.
`actions/setup/js/steering_helpers.cjs`	Adds per-type event counting.
`actions/setup/js/steering_helpers.test.cjs`	Tests schema variants and counters.
`docs/src/content/docs/reference/artifacts.md`	Documents compact steering data.
`pkg/cli/audit_analysis_fanout.go`	Backfills usage-only audit results.
`pkg/cli/logs_usage_activity.go`	Parses and applies compact steering summaries.
`pkg/cli/logs_usage_activity_test.go`	Tests backfill and preservation behavior.
`pkg/cli/token_usage_analyze.go`	Applies steering summaries during analysis.
`pkg/cli/token_usage_find.go`	Adds alternate event-log discovery.
`pkg/cli/token_usage_steering.go`	Aggregates normalized steering event types.
`pkg/cli/token_usage_test.go`	Tests discovery and aggregation.
`pkg/cli/token_usage_types.go`	Extends token usage and event structures.
`schemas/audit.schema.json`	Adds steering counters to the audit contract.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Copilot · 2026-09-26T20:44:03Z

+		for _, relativePath := range relativePaths {
+			candidate := filepath.Join(root, relativePath)
+			if fileutil.FileExists(candidate) {
+				return candidate


Addressed in d66cd7e: event-log discovery skips empty candidates and continues through supported roots, including legacy layouts, until it finds steering data. Regression tests cover the empty preferred file and legacy fallback.

Copilot · 2026-09-26T20:44:03Z

+	Payload        *struct {
+		Event string `json:"event"`
+		Type  string `json:"type"`
+	} `json:"payload,omitempty"`


Addressed in 4235a13: payload decoding is flexible, and nested event/type fields are read only from object payloads. Regression cases cover non-object string and array payloads.

github-actions · 2026-09-26T19:53:58Z

ADR required

This PR introduces an architectural decision that is not currently recorded in docs/adr/ on the branch, so I added a draft ADR at docs/adr/63664-surface-awf-steering-counters-in-compact-usage-data.md.

Evidence reviewed

actions/setup/js/generate_usage_activity_summary.cjs now aggregates steering events into compact usage artifacts.
pkg/cli/logs_usage_activity.go and pkg/cli/token_usage_steering.go backfill steering counts into audit/token-usage paths, including per-event counters.
schemas/audit.schema.json and docs/src/content/docs/reference/artifacts.md extend the public contract and documentation.

Next action

Please review and refine the draft ADR, then keep it in the PR as the explicit record of this design decision.

🏗️ ADR gate enforced by Design Decision Gate 🏗️ · pi · gpt54 · 17.8 AIC · ⌖ 8.1 AIC · ⊞ 10.1K · ◷
_{Comment /review to run again}

github-actions · 2026-09-26T19:54:57Z

Comment Memory

reviewed_at: 2026-09-26T19:51:11.817Z
review_event: REQUEST_CHANGES
top_themes:
  - steering counter fallback stops at first readable proxy-event file
  - compact usage and CLI audit can both under-report steering metrics
files_reviewed:
  - .changeset/minor-audit-gateway-steering.md
  - actions/setup/js/generate_usage_activity_summary.cjs
  - actions/setup/js/generate_usage_activity_summary.test.cjs
  - actions/setup/js/steering_helpers.cjs
  - actions/setup/js/steering_helpers.test.cjs
  - docs/src/content/docs/reference/artifacts.md
  - pkg/cli/audit_analysis_fanout.go
  - pkg/cli/logs_usage_activity.go
  - pkg/cli/logs_usage_activity_test.go
  - pkg/cli/token_usage_analyze.go
  - pkg/cli/token_usage_find.go
  - pkg/cli/token_usage_steering.go
  - pkg/cli/token_usage_test.go
  - pkg/cli/token_usage_types.go
  - schemas/audit.schema.json
comment_count: 2

Note

This comment is managed by comment memory.

It stores persistent context for this thread in the code block at the top of this comment.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 49.5 AIC · ⌖ 9.14 AIC · ⊞ 20.2K · ◷
_{Comment /review to run again}

github-actions

REQUEST_CHANGES

The new steering-counter paths short-circuit on the first candidate proxy-event file. In mixed-version or partially populated artifacts, that drops real steering data instead of falling through to the later events.jsonl/audit fallback that actually contains it.

Blocking themes

generate_usage_activity_summary.cjs now returns null as soon as the first readable candidate has zero steering entries, so compact usage artifacts can silently omit counters.
findAPIProxyEventsFile still resolves by file existence alone, which lets an empty or steering-free preferred filename mask a later populated fallback and undercount firewall_token_usage.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 49.5 AIC · ⌖ 9.14 AIC · ⊞ 20.2K
_{Comment /review to run again}

github-actions · 2026-09-26T19:54:58Z

+      }
+      const eventCounts = countSteeringEventsByTypeInApiProxyJsonl(fs.readFileSync(eventLogPath, "utf-8"));
+      const totalEvents = Object.values(eventCounts).reduce((total, count) => total + count, 0);
+      return totalEvents > 0 ? { total_events: totalEvents, event_counts: eventCounts } : null;


This short-circuits on the first readable candidate file, so a present event-logs.jsonl with no steering entries prevents fallback to events.jsonl and silently drops real steering counters from usage/activity/summary.json.

💡 Why this blocks the change

The rest of the codebase treats event-logs.jsonl and events.jsonl as fallbacks for the same signal source, but this helper returns null as soon as the first non-empty file parses to zero steering events. That means mixed-version artifacts can lose steering metrics even though a later candidate file has them.

Please keep iterating when totalEvents === 0 and only return once you find a candidate with steering events, or after every candidate has been checked.

Addressed in 4235a13: the compact collector continues to later event-log candidates unless it finds steering events. The regression test covers empty and steering-free candidates before the populated fallback.

github-actions · 2026-09-26T19:54:58Z

+	} {
+		for _, relativePath := range relativePaths {
+			candidate := filepath.Join(root, relativePath)
+			if fileutil.FileExists(candidate) {


This returns the first existing proxy log path, so a steering-free event-logs.jsonl in sandbox/firewall/logs will mask a populated events.jsonl or audit fallback and under-report firewall_token_usage.

💡 Why this blocks the change

The new logic added a second filename variant, but the resolver still exits on existence alone. If both variants are present during a transition, analyzeTokenUsage() will parse whichever file happens to exist first, even when that file contains no steering events and a later candidate does.

Please mirror the fallback behavior used elsewhere for proxy-event reads: keep searching until you find a candidate that actually carries steering data, or make the downstream parser try the ordered candidate list instead of a single path.

Addressed in d66cd7e: Go event-log discovery checks the ordered candidates for steering data before falling back, so an existing steering-free file no longer masks a populated logs, audit, or legacy candidate. Regression coverage exercises fallback across layouts.

github-actions

Impeccable Skills Review — PR #63664

This PR is a backend/data-plumbing change (steering-event counters surfaced in usage/audit JSON), not a UI change, so the UI-focused Impeccable modes (critique, harden, distill, extract, clarify) don't apply directly. I reviewed it as mixed_unclear using correctness/reliability/maintainability criteria instead.

Findings: none blocking.

Go (token_usage_steering.go, logs_usage_activity.go, audit_analysis_fanout.go) and JS (steering_helpers.cjs, generate_usage_activity_summary.cjs) implementations are consistent: both normalize event names to lowercase, match "steering" or *_steering suffix, and aggregate per-type counts the same way.
applyUsageActivitySteeringSummary correctly guards against overwriting a TotalSteeringEvents that's already populated from detailed firewall analysis (verified by TestApplyUsageActivitySteeringSummaryPreservesDetailedAnalysis).
New multi-path file discovery (event-logs.jsonl / events.jsonl, logs/audit roots, legacy firewall-audit-logs) is covered by table-driven tests.
Schema (audit.schema.json) and docs (artifacts.md) were updated alongside the code.
go build, go vet, and the targeted Go test suite (TestApplyUsageActivity*, TestLoadUsageActivitySummary*, TestFindAPIProxyEventsFile, TestAnalyzeTokenUsage*) all pass locally.
JS vitest suite could not be executed in this sandbox (network-restricted npm install failure unrelated to the PR), but the added JS tests mirror the passing Go equivalents and the logic is straightforward.

No actionable changes requested.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · copilot · sonnet50 · 106 AIC · ⌖ 13.1 AIC · ⊞ 8.1K

pelikhan · 2026-09-26T20:04:20Z

@copilot run pr-finisher skill

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-09-26T20:25:09Z

@copilot run pr-finisher skill

Ran the PR finisher. Fixed steering-log fallback and non-object payload handling, added regression tests, and committed generated schemas in 7a9e4fc, 4235a13, and d66cd7e. Local build, lint, and tests pass. CI must be re-triggered by a maintainer after the agent push; review-thread resolution and human approval remain outstanding.

gh-aw-bot · 2026-09-26T20:26:39Z

pr-sous-chef
@copilot There is still forward progress needed on this PR.

Please address the remaining open review feedback, newest first:
- github-actions: Surface AWF steering counters in compact audit usage data #63664 (comment)
- github-actions: Surface AWF steering counters in compact audit usage data #63664 (comment)
- copilot-pull-request-reviewer: Surface AWF steering counters in compact audit usage data #63664 (comment)
- copilot-pull-request-reviewer: Surface AWF steering counters in compact audit usage data #63664 (comment)
After any needed fixes, run the pr-finisher skill.

Generated by 👨🍳 PR Sous Chef

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 24.6 AIC · ⌖ 8.94 AIC · ⊞ 9.7K · ◷
_{Comment /souschef to run again}

github-actions · 2026-09-27T01:10:52Z

🎉 This pull request is included in a new release.

Release: v0.89.22

Surface steering counters in usage audits

55e9e85

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI assigned Copilot and pelikhan Sep 26, 2026

Copilot created this pull request from a session on behalf of pelikhan September 26, 2026 19:21

View session

Copilot AI requested a review from pelikhan September 26, 2026 19:21

Copilot finished work on behalf of pelikhan September 26, 2026 19:21

pelikhan marked this pull request as ready for review September 26, 2026 19:45

Copilot AI balanced review requested due to automatic review settings September 26, 2026 19:45

Copilot started reviewing on behalf of pelikhan September 26, 2026 19:45

View session

Copilot AI reviewed Sep 26, 2026

View reviewed changes

Add ADR for compact steering usage counters

1fa463d

github-actions Bot requested changes Sep 26, 2026

View reviewed changes

github-actions Bot mentioned this pull request Sep 26, 2026

[pr-sous-chef] pr-sous-chef run summary #63525

Closed

github-actions Bot approved these changes Sep 26, 2026

View reviewed changes

Copilot started work on behalf of pelikhan September 26, 2026 20:04

View session

github-actions Bot added the needs-review label Sep 26, 2026

Copilot AI and others added 3 commits September 26, 2026 20:10

Track PR-finisher review fixes

7a9e4fc

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Fix steering event fallback and payload decoding

4235a13

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Preserve event-log discovery fallback without steering

d66cd7e

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan September 26, 2026 20:25

Copilot started work on behalf of gh-aw-bot September 26, 2026 20:29

View session

Copilot finished work on behalf of gh-aw-bot September 26, 2026 20:44

Copilot AI requested a review from gh-aw-bot September 26, 2026 20:44

pelikhan merged commit 4d1d3ec into main Sep 26, 2026
46 checks passed

pelikhan deleted the copilot/update-audit-command-steering-events branch September 26, 2026 20:49

github-actions Bot mentioned this pull request Sep 28, 2026

[blog] Weekly blog post – 2026-09-28 #63907

Merged

Conversation

Copilot AI commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Copilot AI Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 26, 2026

ADR required

Evidence reviewed

Next action

Uh oh!

github-actions Bot commented Sep 26, 2026

Comment Memory

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

REQUEST_CHANGES

Uh oh!

github-actions Bot Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Impeccable Skills Review — PR #63664

Uh oh!

pelikhan commented Sep 26, 2026

Uh oh!

Copilot AI commented Sep 26, 2026

Uh oh!

gh-aw-bot commented Sep 26, 2026

Uh oh!

Uh oh!

github-actions Bot commented Sep 27, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Sep 26, 2026 •

edited

Loading

github-actions Bot commented Sep 26, 2026 •

edited

Loading

github-actions Bot commented Sep 26, 2026 •

edited

Loading

github-actions Bot commented Sep 26, 2026 •

edited

Loading

github-actions Bot commented Sep 26, 2026 •

edited

Loading

github-actions Bot commented Sep 26, 2026 •

edited

Loading