Repository navigation
Persist Copilot session telemetry and summarize tool performance - #75706
Conversation
… sizes Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
| conversation_out="/tmp/gh-aw/agent/session-data/logs/${run_id}-conversation.txt" | ||
| rm -f "$conversation_out" | ||
| while IFS= read -r -d '' conversation_file; do | ||
| cat "$conversation_file" >> "$conversation_out" |
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft. Warning Threat detection engine error Threat detection could not complete its analysis due to a tooling error. This is not a security finding. What happenedThe threat detection results could not be parsed. Review the workflow run logs to troubleshoot the error. Design Decision Gate 🏗️Outcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ Test Quality Sentinel completed test quality analysis. Warning Threat detection engine error Threat detection could not complete its analysis due to a tooling error. This is not a security finding. What happenedThe threat detection results could not be parsed. Review the workflow run logs to troubleshoot the error. Test Quality SentinelOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
🧠 Matt Pocock Skills Reviewer is reviewing this pull request using Matt Pocock's engineering skills... |
|
✅ Ponytail Reviewer completed successfully!
|
|
✅ PR Code Quality Reviewer completed the code quality review.
|
|
✅ PR Specification Reviewer completed successfully! PR Specification ReviewerOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved retrieval and parsing defects can omit qualifying sessions, misattribute interleaved agent calls, and let empty artifacts suppress log fallback.
3 open issues
What changed in this PR
This PR addresses the telemetry gap in #72152 by extending gh-aw’s shared Copilot session fetcher and exposing tool measurements to optimization analysis.
Changes:
- Select coding-agent runs, paginate artifact/job listings, retain conversation artifacts, and refresh unsuitable caches.
- Normalize null conclusions and generate tool latency, payload-size, and data-quality summaries.
- Preserve structured event logs, update analysis guidance, and add regression tests.
| File | Description |
|---|---|
scripts/copilot-session-data-fetch.test.cjs |
Adds coverage for fetching, caching, summaries, and event preservation. |
.github/workflows/shared/copilot-session-data-fetch.md |
Updates session collection and generates tool summaries. |
.github/workflows/copilot-session-insights.lock.yml |
Regenerates the insights workflow with the updated fetcher. |
.github/workflows/copilot-opt.md |
Preserves event logs and guides analysis using summaries. |
.github/workflows/copilot-opt.lock.yml |
Regenerates the optimization workflow with collection and trimming changes. |
Review effort: Balanced (auto)
| gh api "repos/$GITHUB_REPOSITORY/actions/runs?per_page=100&created=>=${DATE_30_DAYS_AGO}" \ | ||
| --paginate \ | ||
| --jq ".workflow_runs[] | select(.head_branch | startswith(\"copilot/\")) | select(.created_at >= \"${DATE_30_DAYS_AGO}\") | {id, name, head_branch, created_at, updated_at, status, conclusion, html_url}" \ | ||
| --jq '.workflow_runs[] | select(.path == "dynamic/copilot-swe-agent/copilot") | {id, name, path, head_branch, created_at, updated_at, status, conclusion: (.conclusion // (if .status == "completed" then "unknown" else .status // "unknown" end)), html_url}' \ |
| cat "$events_file" >> "$events_tmp" | ||
| done | ||
| printf '\n' >> "$events_tmp" |
| if event["type"] == "session.start": | ||
| scope = data.get("sessionId", event.get("id")) | ||
| call_id = data.get("toolCallId") | ||
| key = json.dumps([scope, event.get("agentId"), call_id]) |
|
🎉 This pull request is included in a new release. Release: |

Copilot optimization analysis had session metadata but no usable conversation telemetry, preventing latency, payload-size, and validation analysis. Null conclusions also disrupted outcome grouping.
Session collection
dynamic/copilot-swe-agent/copilotruns rather than ordinary CI oncopilot/*branches.Analysis integrity
unknownfor completed runs.events.jsonlfiles instead of byte-trimming them.Tool measurements
tool-summaries.jsonfrom fresh or cached logs, pairing calls by session, agent, and call ID.null.