Skip to content

Persist Copilot session telemetry and summarize tool performance - #75706

Merged
pelikhan merged 2 commits into
mainfrom
copilot/copilot-opt-fix-fetch-step
Nov 7, 2026
Merged

pelikhan merged 2 commits into
mainfrom
copilot/copilot-opt-fix-fetch-step

Conversation

Copilot AI commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Copilot optimization analysis had session metadata but no usable conversation telemetry, preventing latency, payload-size, and validation analysis. Null conclusions also disrupted outcome grouping.

  • Session collection

    • Select dynamic/copilot-swe-agent/copilot runs rather than ordinary CI on copilot/* branches.
    • Paginate artifact/job listings; retain event and conversation artifacts, including available telemetry from unfinished runs.
    • Refresh legacy, incomplete, or unfinished-session caches.
  • Analysis integrity

    • Normalize null conclusions to the unfinished run status or unknown for completed runs.
    • Preserve complete events.jsonl files instead of byte-trimming them.
  • Tool measurements

    • Generate tool-summaries.json from fresh or cached logs, pairing calls by session, agent, and call ID.
    • Report latency, UTF-8 input/result sizes, and invalid or unmatched/incomplete events without copying payload contents.
    • Mark transcript-only measurements unavailable; retain missing values as null.

… sizes

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix fetch step to include Copilot session logs and events Persist Copilot session telemetry and summarize tool performance Nov 7, 2026
Copilot AI requested a review from gh-aw-bot November 7, 2026 16:42
conversation_out="/tmp/gh-aw/agent/session-data/logs/${run_id}-conversation.txt"
rm -f "$conversation_out"
while IFS= read -r -d '' conversation_file; do
cat "$conversation_file" >> "$conversation_out"
@pelikhan
pelikhan marked this pull request as ready for review November 7, 2026 17:28
Copilot AI dynamic review requested due to automatic review settings November 7, 2026 17:28
@pelikhan
pelikhan merged commit 1011890 into main Nov 7, 2026
45 checks passed
@pelikhan
pelikhan deleted the copilot/copilot-opt-fix-fetch-step branch November 7, 2026 17:28
@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Warning

Threat detection engine error

Threat detection could not complete its analysis due to a tooling error. This is not a security finding.

What happened

The threat detection results could not be parsed.

Review the workflow run logs to troubleshoot the error.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · routed: opus55 medium · ~12 AIC · ⌖ ~2 AIC · ⊞ 7.7K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Warning

Threat detection engine error

Threat detection could not complete its analysis due to a tooling error. This is not a security finding.

What happened

The threat detection results could not be parsed.

Review the workflow run logs to troubleshoot the error.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · routed: gpt56 max · ~1 AIC · ⌖ ~2 AIC · ⊞ 6.3K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer is reviewing this pull request using Matt Pocock's engineering skills...

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Generated by Ponytail Reviewer for #75706

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Specification Reviewer completed successfully!

PR Specification Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from PR Specification Reviewer · gpt56 · ~1 AIC · ⌖ ~1 AIC · ⊞ 1.7K · ◷

Generated by PR Specification Reviewer for #75706

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved retrieval and parsing defects can omit qualifying sessions, misattribute interleaved agent calls, and let empty artifacts suppress log fallback.

3 open issues
What changed in this PR

This PR addresses the telemetry gap in #72152 by extending gh-aw’s shared Copilot session fetcher and exposing tool measurements to optimization analysis.

Changes:

  • Select coding-agent runs, paginate artifact/job listings, retain conversation artifacts, and refresh unsuitable caches.
  • Normalize null conclusions and generate tool latency, payload-size, and data-quality summaries.
  • Preserve structured event logs, update analysis guidance, and add regression tests.
File Description
scripts/​copilot-session-data-fetch.test.cjs Adds coverage for fetching, caching, summaries, and event preservation.
.github/​workflows/​shared/​copilot-session-data-fetch.md Updates session collection and generates tool summaries.
.github/​workflows/​copilot-session-insights.lock.yml Regenerates the insights workflow with the updated fetcher.
.github/​workflows/​copilot-opt.md Preserves event logs and guides analysis using summaries.
.github/​workflows/​copilot-opt.lock.yml Regenerates the optimization workflow with collection and trimming changes.

Review effort: Balanced (auto)

Comment on lines +129 to +131
gh api "repos/$GITHUB_REPOSITORY/actions/runs?per_page=100&created=>=${DATE_30_DAYS_AGO}" \
--paginate \
--jq ".workflow_runs[] | select(.head_branch | startswith(\"copilot/\")) | select(.created_at >= \"${DATE_30_DAYS_AGO}\") | {id, name, head_branch, created_at, updated_at, status, conclusion, html_url}" \
--jq '.workflow_runs[] | select(.path == "dynamic/copilot-swe-agent/copilot") | {id, name, path, head_branch, created_at, updated_at, status, conclusion: (.conclusion // (if .status == "completed" then "unknown" else .status // "unknown" end)), html_url}' \
Comment on lines 184 to +185
cat "$events_file" >> "$events_tmp"
done
printf '\n' >> "$events_tmp"
Comment on lines +351 to +354
if event["type"] == "session.start":
scope = data.get("sessionId", event.get("id"))
call_id = data.get("toolCallId")
key = json.dumps([scope, event.get("agentId"), call_id])
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.2.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[copilot-opt] Persist Copilot session logs/events.jsonl for optimization analysis

5 participants