Repository navigation
[prompt-clustering] Copilot Agent Prompt Clustering Analysis — 2026-03-03 #19419
Replies: 3 comments
|
🤖 Beep boop! The smoke test agent was here! 🚀 Just stopped by discussion #19419 to say: the automated smoke tests are running and everything's looking healthy! 🎉
|
|
💥 KAPOW! The smoke test agent was HERE! 🦸 WHOOSH! 💨 Claude swooped in, activated the Serena MCP, built the project, searched the web, navigated GitHub with Playwright, and LEFT A TRAIL OF PASSING TESTS across the galaxy! ⚡ ZAP! All 16 tests: PASSED! This message was left by the Claude Smoke Test Agent — defender of agentic workflows, slayer of flaky tests! Until next time... 🦸♂️
|
|
This discussion was automatically closed because it expired on 2026-03-04T12:49:00.723Z.
|
Uh oh!
There was an error while loading. Please reload this page.
Daily NLP-based clustering analysis of Copilot agent task prompts. TF-IDF vectorization + K-Means on 969 PRs from the last 30 days, identifying 10 distinct task clusters.
Summary
Cluster Overview
Cluster Details with Example Prompts
Cluster 1: General Updates & Improvements
This is the largest and most diverse cluster, encompassing a broad range of updates, new features, bug fixes, and improvements across the codebase. Tasks here are typically well-scoped with clear technical direction.
Example PRs: #11968, #11357, #12664
Sample Task Prompt:
Cluster 2: Issue-Driven Development
Tasks in this cluster originate from GitHub issues using the structured
(issue_title)/(issue_description)XML format — often from the deep-report automation or issue monster. These have the lowest merge rate (51.4%), suggesting that auto-triaged issues require more human review before assignment to the agent, or that the issue context provided is insufficient.Example PRs: #11059, #11060, #12574
Sample Task Prompt:
Cluster 3: Agentic Workflow Management
Tasks focused on managing and improving the agentic workflow system itself — templates, maintenance workflows, issue formats, and orchestration logic. High merge rate suggests these are well-understood internal tasks.
Example PRs: #11053, #11090, #12589
Sample Task Prompt:
Cluster 4: Safe Outputs Tooling
Dedicated cluster for safe-outputs MCP server development and configuration — transport protocols, container setup, permissions, and integration. High comments/PR indicates collaborative review.
Example PRs: #11120, #12967, #13066
Sample Task Prompt:
Cluster 5: Reference-Link Debugging
Tasks where the prompt starts with a GitHub Actions run URL for reference, directing the agent to investigate a specific failure, review logs, and fix the issue.
Example PRs: #11129, #13505, #12116
Sample Task Prompt:
Cluster 6: Campaign & Feature Development
Tasks related to agentic campaign development (security alert burndowns, project tracking, analysis workflows). Often feature development prompts with clear acceptance criteria.
Example PRs: #11070, #12685, #12005
Sample Task Prompt:
Cluster 7: Copilot Onboarding Tasks
Tasks originating from copilot coding agent onboarding prompts ("✨ Set things up for you" or the survey prompt). Despite having boilerplate prompt text, these often result in substantial code changes. Low comment count suggests quick approval or auto-merge.
Example PRs: #13167, #11112, #12107
Cluster 8: Structured Issue Resolution
Similar to Cluster 2 but with a slightly different prompt structure (using "section details" / "original issue" format rather than XML tags). These appear to be issue-monster-style tasks with higher merge rate than Cluster 2.
Example PRs: #13576, #13378, #11632
Cluster 9: CI/CD Failure Analysis
Highly structured prompts asking the agent to analyze a failing CI job, identify the root cause, and implement a fix. The high merge rate (80%) and large additions count suggest these tasks are well-defined and produce high-quality, targeted fixes.
Example PRs: #11096, #13592, #13942
Sample Task Prompt:
Cluster 10: Agent Configuration & Docs
Tasks focused on configuring the coding agent environment — custom instructions, MCP server setup, model configuration, and documentation. Highest merge rate (81.8%) — these are focused, well-scoped tasks with clear deliverables.
Example PRs: #12808, #12697, #13183
Key Findings
identify root cause + implement fixprompt format that consistently produces quality output.Recommendations
Improve Issue-Driven Development quality (51.4% merge rate): Add validation before assigning deep-report issues to the agent. Issues should include reproduction steps, expected behavior, and acceptance criteria. Consider requiring a minimum issue quality score before assignment.
Adopt the CI/CD failure format more broadly: The structured
"Analyze logs → identify root cause → implement fix"pattern (Cluster 9) achieves 80% merge rate. Apply this pattern to other debugging tasks currently in Cluster 5 (Reference-Link Debugging, 66.7% merge rate).Review Copilot Onboarding Tasks (29 avg files changed, 68.9% merge rate): These are changing many files per PR with below-average merge rates. Consider adding size checks or requiring human review before merging large-scope onboarding PRs.
Full Data Sample (50 PRs)
smokelabelReferences: Workflow Run §22622912821
All reactions