Repository navigation
Make incomplete-task failures actionable - #77183
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
| // eslint-disable-next-line gh-aw-custom/require-return-after-core-setfailed | ||
| core.setFailed("Agent reported that it could not complete the task"); | ||
| core.setFailed( | ||
| "Agent reported that it could not complete the task (report_incomplete). A successful agent job does not confirm task completion. See the conclusion job summary for the reported reason and runtime evidence; inspect agent_output.json in the agent artifact for the original report." |
There was a problem hiding this comment.
Updated the failure annotation to recommend gh aw audit <run-id> for runtime evidence. Included in b903a75e.
There was a problem hiding this comment.
The existing change already uses gh aw audit <run-id> for runtime evidence, and that guidance remains in the updated annotation.
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft. Design Decision Gate 🏗️Outcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ PR Code Quality Reviewer completed the code quality review. PR Code Quality ReviewerOutcome: The agent completed and reported one or more actions. The no-op output is a completion signal, not a claim that no action was taken. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ PR Specification Reviewer completed successfully! PR Specification ReviewerOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ Ponytail Reviewer completed successfully! Ponytail ReviewerOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
@copilot make the message in the generated issue more delightful with more context about the failure (agentically safe) |
|
✅ Test Quality Sentinel completed test quality analysis. Test Quality SentinelOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅
|
Comment MemoryPeek at saved memory (pr-code-quality-reviewer)Note This comment is managed by comment memory.Expand the saved memory block to view or edit the persistent context for this thread.
|
There was a problem hiding this comment.
Found one medium-severity correctness issue: the new failure annotation can promise a conclusion summary that is not written for empty/invalid reasons or redaction-failure paths. I left an inline comment with the impact and suggested conditional guidance/tests.
🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~2 AIC · ⊞ 1.6K
Comment /review to run again
| // eslint-disable-next-line gh-aw-custom/require-return-after-core-setfailed | ||
| core.setFailed("Agent reported that it could not complete the task"); | ||
| core.setFailed( | ||
| "Agent reported that it could not complete the task (report_incomplete). A successful agent job does not confirm task completion. See the conclusion job summary for the reported reason and runtime evidence; inspect agent_output.json in the agent artifact for the original report." |
There was a problem hiding this comment.
The failure annotation always promises a conclusion job summary even when this report_incomplete item produces no summary, leaving maintainers without the promised diagnostic.
💡 Keep the guidance conditional on available diagnostics
buildReportIncompleteContext can return empty content for an empty/whitespace reason (and the summary is also skipped after secret-redaction failure), while this core.setFailed message is emitted for every report_incomplete item. The existing empty-reason tests cover that reachable path but do not assert the misleading annotation. Only point to the conclusion summary when it was actually written, or use wording that does not promise it; add coverage for empty reasons and redaction failures.
There was a problem hiding this comment.
Updated in 00e2beb: the annotation only refers to the conclusion summary conditionally, directs blank-reason cases to agent_output.json in the agent artifact, and now reflects when secret-redaction failure suppresses the summary. Added regression coverage for blank reasons and redaction failure.
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the incomplete-task diagnostic changes.
No open issues 🎉
What changed in this PR
This PR makes gh-aw incomplete-task failures actionable while preserving failure semantics and keeping agent-provided payloads out of annotations and logs.
Changes:
- Expands the failure annotation to identify
report_incomplete, distinguish job success from task completion, and point to diagnostic artifacts. - Adds regression coverage for clean-exit incompletion with issue reporting disabled, checking privacy and summary diagnostics, and updates existing message assertions.
| File | Description |
|---|---|
actions/setup/js/handle_agent_failure.test.cjs |
Tests actionable diagnostics, payload privacy, and unchanged failure behavior. |
actions/setup/js/handle_agent_failure.cjs |
Adds diagnostic guidance to the incomplete-task failure annotation. |
Review effort: Balanced (auto)
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /diagnosing-bugs and /tdd. Overall the change is sound: the diagnostic points at the right artifact, the failure semantics are unchanged, and the new test checks that the agent payload stays out of logs. One edge case to fix before merge.
📋 Key Themes & Highlights
Key Themes
- Blank-reason mismatch: the
setFailedmessage always points to the conclusion job summary, but a blankreport_incompletereason produces no summary. See the inline comment.
Positive Highlights
- ✅ Privacy holds: no
core.error,core.notice, orcore.debugcall logs the reason or details. - ✅ The
agent_output.jsonpointer is accurate. It is part of theagentartifact whenever safe outputs are configured. - ✅ The
eslint-disable-next-linestill covers the multi-linesetFailedcall, because the rule reports on the statement node. - ✅ The new test asserts both the message content and the payload-leak exclusions.
Limitations
- I could not run the Vitest suite.
npm cifails on a TLS certificate error from the package proxy, so the test results are unverified. Please confirm CI is green.
@copilot please address the review comments above.
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~6 AIC · ⌖ ~2 AIC · ⊞ 6.6K
Comment /matt to run again
| // eslint-disable-next-line gh-aw-custom/require-return-after-core-setfailed | ||
| core.setFailed("Agent reported that it could not complete the task"); | ||
| core.setFailed( | ||
| "Agent reported that it could not complete the task (report_incomplete). A successful agent job does not confirm task completion. See the conclusion job summary for the reported reason and runtime evidence; inspect agent_output.json in the agent artifact for the original report." |
There was a problem hiding this comment.
[/diagnosing-bugs] The new message is misleading when a report_incomplete has an empty or whitespace-only reason. Line 4547 counts every report_incomplete item, but loadReportIncompleteMessages drops blank reasons, so buildReportIncompleteContext returns nothing and no conclusion summary is written. The annotation still sends maintainers to a summary that does not exist.
💡 Suggested fix and test
Choose the pointer from whether a usable reason exists:
- Non-empty reason: keep the current wording.
- Blank reason: say that
report_incompletewas emitted without a reason, and point only toagent_output.jsonin the agent artifact.
Add a blank-reason case next to the new clean-exit test, following the existing invalid-reason it.each pattern. Assert that the message names agent_output.json and does not mention the conclusion job summary.
@copilot please address this.
There was a problem hiding this comment.
Updated in 00e2beb: blank or whitespace-only reasons now say no usable reason was provided and point to agent_output.json in the agent artifact without referring to a conclusion summary. Added regression coverage for both blank-reason cases.
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Expanded the generated incomplete-result context with a safe handoff summary, distinction between agent-job success and task completion, and a run-specific audit command while keeping agent diagnostics out. Included in |
|
🍳 A little prep to help this PR along! Copilot assignment requested for the following outstanding work:
Keeping the changes focused; human review and CI get the final taste test. No merging or CI triggering. Structured data{
"completed_task": 0,
"head": "b903a75e04919f0fba688472fa3b163ac2e05c9c",
"kind": "requested",
"schema_version": "1",
"state": "d6078db5eb6679273f8259ec080b4b9e1b802811cb7826e077282601d6e44114",
"work": [
"3a19f76a38d378f38475e7ef0fca63664677d14805fe280c8111e1e6890ce037",
"786c321426ad4f0ffba258f2aa239744cdd08e2e957dc4a03f324a8db7efab3a"
],
"workflow": "pr-sous-chef"
}
|
…ter-diagnostic-message Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
|
🎉 This pull request is included in a new release. Release: |
The linked run’s agent exited successfully but reported blocked compilation. The conclusion correctly failed, yet its annotation offered no diagnostic next step; the underlying build-cache provisioning is already addressed on main.
report_incomplete, distinguish agent-job success from task completion, and direct maintainers to the conclusion summary and agent artifact’sagent_output.json.