Skip to content

Make incomplete-task failures actionable - #77183

Merged
pelikhan merged 4 commits into
mainfrom
copilot/fix-issue-better-diagnostic-message
Nov 12, 2026
Merged

pelikhan merged 4 commits into
mainfrom
copilot/fix-issue-better-diagnostic-message

Conversation

Copilot AI commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

The linked run’s agent exited successfully but reported blocked compilation. The conclusion correctly failed, yet its annotation offered no diagnostic next step; the underlying build-cache provisioning is already addressed on main.

  • Diagnostics: Identify report_incomplete, distinguish agent-job success from task completion, and direct maintainers to the conclusion summary and agent artifact’s agent_output.json.
  • Privacy: Keep agent-provided reasons and details out of failure annotations and logs.
  • Regression coverage: Cover clean-exit incompletion with issue reporting disabled, preserving failure semantics and summary diagnostics.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title Improve diagnostics for agent-reported incomplete tasks Make incomplete-task failures actionable Nov 12, 2026
Copilot AI requested a review from pelikhan November 12, 2026 15:23
@pelikhan
pelikhan marked this pull request as ready for review November 12, 2026 15:23
Copilot AI dynamic review requested due to automatic review settings November 12, 2026 15:23
// eslint-disable-next-line gh-aw-custom/require-return-after-core-setfailed
core.setFailed("Agent reported that it could not complete the task");
core.setFailed(
"Agent reported that it could not complete the task (report_incomplete). A successful agent job does not confirm task completion. See the conclusion job summary for the reported reason and runtime evidence; inspect agent_output.json in the agent artifact for the original report."

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot recommend using audit instead.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the failure annotation to recommend gh aw audit <run-id> for runtime evidence. Included in b903a75e.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The existing change already uses gh aw audit <run-id> for runtime evidence, and that guidance remains in the updated annotation.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ⊞ 8.5K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

PR Code Quality Reviewer

Outcome: reported_action

The agent completed and reported one or more actions. The no-op output is a completion signal, not a claim that no action was taken.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from PR Code Quality Reviewer · gpt56 · ~3 AIC · ⌖ ~2 AIC · ⊞ 1.6K · ◷

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Specification Reviewer completed successfully!

PR Specification Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from PR Specification Reviewer · gpt56 · ~2 AIC · ⌖ ~2 AIC · ⊞ 1.6K · ◷

Generated by PR Specification Reviewer for #77183

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · routed: gpt61 medium · ~4 AIC · ⌖ ~1 AIC · ⊞ 181 · ◷

Generated by Ponytail Reviewer for #77183

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot make the message in the generated issue more delightful with more context about the failure (agentically safe)

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ⊞ 7.2K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

Copy link
Copy Markdown
Contributor

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)
reviewed_at: 2026-11-12T15:30:00Z
review_event: COMMENT
top_themes:
  - failure diagnostics must not promise summaries that are skipped on empty or redaction-failure paths
files_reviewed:
  - actions/setup/js/handle_agent_failure.cjs
  - actions/setup/js/handle_agent_failure.test.cjs
comment_count: 1

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~2 AIC · ⊞ 1.6K · ◷
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one medium-severity correctness issue: the new failure annotation can promise a conclusion summary that is not written for empty/invalid reasons or redaction-failure paths. I left an inline comment with the impact and suggested conditional guidance/tests.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~2 AIC · ⊞ 1.6K
Comment /review to run again

// eslint-disable-next-line gh-aw-custom/require-return-after-core-setfailed
core.setFailed("Agent reported that it could not complete the task");
core.setFailed(
"Agent reported that it could not complete the task (report_incomplete). A successful agent job does not confirm task completion. See the conclusion job summary for the reported reason and runtime evidence; inspect agent_output.json in the agent artifact for the original report."

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The failure annotation always promises a conclusion job summary even when this report_incomplete item produces no summary, leaving maintainers without the promised diagnostic.

💡 Keep the guidance conditional on available diagnostics

buildReportIncompleteContext can return empty content for an empty/whitespace reason (and the summary is also skipped after secret-redaction failure), while this core.setFailed message is emitted for every report_incomplete item. The existing empty-reason tests cover that reachable path but do not assert the misleading annotation. Only point to the conclusion summary when it was actually written, or use wording that does not promise it; add coverage for empty reasons and redaction failures.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 00e2beb: the annotation only refers to the conclusion summary conditionally, directs blank-reason cases to agent_output.json in the agent artifact, and now reflects when secret-redaction failure suppresses the summary. Added regression coverage for blank reasons and redaction failure.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the incomplete-task diagnostic changes.

No open issues 🎉

What changed in this PR

This PR makes gh-aw incomplete-task failures actionable while preserving failure semantics and keeping agent-provided payloads out of annotations and logs.

Changes:

  • Expands the failure annotation to identify report_incomplete, distinguish job success from task completion, and point to diagnostic artifacts.
  • Adds regression coverage for clean-exit incompletion with issue reporting disabled, checking privacy and summary diagnostics, and updates existing message assertions.
File Description
actions/​setup/​js/​handle_agent_failure.test.cjs Tests actionable diagnostics, payload privacy, and unchanged failure behavior.
actions/​setup/​js/​handle_agent_failure.cjs Adds diagnostic guidance to the incomplete-task failure annotation.

Review effort: Balanced (auto)

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs and /tdd. Overall the change is sound: the diagnostic points at the right artifact, the failure semantics are unchanged, and the new test checks that the agent payload stays out of logs. One edge case to fix before merge.

📋 Key Themes & Highlights

Key Themes

  • Blank-reason mismatch: the setFailed message always points to the conclusion job summary, but a blank report_incomplete reason produces no summary. See the inline comment.

Positive Highlights

  • ✅ Privacy holds: no core.error, core.notice, or core.debug call logs the reason or details.
  • ✅ The agent_output.json pointer is accurate. It is part of the agent artifact whenever safe outputs are configured.
  • ✅ The eslint-disable-next-line still covers the multi-line setFailed call, because the rule reports on the statement node.
  • ✅ The new test asserts both the message content and the payload-leak exclusions.

Limitations

  • I could not run the Vitest suite. npm ci fails on a TLS certificate error from the package proxy, so the test results are unverified. Please confirm CI is green.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~6 AIC · ⌖ ~2 AIC · ⊞ 6.6K
Comment /matt to run again

// eslint-disable-next-line gh-aw-custom/require-return-after-core-setfailed
core.setFailed("Agent reported that it could not complete the task");
core.setFailed(
"Agent reported that it could not complete the task (report_incomplete). A successful agent job does not confirm task completion. See the conclusion job summary for the reported reason and runtime evidence; inspect agent_output.json in the agent artifact for the original report."

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] The new message is misleading when a report_incomplete has an empty or whitespace-only reason. Line 4547 counts every report_incomplete item, but loadReportIncompleteMessages drops blank reasons, so buildReportIncompleteContext returns nothing and no conclusion summary is written. The annotation still sends maintainers to a summary that does not exist.

💡 Suggested fix and test

Choose the pointer from whether a usable reason exists:

  • Non-empty reason: keep the current wording.
  • Blank reason: say that report_incomplete was emitted without a reason, and point only to agent_output.json in the agent artifact.

Add a blank-reason case next to the new clean-exit test, following the existing invalid-reason it.each pattern. Assert that the message names agent_output.json and does not mention the conclusion job summary.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 00e2beb: blank or whitespace-only reasons now say no usable reason was provided and point to agent_output.json in the agent artifact without referring to a conclusion summary. Added regression coverage for both blank-reason cases.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

@copilot make the message in the generated issue more delightful with more context about the failure (agentically safe)

Expanded the generated incomplete-result context with a safe handoff summary, distinction between agent-job success and task completion, and a run-specific audit command while keeping agent diagnostics out. Included in b903a75e.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot

🍳 A little prep to help this PR along! Copilot assignment requested for the following outstanding work:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Review (actions/setup/js/handle_agent_failure.cjs:4557): address unresolved feedback - Review comment
  3. Review (actions/setup/js/handle_agent_failure.cjs:4557): address unresolved feedback - Review comment

Keeping the changes focused; human review and CI get the final taste test. No merging or CI triggering.

Structured data
{
  "completed_task": 0,
  "head": "b903a75e04919f0fba688472fa3b163ac2e05c9c",
  "kind": "requested",
  "schema_version": "1",
  "state": "d6078db5eb6679273f8259ec080b4b9e1b802811cb7826e077282601d6e44114",
  "work": [
    "3a19f76a38d378f38475e7ef0fca63664677d14805fe280c8111e1e6890ce037",
    "786c321426ad4f0ffba258f2aa239744cdd08e2e957dc4a03f324a8db7efab3a"
  ],
  "workflow": "pr-sous-chef"
}

Generated by 👨🍳 PR Sous Chef · pi · routed: haiku55 xhigh · ~4 AIC · ⌖ ~2 AIC · ⊞ 6.9K · ◷
Comment /souschef to run again

Copilot AI and others added 2 commits November 12, 2026 16:22
…ter-diagnostic-message

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot AI requested a review from gh-aw-bot November 12, 2026 16:45
@pelikhan
pelikhan merged commit 770ce9c into main Nov 12, 2026
76 checks passed
@pelikhan
pelikhan deleted the copilot/fix-issue-better-diagnostic-message branch November 12, 2026 17:27
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.2.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants