Skip to content

Preserve retry diagnostics in external threat-detection artifacts - #74940

Closed
pelikhan with Copilot wants to merge 8 commits into
mainfrom
copilot/fix-retry-diagnostics-upload
Closed

pelikhan with Copilot wants to merge 8 commits into
mainfrom
copilot/fix-retry-diagnostics-upload

Conversation

Copilot AI commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

External threat detection omits retry diagnostics from its artifact, preventing the conclusion job from collecting them into the usage artifact.

  • Artifact transfer: Include detection_harness_retry.json in the external detection upload, preserving existing redaction conditions and missing-file handling.
  • Regression coverage: Assert inclusion of the retry file, retention of the redaction gate, and exclusion of the raw detection log.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix external threat-detection artifact to include retry diagnostics Preserve retry diagnostics in external threat-detection artifacts Nov 4, 2026
Copilot AI requested a review from pelikhan November 4, 2026 19:58
@pelikhan
pelikhan marked this pull request as ready for review November 5, 2026 15:44
Copilot AI dynamic review requested due to automatic review settings November 5, 2026 15:44
@pelikhan

pelikhan commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

@copilot run pr-finisher skill

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · routed: opus55 medium · ~12 AIC · ⌖ ~2 AIC · ⊞ 7.7K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · routed: gpt56 max · ~1 AIC · ⌖ ~1 AIC · ⊞ 118 · ◷

Generated by Ponytail Reviewer for #74940

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · routed: gpt56 max · ~1 AIC · ⌖ ~1 AIC · ⊞ 6.3K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the change preserving external threat-detection retry diagnostics across jobs.

No open issues 🎉

What changed in this PR

This PR addresses #73006 by preserving external threat-detection retry diagnostics across jobs so the conclusion job can collect them into the usage artifact.

Changes:

  • Adds detection_harness_retry.json to the external detection artifact upload.
  • Adds regression coverage for retry-file inclusion, the existing redaction gate, missing-file handling, and raw-log exclusion.
File Description
pkg/​workflow/​threat_detection_steps_test.go Tests retry-diagnostic inclusion and existing upload safeguards.
pkg/​workflow/​threat_detection_external.go Includes retry diagnostics in the external detection artifact.

Review effort: Balanced (auto)

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review complete: no actionable issues found in the changed lines. The external artifact upload now includes the retry diagnostics file, preserves the existing redaction gate and missing-file handling, and adds focused regression coverage. Static review only; tests were not run.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~2 AIC · ⌖ ~1 AIC · ⊞ 1.6K
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs and /tdd. The source fix is correct and well-tested, but the compiled lock files are stale, so the fix does not reach workflows yet.

📋 Key Findings

Key Findings

  • Stale lock files (blocking): 258 .lock.yml files use the external threat-detection artifact step but lack detection_harness_retry.json. Recompiling ab-testing-advisor.lock.yml adds exactly that one line. Run make recompile and commit the regenerated locks.
  • Regression test (verified): TestBuildUploadDetectionArtifactStep fails without the one-line change and passes with it. It asserts inclusion of the retry file, the redaction gate, if-no-files-found: ignore, and exclusion of detection.log.
  • Safety: detection_harness_retry.json holds only attempt metadata (counts, timings, failure classes), not raw transcript content, so uploading it does not create the exfiltration path the function comment warns about.

Positive Highlights

  • ✅ Redaction gate retained and asserted.
  • ✅ Raw detection.log exclusion covered by a NotContains assertion.

Uncertainty

  • The external (non-ARC) detector path has no retry-copy step, unlike the inline ARC path. I did not confirm that the harness writes detection_harness_retry.json into /tmp/gh-aw/threat-detection/ there. If it does not, the upload is a no-op, which is harmless but means the fix may not deliver retry data.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~5 AIC · ⌖ ~1 AIC · ⊞ 5.4K
Comment /matt to run again

" " + detectionExecutionEvidencePath + "\n",
" " + constants.TmpGhAwDir + "/threat-detection/detection_usage.json\n",
" " + constants.TmpGhAwDir + "/threat-detection/detection_usage.jsonl\n",
" " + constants.TmpGhAwDir + "/threat-detection/detection_harness_retry.json\n",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] The source fix is right, but compiled workflows do not pick it up yet. The 258 external-detection .lock.yml files still lack detection_harness_retry.json. Recompiling ab-testing-advisor.lock.yml adds exactly this one line. Run make recompile and commit the regenerated locks so the fix ships.

💡 How to verify

Run make recompile and inspect git diff --stat -- .github/workflows/*.lock.yml. Each external-detection lock should gain one line: /tmp/gh-aw/threat-detection/detection_harness_retry.json.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make merge-main completed and ran make recompile; no new lockfile diff was needed because the retry path is already present in all 258 workflow lockfiles changed by 0fc1098. I verified each of those locks contains exactly one detection_harness_retry.json entry. The branch now also includes main at 1580090 via merge commit 347cf85.

Copilot AI and others added 2 commits November 5, 2026 16:04
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

@copilot run pr-finisher skill

Finisher pass found no unresolved review threads. I fixed the failing WASM golden check in local commits 0fc1098 and f1d6eb6; local validation passes. Both pushes were rejected by commit_refs_rules, so the commits are not yet on the PR. CI still needs a maintainer re-trigger after the changes are published.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot

Copilot assignment requested for the following outstanding work:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Review (pkg/workflow/threat_detection_external.go:804): [/diagnosing-bugs] The source fix is right, but compiled workflows do not pick it up yet. The 258 external-detection .lock.yml files still lack detection_harness_retry.json. Recompiling ab-testing-advisor.lock.yml adds exactly this one line. Run make recompile and commit the regenerated locks so the fix ships. - Preserve retry diagnostics in external threat-detection artifacts #74940 (comment)

Once you start work, post this work summary as a PR comment without the leading Copilot mention, including the Sous-chef head, work, and state lines below, to record that the assignment is running. Apply the necessary fixes on this PR's existing branch, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved; stop when only human review or CI remains. Do not merge or trigger CI.

Sous-chef requested-head: f1d6eb6
Sous-chef requested-work: e9f2e9b720919b0ae5f12266435b406ea18ac50ebcd9b18595c256f16e82e21d
Sous-chef requested-state: 0c5d669946634ae816b1489c525081a9ed32563a5b229551666b8174590b6807

Generated by 👨🍳 PR Sous Chef · pi · haiku45 · ~5 AIC · ⌖ ~12 AIC · ⊞ 1.2K · ◷
Comment /souschef to run again

Copilot AI commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

@copilot

Copilot assignment requested for the following outstanding work:...

Starting the requested branch update and workflow recompilation for the external-detection retry artifact fix.

Sous-chef requested-head: f1d6eb6
Sous-chef requested-work: e9f2e9b720919b0ae5f12266435b406ea18ac50ebcd9b18595c256f16e82e21d
Sous-chef requested-state: 0c5d669946634ae816b1489c525081a9ed32563a5b229551666b8174590b6807

…gnostics-upload

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
@pelikhan

pelikhan commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

@copilot try to reuse the existing files that are already stored in artifacts

Copilot AI commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

@copilot try to reuse the existing files that are already stored in artifacts

The branch already contains the generated lockfiles, so I reused them instead of regenerating them. I verified all 258 lockfiles updated in 0fc1098 include exactly one detection_harness_retry.json entry; the merge with current main is already at 347cf85.

@pelikhan

pelikhan commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

@copilot try to reuse the existing json files that are already stored in artifacts instead of adding a new file

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

@copilot try to reuse the existing json files that are already stored in artifacts instead of adding a new file

Updated in 47e0fe75: detection retry telemetry is now merged into the existing execution.json artifact, with no standalone retry JSON uploaded. Added legacy compatibility coverage and regenerated the workflow locks.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot

Copilot assignment requested for the following outstanding work:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Fix failing check build-wasm (FAILURE): https://github.com/github/gh-aw/actions/runs/42162650666/job/127711849081.
  3. Fix failing check impacted-go-tests (FAILURE): https://github.com/github/gh-aw/actions/runs/42162650666/job/127711848986.

Once you start work, post this work summary as a PR comment without the leading Copilot mention, including the Sous-chef head, work, and state lines below, to record that the assignment is running. Apply the necessary fixes on this PR's existing branch, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved; stop when only human review or CI remains. Do not merge or trigger CI.

Sous-chef requested-head: 47e0fe7
Sous-chef requested-work: 42e5d572d9d75b7168fe7d0aeda0717cc07e65ed240f6d299e6fec661214f1ac 8acb61a077717e3a460018aacdfde96fd489f77606cd36169e2328d9079b75bb
Sous-chef requested-state: 645b485d00b268717bd4065a2a264d952d8c0fcec0b005a57f7d05617945c9f3

Generated by 👨🍳 PR Sous Chef · pi · haiku45 · ~4 AIC · ⌖ ~12 AIC · ⊞ 1.2K · ◷
Comment /souschef to run again

…gnostics-upload

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot AI commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

@copilot

Copilot assignment requested for the following outstanding work:...

Starting the requested main-branch merge and CI failure fixes.

Sous-chef requested-head: f1d6eb6
Sous-chef requested-work: 42e5d572d9d75b7168fe7d0aeda0717cc07e65ed240f6d299e6fec661214f1ac 8acb61a077717e3a460018aacdfde96fd489f77606cd36169e2328d9079b75bb
Sous-chef requested-state: 645b485d00b268717bd4065a2a264d952d8c0fcec0b005a57f7d05617945c9f3

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
@pelikhan pelikhan closed this Nov 6, 2026
@github-actions
github-actions Bot deleted the copilot/fix-retry-diagnostics-upload branch November 14, 2026 02:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

External threat-detection artifact omits retry diagnostics

4 participants