Skip to content

Show detector-defined threat categories in generated warnings - #75774

Merged
pelikhan merged 3 commits into
mainfrom
copilot/detection-threat-category-messages
Nov 8, 2026
Merged

pelikhan merged 3 commits into
mainfrom
copilot/detection-threat-category-messages

Conversation

Copilot AI commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Export only schema-defined boolean findings (prompt_injection, secret_leak, malicious_patch) from both inline and external detector conclusion paths, and propagate them to safe-output and conclusion jobs.
  • Include framework-authored threat types in generated issue/body warnings, strict-failure notifications, tracking comments, request-changes reviews, converted review PRs, expired-entity warnings, and blocked-output messages. Never use detector-generated explanations as public threat details; unknown categories are omitted, and missing types are reported explicitly.
  • Preserve distinct detector-error warnings and threat markers. Use typed report blocks in the updated publication paths, document the message placeholders, and regenerate workflow locks. The 317 generated lock-file changes only add category outputs and environment wiring.

Validation

  • make agent-report-progress passed, including Go/custom/JavaScript/shell lint, impacted tests, schema freshness, and full workflow drift checking.
  • make recompile: 361/361 workflows compiled successfully.
  • Focused Vitest publication suites: 1,102 tests passed; an additional generated-issue regression passed, and the final gate ran 747 impacted JavaScript tests.
  • npm run typecheck passed.
  • go test ./pkg/workflow -run '^(TestConcludeThreatDetectionScript_.*|TestDetectionJobHasSuccessOutput|TestSafeOutputJobsCheckDetectionSuccess|TestDetectionRunsStepInConclusionJob)$' -count=1 passed.

No workflows were dispatched.

@pelikhan
pelikhan marked this pull request as ready for review November 7, 2026 23:56
Copilot AI dynamic review requested due to automatic review settings November 7, 2026 23:56
@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

❌ PR Specification Reviewer failed to deliver outputs. Please review the logs for details.

Generated by PR Specification Reviewer for #75774

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · routed: opus55 medium · ~12 AIC · ⌖ ~2 AIC · ⊞ 7.7K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · routed: gpt56 max · ~1 AIC · ⌖ ~2 AIC · ⊞ 6.3K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Generated by Ponytail Reviewer for #75774

@@ -553,6 +557,9 @@ async function main() {
* @param {string} sourceLabel
*/
function evaluateAndReportVerdict(verdict, sourceLabel) {
const threatKinds = STRUCTURED_OUTPUT_REQUIRED_FIELDS.filter(field => verdict[field] === true).join(",");

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review complete: no actionable issues found in the changed lines. Static analysis was limited to the pre-fetched diff and targeted source reads; the advisory specialist pass was unavailable because its tool invocation was rejected.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 5fdbd49 against Safe Outputs MCP Gateway Specification v1.29.20, Section 10.5 WTD1 and template/publication behavior. The implementation and tests support the intentional category-only publication behavior, explicit unavailable-category messaging, and separation of tooling failures. Synchronized docs/src/content/docs/specs/safe-outputs-specification.md to v1.29.21 (2026-11-08), updating WTD1 wording, template placeholder semantics, and the changelog; the commit was queued for publication to the same-repository PR branch (output so_78D3oa_qSdya). No blocking conformance findings. Static review only; tests were not run by this reviewer.

Generated by PR Specification Reviewer for #75774 · copilot · gpt50 · ~5 AIC · ⌖ ~2 AIC · ⊞ 1.7K
Comment /review-specs to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What this change does: It carries the three schema-defined threat categories through inline and external detection jobs into public warnings and report blocks. It also replaces the review-PR and completion-comment warning paths with the typed report renderer.

Should fix:

  1. Keep one warning presentation path (actions/setup/js/push_to_pull_request_branch.cjs:L35)

    • What this is: The PR removes the last runtime call to getThreatWarningPresentation and uses getDetectionWarningBlocks instead.
    • Problem: The old helper is now referenced only by tests, so the repository keeps a second warning API that cannot affect production output.
    • Fix: Delete getThreatWarningPresentation, its export, and its tests, or keep a real caller if the object is still needed.
    • If we skip it: Future warning-copy changes must be checked against dead presentation code and can leave two descriptions out of sync.
  2. Remove the unused string failure renderer (actions/setup/js/messages_run_status.cjs:L111)

    • What this is: getDetectionFailureBlocks is the new runtime path, while getDetectionFailureMessage remains beside it.
    • Problem: A repository search leaves the string helper in the preview and tests only; both helpers select the same template and construct the same context.
    • Fix: Remove the string helper and its preview/tests, or derive the preview from the block renderer so there is one failure-message implementation.
    • If we skip it: The two failure-message paths can drift while one of them is never used by workflows.

Verdict: fix 1 and 2 first.
Lean: -40 lines possible.

Generated by ✂️ Ponytail Reviewer for #75774 · codex · routed: gpt56 max · ~8 AIC · ⌖ ~1 AIC · ⊞ 118
Comment /ponytail to run again

@@ -31,7 +31,10 @@ const gitHelpers = require("./git_helpers.cjs");
const { ensureFullHistoryForBundle, importLocalBundle, offlineGitOptions, linearizeRangeAsCommit, ensureSafeDirectoryTrust } = gitHelpers;
const { extractPatchBaseCommit } = require("./commit_sha_helpers.cjs");
const { findRepoCheckout } = require("./find_repo_checkout.cjs");
const { getThreatWarningPresentation, getEffectiveDetectionConclusion } = require("./threat_detection_warning.cjs");
const { getEffectiveDetectionConclusion } = require("./threat_detection_warning.cjs");
const { getDetectionWarningBlocks } = require("./messages_run_status.cjs");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lean: Removing the call here leaves getThreatWarningPresentation with no production callers; rg finds it only in tests. Delete the helper/export and those tests so the template-backed getDetectionWarningBlocks path is the single warning presentation and cannot drift.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed getThreatWarningPresentation, its export, and the remaining test/mocked usage. Template-backed warning rendering is now the sole presentation path. Commit 3e48d5b33b.

* @param {DetectionFailureContext} ctx
* @returns {import("./report_ast.cjs").ReportBlock[]}
*/
function getDetectionFailureBlocks(ctx) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lean: This new block renderer leaves getDetectionFailureMessage used only by the preview/tests, while both functions select the same template and build the same context. Remove the unused string API and its preview/tests (or derive the preview from this renderer) to avoid maintaining two failure-message paths.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the unused getDetectionFailureMessage string API and its preview. Failure message coverage now exercises the typed report-block renderer. Commit 3e48d5b33b.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The test relies on GH_AW_DETECTION_THREAT_KINDS leaking from an earlier case, causing it to fail when run independently.

1 open issue
What changed in this PR

This PR propagates schema-defined threat categories (prompt_injection, secret_leak, and malicious_patch) into generated warnings and reviews, using framework-authored descriptions rather than detector-generated explanations.

Changes:

  • Export threat categories and pass them to safe-output and conclusion jobs.
  • Include category descriptions in warnings, tracking comments, blocked-output messages, and request-changes reviews.
  • Update templates, documentation, regression assertions, and generated workflow wiring.

This pull request requires changes.

Outstanding review comments:

  • moderate, 8 votes — actions/setup/js/generate_footer.test.cjs:526: The assertion depends on GH_AW_DETECTION_THREAT_KINDS leaking from an earlier test and fails independently. Set the variable explicitly in this test.
  • nit, 3 votes — actions/setup/js/parse_threat_detection_results.cjs:560: The export assertions are inside a skipped suite, so neither warn nor strict mode exercises category extraction/export. Add an enabled regression using a real result-file fixture or injectable filesystem.
  • nit, 3 votes — actions/setup/js/parse_threat_detection_results.test.cjs:1007: The parameterized category-export regression and related assertions never execute. Move them into an active suite to cover the inline producer’s output and environment propagation.
  • nit, 1 vote — actions/setup/js/safe_output_handler_manager.cjs:1112: Optional: extend the blocked-output test to verify the framework-authored category text and exclusion of unknown tokens, rather than checking only errorCode.
  • nit, 2 votes — docs/src/content/docs/specs/safe-outputs-specification.md:5948: The documented detection-warning setting is unsupported by the schema and Go configuration, while the schema’s detection-failure description still lists old placeholders and default text. Align the documentation and configuration support.
File Description
pkg/​workflow/​threat_detection_job.go Exposes threat categories as a detection-job output.
pkg/​workflow/​notify_comment_conclusion_helpers.go Passes categories to conclusion handlers.
pkg/​workflow/​detection_success_test.go Checks category output and environment wiring.
pkg/​workflow/​compiler_safe_outputs_envvars.go Passes categories to safe-output handlers.
docs/​src/​content/​docs/​specs/​safe-outputs-specification.md Documents category disclosure and message placeholders.
actions/​setup/​md/​threat_warning_request_changes_review.md Includes framework-authored threat details.
actions/​setup/​md/​detection_runs_issue.md Explains category reporting in tracking comments.
actions/​setup/​js/​threat_detection_warning.test.cjs Tests category allowlisting and descriptions.
actions/​setup/​js/​safe_output_handler_manager.cjs Uses framework-authored blocked-output descriptions.
actions/​setup/​js/​parse_threat_detection_results.test.cjs Adds category-export assertions within a skipped suite.
actions/​setup/​js/​parse_threat_detection_results.cjs Exports boolean findings and clears unavailable categories.
actions/​setup/​js/​messages_core.cjs Documents detection-message placeholders.
actions/​setup/​js/​issue_body_report_builder.test.cjs Checks trusted warnings remain separate from agent prose.
actions/​setup/​js/​handle_detection_runs.test.cjs Tests allowlisted categories in tracking comments.
actions/​setup/​js/​handle_detection_runs.cjs Includes category descriptions in tracked findings.
actions/​setup/​js/​generate_footer.test.cjs Adds expired-warning assertions with an environment-state dependency.
actions/​setup/​js/​create_pull_request.cjs Supplies safe details to request-changes reviews.
.github/​workflows/​workflow-skill-extractor.lock.yml Generated category wiring.
.github/​workflows/​workflow-normalizer.lock.yml Generated category wiring.
.github/​workflows/​workflow-health-manager.lock.yml Generated category wiring.
.github/​workflows/​workflow-generator.lock.yml Generated category wiring.
.github/​workflows/​windows-grower.lock.yml Generated category wiring.
.github/​workflows/​weekly-otel-reference.lock.yml Generated category wiring.
.github/​workflows/​weekly-issue-summary.lock.yml Generated category wiring.
.github/​workflows/​weekly-editors-health-check.lock.yml Generated category wiring.
.github/​workflows/​weekly-blog-post-writer.lock.yml Generated category wiring.
.github/​workflows/​visual-regression-checker.lock.yml Generated category wiring.
.github/​workflows/​video-analyzer.lock.yml Generated category wiring.
.github/​workflows/​update-astro.lock.yml Generated category wiring.
.github/​workflows/​ubuntu-image-analyzer.lock.yml Generated category wiring.
.github/​workflows/​typist.lock.yml Generated category wiring.
.github/​workflows/​terminal-stylist.lock.yml Generated category wiring.
.github/​workflows/​technical-doc-writer.lock.yml Generated category wiring.
.github/​workflows/​super-linter.lock.yml Generated category wiring.
.github/​workflows/​step-name-alignment.lock.yml Generated category wiring.
.github/​workflows/​static-analysis-report.lock.yml Generated category wiring.
.github/​workflows/​stale-pr-cleanup.lock.yml Generated category wiring.
.github/​workflows/​squad-implement-worker.lock.yml Generated category wiring.
.github/​workflows/​squad-game-planner.lock.yml Generated category wiring.
.github/​workflows/​spec-librarian.lock.yml Generated category wiring.
.github/​workflows/​spec-extractor.lock.yml Generated category wiring.
.github/​workflows/​spec-enforcer.lock.yml Generated category wiring.
.github/​workflows/​smoke-workflow-call.lock.yml Generated category wiring.
.github/​workflows/​smoke-security-alerts.lock.yml Generated category wiring.
.github/​workflows/​smoke-pi-routed.lock.yml Generated category wiring.
.github/​workflows/​smoke-issues.lock.yml Generated category wiring.
.github/​workflows/​smoke-factory.lock.yml Generated category wiring.
.github/​workflows/​smoke-atomic.lock.yml Generated category wiring.
.github/​workflows/​slide-deck-maintainer.lock.yml Generated category wiring.
.github/​workflows/​sighthound-security-scan.lock.yml Generated category wiring.
.github/​workflows/​sergo.lock.yml Generated category wiring.
.github/​workflows/​semantic-function-refactor.lock.yml Generated category wiring.
.github/​workflows/​security-compliance.lock.yml Generated category wiring.
.github/​workflows/​schema-feature-coverage.lock.yml Generated category wiring.
.github/​workflows/​schema-consistency-checker.lock.yml Generated category wiring.
.github/​workflows/​safe-output-health.lock.yml Generated category wiring.
.github/​workflows/​research.lock.yml Generated category wiring.
.github/​workflows/​repository-quality-improver.lock.yml Generated category wiring.
.github/​workflows/​repo-tree-map.lock.yml Generated category wiring.
.github/​workflows/​refiner.lock.yml Generated category wiring.
.github/​workflows/​refactoring-cadence.lock.yml Generated category wiring.
.github/​workflows/​python-data-charts.lock.yml Generated category wiring.
.github/​workflows/​purelock.lock.yml Generated category wiring.
.github/​workflows/​prompt-clustering-analysis.lock.yml Generated category wiring.
.github/​workflows/​pr-finisher.lock.yml Generated category wiring.
.github/​workflows/​pr-description-caveman.lock.yml Generated category wiring.
.github/​workflows/​portfolio-analyst.lock.yml Generated category wiring.
.github/​workflows/​org-health-report.lock.yml Generated category wiring.
.github/​workflows/​objective-impact-report.lock.yml Generated category wiring.
.github/​workflows/​metrics-collector.lock.yml Generated category wiring.
.github/​workflows/​lockfile-stats.lock.yml Generated category wiring.
.github/​workflows/​linter-miner.lock.yml Generated category wiring.
.github/​workflows/​lint-monster.lock.yml Generated category wiring.
.github/​workflows/​layout-spec-maintainer.lock.yml Generated category wiring.
.github/​workflows/​jsweep.lock.yml Generated category wiring.
.github/​workflows/​issue-triage-agent.lock.yml Generated category wiring.
.github/​workflows/​issue-monster.lock.yml Generated category wiring.
.github/​workflows/​issue-arborist.lock.yml Generated category wiring.
.github/​workflows/​instructions-janitor.lock.yml Generated category wiring.
.github/​workflows/​hourly-ci-cleaner.lock.yml Generated category wiring.
.github/​workflows/​gpclean.lock.yml Generated category wiring.
.github/​workflows/​go-pattern-detector.lock.yml Generated category wiring.
.github/​workflows/​go-logger.lock.yml Generated category wiring.
.github/​workflows/​go-fan.lock.yml Generated category wiring.
.github/​workflows/​glossary-maintainer.lock.yml Generated category wiring.
.github/​workflows/​github-mcp-tools-report.lock.yml Generated category wiring.
.github/​workflows/​front-page-copy-guard.lock.yml Generated category wiring.
.github/​workflows/​firewall-escape.lock.yml Generated category wiring.
.github/​workflows/​feature-grower.lock.yml Generated category wiring.
.github/​workflows/​example-workflow-analyzer.lock.yml Generated category wiring.
.github/​workflows/​example-failure-category-filter.lock.yml Generated category wiring.
.github/​workflows/​evoskill-evolver.lock.yml Generated category wiring.
.github/​workflows/​eslint-refiner.lock.yml Generated category wiring.
.github/​workflows/​eslint-monster.lock.yml Generated category wiring.
.github/​workflows/​eslint-miner.lock.yml Generated category wiring.
.github/​workflows/​eslint-factory-dispatcher.lock.yml Generated category wiring.
.github/​workflows/​enterprise-docs-audit.lock.yml Exposes categories to safe outputs.
.github/​workflows/​duplicate-code-detector.lock.yml Generated category wiring.
.github/​workflows/​draft-pr-cleanup.lock.yml Generated category wiring.
.github/​workflows/​docs-noob-tester.lock.yml Generated category wiring.
.github/​workflows/​dictation-prompt.lock.yml Generated category wiring.
.github/​workflows/​dev-hawk.lock.yml Generated category wiring.
.github/​workflows/​detection-analysis-report.lock.yml Generated category wiring.
.github/​workflows/​designer-drift-audit.lock.yml Generated category wiring.
.github/​workflows/​deployment-incident-monitor.lock.yml Generated category wiring.
.github/​workflows/​delight.lock.yml Generated category wiring.
.github/​workflows/​deep-report.lock.yml Generated category wiring.
.github/​workflows/​dead-code-remover.lock.yml Generated category wiring.
.github/​workflows/​daily-workflow-updater.lock.yml Generated category wiring.
.github/​workflows/​daily-workflow-security-model.lock.yml Generated category wiring.
.github/​workflows/​daily-work-queues-report.lock.yml Generated category wiring.
.github/​workflows/​daily-work-queue-formal-verification.lock.yml Exposes categories to safe outputs.
.github/​workflows/​daily-token-consumption-report.lock.yml Generated category wiring.
.github/​workflows/​daily-team-status.lock.yml Generated category wiring.
.github/​workflows/​daily-team-evolution-insights.lock.yml Generated category wiring.
.github/​workflows/​daily-syntax-error-quality.lock.yml Generated category wiring.
.github/​workflows/​daily-storify.lock.yml Generated category wiring.
.github/​workflows/​daily-spending-forecast.lock.yml Generated category wiring.
.github/​workflows/​daily-security-red-team.lock.yml Generated category wiring.
.github/​workflows/​daily-security-observability.lock.yml Generated category wiring.
.github/​workflows/​daily-secrets-analysis.lock.yml Generated category wiring.
.github/​workflows/​daily-safe-output-optimizer.lock.yml Generated category wiring.
.github/​workflows/​daily-safe-output-integrator.lock.yml Generated category wiring.
.github/​workflows/​daily-report-dispatcher.lock.yml Generated category wiring.
.github/​workflows/​daily-reliability-review.lock.yml Generated category wiring.
.github/​workflows/​daily-regulatory.lock.yml Generated category wiring.
.github/​workflows/​daily-pr-review-cursor.lock.yml Generated category wiring.
.github/​workflows/​daily-performance-summary.lock.yml Generated category wiring.
.github/​workflows/​daily-news.lock.yml Generated category wiring.
.github/​workflows/​daily-model-resolution.lock.yml Generated category wiring.
.github/​workflows/​daily-model-inventory.lock.yml Generated category wiring.
.github/​workflows/​daily-issues-report.lock.yml Generated category wiring.
.github/​workflows/​daily-graft-intelligence.lock.yml Generated category wiring.
.github/​workflows/​daily-grader-audit.lock.yml Generated category wiring.
.github/​workflows/​daily-geo-optimizer.lock.yml Generated category wiring.
.github/​workflows/​daily-function-namer.lock.yml Generated category wiring.
.github/​workflows/​daily-firewall-report.lock.yml Generated category wiring.
.github/​workflows/​daily-file-diet.lock.yml Generated category wiring.
.github/​workflows/​daily-fact.lock.yml Generated category wiring.
.github/​workflows/​daily-experiment-report.lock.yml Generated category wiring.
.github/​workflows/​daily-evals-report.lock.yml Generated category wiring.
.github/​workflows/​daily-documentation-diagram.lock.yml Generated category wiring.
.github/​workflows/​daily-doc-updater.lock.yml Generated category wiring.
.github/​workflows/​daily-doc-healer.lock.yml Generated category wiring.
.github/​workflows/​daily-credit-limit-test.lock.yml Generated category wiring.
.github/​workflows/​daily-compiler-quality.lock.yml Generated category wiring.
.github/​workflows/​daily-community-attribution.lock.yml Generated category wiring.
.github/​workflows/​daily-code-metrics.lock.yml Generated category wiring.
.github/​workflows/​daily-code-debt-aider.lock.yml Generated category wiring.
.github/​workflows/​daily-cli-tools-tester.lock.yml Generated category wiring.
.github/​workflows/​daily-cli-performance.lock.yml Generated category wiring.
.github/​workflows/​copilot-session-insights.lock.yml Generated category wiring.
.github/​workflows/​copilot-opt.lock.yml Generated category wiring.
.github/​workflows/​copilot-agent-analysis.lock.yml Generated category wiring.
.github/​workflows/​contribution-check.lock.yml Generated category wiring.
.github/​workflows/​constraint-solving-potd.lock.yml Generated category wiring.
.github/​workflows/​code-simplifier.lock.yml Generated category wiring.
.github/​workflows/​code-scanning-fixer.lock.yml Generated category wiring.
.github/​workflows/​cli-version-checker.lock.yml Generated category wiring.
.github/​workflows/​cli-consistency-checker.lock.yml Generated category wiring.
.github/​workflows/​ci-coach.lock.yml Generated category wiring.
.github/​workflows/​changeset.lock.yml Generated category wiring.
.github/​workflows/​breaking-change-checker.lock.yml Generated category wiring.
.github/​workflows/​blog-auditor.lock.yml Generated category wiring.
.github/​workflows/​avenger.lock.yml Generated category wiring.
.github/​workflows/​auditor.lock.yml Generated category wiring.
.github/​workflows/​audit-workflows.lock.yml Generated category wiring.
.github/​workflows/​artifacts-summary.lock.yml Generated category wiring.
.github/​workflows/​architecture-guardian.lock.yml Generated category wiring.
.github/​workflows/​api-consumption-report.lock.yml Generated category wiring.
.github/​workflows/​agent-persona-explorer.lock.yml Generated category wiring.
.github/​workflows/​agent-performance-analyzer.lock.yml Generated category wiring.
.github/​workflows/​agent-job-health.lock.yml Generated category wiring.
.github/​workflows/​ab-testing-advisor.lock.yml Generated category wiring.

Review effort: Max (auto)

@@ -522,6 +523,8 @@ describe("generate_footer.cjs", () => {
expect(result).toContain("agentic threat detected");
expect(result).toContain("<!-- gh-aw-threat-detected -->");
expect(result).toContain("Potential security threats were detected");
expect(result).toContain("Detected threat types: secret leak.");

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /tdd and /grill-with-docs. Requesting changes on one test-isolation defect and on docs that describe a placeholder users cannot reach.

📋 Key Themes & Highlights

Key Themes

  • Test isolation: generate_footer.test.cjs sets GH_AW_DETECTION_THREAT_KINDS without clearing it, so the value leaks into later tests.
  • Documentation accuracy: the spec and messages_core.cjs document {reason_text} and {threat_kinds} for detection-warning, but Go only maps detection-failure (pkg/workflow/safe_outputs_messages_config.go:40), and the schema lists placeholders only for detection-failure.
  • Schema drift: pkg/parser/schemas/main_workflow_schema.json (detection-failure, around line 13434) still lists only {workflow_name}, {run_url}. frontmatter-full.md is generated from it, so it is stale too.

Positive Highlights

  • ✅ Threat-kind handling is allowlisted (THREAT_KIND_LABELS), and private detector text never reaches public output. Tests cover this with a PRIVATE_TEXT non-leakage check.
  • ✅ conclude_threat_detection.sh validates all three booleans with jq -er and fails closed to an empty category list on invalid input.
  • ✅ notify_comment_error.cjs assigns message in every branch, so the block-based refactor cannot crash on an unset value.
  • ✅ getDetectionReasonText only adds category text for threat_detected, so tooling failures never show threat types.

Uncertainty

  • Lock-file consistency could not be verified: the local worktree has about 324 uncommitted .lock.yml edits, which do not match HEAD. I did not treat them as a finding.
  • The usage-artifact collector in pkg/workflow/notify_comment.go (~line 428) omits GH_AW_DETECTION_THREAT_KINDS. This may be intentional, but no test documents that.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~8 AIC · ⌖ ~2 AIC · ⊞ 5.4K
Comment /matt to run again

@@ -488,6 +488,7 @@ describe("generate_footer.cjs", () => {
it("should NOT include caution alert (caution is now injected by callers at the top)", () => {
process.env.GH_AW_DETECTION_CONCLUSION = "warning";
process.env.GH_AW_DETECTION_REASON = "threat_detected";
process.env.GH_AW_DETECTION_THREAT_KINDS = "secret_leak PRIVATE_TEXT";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] Test isolation: GH_AW_DETECTION_THREAT_KINDS is set here but never cleared. The beforeEach at lines 55-56 only deletes GH_AW_DETECTION_CONCLUSION and GH_AW_DETECTION_REASON, so the value persists into later tests and can change footer text in an order-dependent way.

Add delete process.env.GH_AW_DETECTION_THREAT_KINDS; alongside the other deletes.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added GH_AW_DETECTION_THREAT_KINDS to per-test cleanup and set it explicitly in the category assertion test. Commit 3e48d5b33b.

@@ -5943,6 +5943,9 @@ When threat detection executes in `warn` mode and reports a threat signal for a

2. A visible threat label string: `agentic threat detected`.
3. An XML comment marker in emitted markdown content: `<!-- gh-aw-threat-detected -->`.
4. Detector-defined threat types from the boolean verdict fields: `prompt_injection`, `secret_leak`, and `malicious_patch`. Public warnings, failure notifications, tracking comments, and review messages MUST describe only the fields set to `true`, using framework-authored labels. They MUST NOT publish detector-generated explanations or agent text as threat details. If the types are unavailable, the message MUST say they were not reported rather than infer a finding. Tooling failures MUST remain distinct from threat findings.

Configured `detection-failure` and `detection-warning` templates can use `{reason_text}` for a framework-authored description and `{threat_kinds}` for the comma-separated category identifiers (`unknown` when unavailable).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/grill-with-docs] This sentence says detection-warning is a configurable template, but no user-facing key reads it. pkg/workflow/safe_outputs_messages_config.go:40 maps only detection-failure, and the schema defines only detection-failure. So {reason_text} and {threat_kinds} are unreachable for warnings in practice.

Either drop detection-warning from this sentence, or wire the key through the Go config and the schema.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the specification to document only detection-failure as configurable; warn-mode warning templates are framework-provided. Commit 3e48d5b33b.

Comment thread actions/setup/js/messages_core.cjs Outdated
* @property {string} [pullRequestCreated] - Custom template for pull request creation link. Placeholders: {item_number}, {item_url}
* @property {string} [issueCreated] - Custom template for issue creation link. Placeholders: {item_number}, {item_url}
* @property {string} [commitPushed] - Custom template for commit push link. Placeholders: {commit_sha}, {short_sha}, {commit_url}
* @property {string} [detectionWarning] - Custom detection warning message template (continue-on-error mode). Placeholders: {workflow_name}, {run_url}, {reason_text}
* @property {string} [detectionWarning] - Custom detection warning message template (continue-on-error mode). Placeholders: {workflow_name}, {run_url}, {reason_text}, {threat_kinds}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/grill-with-docs] This JSDoc advertises {reason_text} and {threat_kinds} for detectionWarning, but nothing maps a user-facing key to it (see the spec comment above). Keep the placeholder list in line with what users can actually configure, or add the mapping.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the unsupported detectionWarning JSDoc configuration field and placeholder claim; only detectionFailure is user-configurable. Commit 3e48d5b33b.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot

Copilot assignment requested for the following outstanding work:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Review (actions/setup/js/push_to_pull_request_branch.cjs:35): Lean: Removing the call here leaves getThreatWarningPresentation with no production callers; rg finds it only in tests. Delete the helper/export and those tests so the template-backed getDetectionWarningBlocks path is the single warning presentation and cannot drift. - Show detector-defined threat categories in generated warnings #75774 (comment)
  3. Review (actions/setup/js/messages_run_status.cjs:111): Lean: This new block renderer leaves getDetectionFailureMessage used only by the preview/tests, while both functions select the same template and build the same context. Remove the unused string API and its preview/tests (or derive the preview from this renderer) to avoid maintaining two failure-message paths. - Show detector-defined threat categories in generated warnings #75774 (comment)
  4. Review (actions/setup/js/generate_footer.test.cjs:491): [/tdd] Test isolation: GH_AW_DETECTION_THREAT_KINDS is set here but never cleared. The beforeEach at lines 55-56 only deletes GH_AW_DETECTION_CONCLUSION and GH_AW_DETECTION_REASON, so the value persists into later tests and can change footer text in an order-dependent way. - Show detector-defined threat categories in generated warnings #75774 (comment)
  5. Review (docs/src/content/docs/specs/safe-outputs-specification.md:5948): [/grill-with-docs] This sentence says detection-warning is a configurable template, but no user-facing key reads it. pkg/workflow/safe_outputs_messages_config.go:40 maps only detection-failure, and the schema defines only detection-failure. So {reason_text} and {threat_kinds} are unreachable for warnings in practice. - Show detector-defined threat categories in generated warnings #75774 (comment)
  6. Review (actions/setup/js/messages_core.cjs:46): [/grill-with-docs] This JSDoc advertises {reason_text} and {threat_kinds} for detectionWarning, but nothing maps a user-facing key to it (see the spec comment above). Keep the placeholder list in line with what users can actually configure, or add the mapping. - Show detector-defined threat categories in generated warnings #75774 (comment)
  7. Fix failing check build-wasm (FAILURE): https://github.com/github/gh-aw/actions/runs/42427518955/job/128568981313.
  8. Fix failing check build-wasm (FAILURE): https://github.com/github/gh-aw/actions/runs/42427294825/job/128568278198.
  9. Fix failing check impacted-go-tests (FAILURE): https://github.com/github/gh-aw/actions/runs/42427518955/job/128568981654.
  10. Fix failing check safe_outputs (FAILURE): https://github.com/github/gh-aw/actions/runs/42427519156/job/128570602801.

Once you start work, post this work summary as a PR comment without the leading Copilot mention, including the Sous-chef head, work, and state lines below, to record that the assignment is running. Apply the necessary fixes on this PR's existing branch, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved; stop when only human review or CI remains. Do not merge or trigger CI.

Sous-chef requested-head: 5fdbd49
Sous-chef requested-work: 05e6507b7eaee4f43ea0f7e902943036dc88cd0bb6039d835412f44b501361c3 42e5d572d9d75b7168fe7d0aeda0717cc07e65ed240f6d299e6fec661214f1ac 55904aa639e9b427a3424d8303c9de7e06183ce08f738cb1c58256d6c16e03e9 5c0e183041dd4382dcdbe6d30a1608c2ea6ee1c6a9f3c454f9e98557ddd73368 8acb61a077717e3a460018aacdfde96fd489f77606cd36169e2328d9079b75bb 8c4fd5b4dd6218bf6bf97890bdde1e43010fecb0e6315873b9881d46fec831a7 c7cf93e497befdd5e698d00101b0703e4101fe4133d86d433f7c170d07402b7e fc62d1d5ccab0decfc35e9185286be76a01c3dc1b9c9115643d9ef026af8620c
Sous-chef requested-state: e446550b7f4bc026233530d211274e1ef13dbea8a5ba7a2aa6df0a7b81989689

Generated by 👨🍳 PR Sous Chef · pi · haiku45 · ~13 AIC · ⌖ ~19 AIC · ⊞ 1.2K · ◷
Comment /souschef to run again

…eat-category-messages

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
elsisiem added a commit to elsisiem/receipts that referenced this pull request Nov 8, 2026
…mports, polished README with logo

- receipts.py: tasks, streamed events per step, paste mode, recheck, low-thinking Gemini
- server.py: POST /api/run streams each sponsor step; rate-limited for the public site
- index.html: four-step pipeline with live clocks, claim highlights in the message and the log, gallery lies-first
- tools/import_pr.py: judge a real agent-written PR against its CI logs
- runs_evidence: 6 live Gemini runs + 2 real GitHub Copilot PRs (github/gh-aw#75774: contradicted)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI requested a review from gh-aw-bot November 8, 2026 01:13
@pelikhan
pelikhan merged commit 81e3373 into main Nov 8, 2026
111 checks passed
@pelikhan
pelikhan deleted the copilot/detection-threat-category-messages branch November 8, 2026 01:29
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.2.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants