Repository navigation
[safeoutputs] Clarify submit_pull_request_review allowed-events in tool description - #76610
Conversation
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft. Design Decision Gate 🏗️Outcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ Test Quality Sentinel completed test quality analysis. Test Quality SentinelOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅ Matt Pocock Skills ReviewerOutcome: The agent completed and reported one or more actions. The no-op output is a completion signal, not a claim that no action was taken. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ PR Code Quality Reviewer completed the code quality review.
|
|
✅ Ponytail Reviewer completed successfully! Ponytail ReviewerOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
Comment MemoryPeek at saved memory (pr-code-quality-reviewer)Note This comment is managed by comment memory.Expand the saved memory block to view or edit the persistent context for this thread.
|
There was a problem hiding this comment.
Review
Request changes: the source change is not accompanied by regenerated lock workflows, and the new event list duplicates canonical supported values.
Findings
- Regenerate and commit the affected
.lock.ymlartifacts withmake recompile. - Share one canonical review-event set across validation, schema generation, and description enhancement to prevent policy drift.
🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K
Comment /review to run again
| appendMaxConstraint(constraints, config.Max, "Maximum %d review(s) can be submitted.") | ||
| appendTargetConstraint(constraints, config.Target) | ||
| appendStringConstraint(constraints, config.TargetRepoSlug, "Reviews will be submitted in repository %q.") | ||
| appendAllowedReviewEventsConstraint(constraints, config.AllowedEvents) |
There was a problem hiding this comment.
This changes the compiled tool description without updating the checked-in generated lock workflows, so the repository can fail its recompile/drift check and affected workflows will continue shipping the old description.
💡 Regenerate the compiled artifacts
Run make recompile and commit the resulting .lock.yml changes alongside this source change; verify the generated diff is intentional.
| } | ||
|
|
||
| // reviewEventValues lists every review event the submit_pull_request_review tool schema supports. | ||
| var reviewEventValues = []string{"APPROVE", "COMMENT", "REQUEST_CHANGES"} |
There was a problem hiding this comment.
reviewEventValues duplicates the supported-event set maintained elsewhere, so adding or renaming a review event can make this guidance claim a valid event is rejected (or omit a newly valid event).
💡 Share the canonical event set
Define one canonical supported-events list and reuse it for parsing/validation, schema generation, and this description constraint. That keeps the agent-facing policy synchronized with runtime enforcement as the enum evolves.
There was a problem hiding this comment.
🟡 Changes recommended
A moderate active finding remains because supported-event allowlists are duplicated, so future event additions could make generated descriptions incorrectly reject valid events.
1 open issue
What changed in this PR
This pull request improves agent guidance for submit_pull_request_review by exposing workflow-specific allowed-events constraints in generated tool descriptions.
Changes:
- Adds guidance listing allowed and rejected review events when configured.
- Integrates the constraint into submit-review tool descriptions.
- Adds table-driven tests covering restricted, unrestricted, and unconfigured event lists.
| File | Description |
|---|---|
| pkg/workflow/tool_description_enhancer.go | Generates explicit allowed and rejected review-event guidance for configured submit-review tools. |
| pkg/workflow/tool_description_enhancer_test.go | Tests description output across configured, unrestricted, and unconfigured review-event policies. |
Review effort: Lite (auto)
| // reviewEventValues lists every review event the submit_pull_request_review tool schema supports. | ||
| var reviewEventValues = []string{"APPROVE", "COMMENT", "REQUEST_CHANGES"} |
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /diagnosing-bugs and /codebase-design. Requesting changes: the new rule may be invisible on the CLI path where the failure occurred, and committed lock files will go stale.
📋 Key Themes & Highlights
Key Themes
- Visibility: The new sentence is appended after the long preamble.
--helpcuts descriptions at 90 characters (TOOL_DESC_MAX_LENinactions/setup/js/mcp_cli_bridge.cjs), so the rule is likely never shown to the agent on that path. - Stale lock files: Recompiling
pr-code-quality-reviewerchanged its embeddedsubmit_pull_request_reviewtext.ponytail-reviewerandpr-specification-reviewerembed the same text, so thecheck-workflow-driftjob should fail untilmake recompileis run and the locks are committed. - Duplicated event list: The event values are repeated in
submit_pr_review.go,safe_outputs_validation_config.go, andmain_workflow_schema.json.
Validation Performed
go test -run TestEnhanceToolDescription ./pkg/workflowpasses, including the new table test. This covers the "not run" gap in the PR description.gofmt -lis clean on both files.- Building
cmd/gh-awand runningcompile pr-code-quality-reviewerreproduced the lock change. The lock file was restored afterward, and the tree is clean.
Positive Highlights
- ✅ The helper is short, handles the unset case, and has a table test covering the excluded-event cases.
- ✅ The schema enum and the description now describe the same narrowed event set.
@copilot please address the review comments above.
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~5 AIC · ⌖ ~2 AIC · ⊞ 6.6K
Comment /matt to run again
| } | ||
|
|
||
| // reviewEventValues lists every review event the submit_pull_request_review tool schema supports. | ||
| var reviewEventValues = []string{"APPROVE", "COMMENT", "REQUEST_CHANGES"} |
There was a problem hiding this comment.
[/codebase-design] The review event list is now hardcoded in several places. submit_pr_review.go (validEvents) and safe_outputs_validation_config.go (line 394) repeat the same values, and main_workflow_schema.json (line 9493) does too. A future event addition could leave this description out of sync.
💡 Suggested refactor
Define one package-level slice of review events and use it in the parser in submit_pr_review.go and in reviewEventValues. Keep the same order so the rendered text stays stable.
@copilot please address this.
| appendMaxConstraint(constraints, config.Max, "Maximum %d review(s) can be submitted.") | ||
| appendTargetConstraint(constraints, config.Target) | ||
| appendStringConstraint(constraints, config.TargetRepoSlug, "Reviews will be submitted in repository %q.") | ||
| appendAllowedReviewEventsConstraint(constraints, config.AllowedEvents) |
There was a problem hiding this comment.
[/codebase-design] This changes the compiled tool description, so committed lock files go stale. Recompiling pr-code-quality-reviewer changed its embedded submit_pull_request_review text, and ponytail-reviewer and pr-specification-reviewer embed the same text. The check-workflow-drift CI job should fail until the locks are regenerated.
💡 Suggested fix
Run make recompile in this PR and commit the regenerated .lock.yml files. Do not leave this to a maintainer, since the drift gate fails CI.
@copilot please address this.
| } | ||
| } | ||
| if len(excluded) > 0 { | ||
| text += " Events not listed (" + strings.Join(excluded, ", ") + ") are rejected; do not use them." |
There was a problem hiding this comment.
[/diagnosing-bugs] The new sentence is appended after the long WRITE-ONCE preamble, so it falls outside the visible part of the description. safeoutputs submit_pull_request_review --help cuts the description at 90 characters (TOOL_DESC_MAX_LEN in actions/setup/js/mcp_cli_bridge.cjs, line 78). That is the CLI path where the reported call failed, so this fix may not reach it.
💡 Suggested fix
Put the allowed-events sentence before the preamble so its first 90 characters carry the rule. Add a regression test that asserts the rule appears within the first 90 characters. Then confirm the output of safeoutputs submit_pull_request_review --help after recompiling.
@copilot please address this.
…n-c24f9fc5bc0ffe60
|
@copilot run pr-finisher skill |
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Peli de Halleux <pelikhan@users.noreply.github.com>
* Apply filtered changes (#76610) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Peli de Halleux <pelikhan@users.noreply.github.com> * eslint-factory: add require-null-check-with-typeof-object rule Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix typeof-object null guard analysis Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Apply remaining changes Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Peli de Halleux <pelikhan@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
🎉 This pull request is included in a new release. Release: |

Summary
When a workflow sets
safe-outputs.submit-pull-request-review.allowed-events, the compiler already narrows theeventenum, but the tool's top-level description still describesAPPROVEas usable and never lists the permitted events. This PR appends an explicit constraint to the description, for example:Allowed review events: COMMENT, REQUEST_CHANGES. Events not listed (APPROVE) are rejected; do not use them.Evidence (last 24h, 219 sessions analyzed from the pre-downloaded bundle)
Only 3 safe-output tool-call validation errors appeared in the session logs. The workflow prompt for the affected run is correct, so this is a tool-description issue.
submit_pull_request_review:event: "APPROVE"rejected. WorkflowPR Code Quality Reviewer, run §42834684561. The workflow setsallowed-events: [COMMENT, REQUEST_CHANGES]and its prompt does not mention APPROVE. The agent ran--help, which only prints a truncated description and option names (no enum), then called with"event":"APPROVE"and gotevent must be one of: COMMENT, REQUEST_CHANGES. It retried withCOMMENT. The base description mentionsAPPROVEas a normal option and does not list the allowed events, so nothing visible to the agent said it was excluded. This is a recurring pattern in the cache (previously 4 occurrences).create_discussioncalled throughsafeoutputs ... --title ... --body .returned "missing or empty 'title'" inDaily Security Observability Report, run §42849033654. This looks like a CLI per-field stdin handling problem rather than a description problem.create_pull_request_review_commentwas called with unknown paramsoutput_idandresultinMatt Pocock Skills Reviewer, run §42880625191. It was a single occurrence and the error message already lists the supported parameters.Change
pkg/workflow/tool_description_enhancer.go: newappendAllowedReviewEventsConstraint, used bysubmitPullRequestReviewConstraints. It lists the allowed events and names the excluded ones. Nothing is added whenallowed-eventsis unset.pkg/workflow/tool_description_enhancer_test.go: table testTestEnhanceToolDescriptionSubmitPullRequestReviewAllowedEvents(APPROVE excluded, only COMMENT, all allowed, unconfigured).Validation
gofmt -lon both files: clean.go test ./pkg/workflow,make fmt,make recompile. The sandbox cannot download Go modules (proxy.golang.orgTLS failure), so the package does not build here. I did compile and run the new function in isolation (copied verbatim into a scratch module) and its output matches the strings the new test asserts:[COMMENT, REQUEST_CHANGES]→Allowed review events: COMMENT, REQUEST_CHANGES. Events not listed (APPROVE) are rejected; do not use them.[COMMENT]→... Events not listed (APPROVE, REQUEST_CHANGES) are rejected; do not use them.go test ./pkg/workflow -run TestEnhanceToolDescriptionandmake recompile, then commit the regenerated.lock.ymlfiles. Workflows usingallowed-events(for examplepr-code-quality-reviewer) will change.🤖 Generated with [Claude Code]((claude.com/redacted)