Skip to content

[safeoutputs] Clarify submit_pull_request_review allowed-events in tool description - #76610

Merged
pelikhan merged 2 commits into
mainfrom
safeoutputs/review-allowed-events-description-c24f9fc5bc0ffe60
Nov 11, 2026
Merged

pelikhan merged 2 commits into
mainfrom
safeoutputs/review-allowed-events-description-c24f9fc5bc0ffe60

Conversation

@github-actions

Copy link
Copy Markdown
Contributor

Summary

When a workflow sets safe-outputs.submit-pull-request-review.allowed-events, the compiler already narrows the event enum, but the tool's top-level description still describes APPROVE as usable and never lists the permitted events. This PR appends an explicit constraint to the description, for example: Allowed review events: COMMENT, REQUEST_CHANGES. Events not listed (APPROVE) are rejected; do not use them.

Evidence (last 24h, 219 sessions analyzed from the pre-downloaded bundle)

Only 3 safe-output tool-call validation errors appeared in the session logs. The workflow prompt for the affected run is correct, so this is a tool-description issue.

  • submit_pull_request_review: event: "APPROVE" rejected. Workflow PR Code Quality Reviewer, run §42834684561. The workflow sets allowed-events: [COMMENT, REQUEST_CHANGES] and its prompt does not mention APPROVE. The agent ran --help, which only prints a truncated description and option names (no enum), then called with "event":"APPROVE" and got event must be one of: COMMENT, REQUEST_CHANGES. It retried with COMMENT. The base description mentions APPROVE as a normal option and does not list the allowed events, so nothing visible to the agent said it was excluded. This is a recurring pattern in the cache (previously 4 occurrences).
  • Not addressed here (implementation or CLI issues, out of scope):
    • create_discussion called through safeoutputs ... --title ... --body . returned "missing or empty 'title'" in Daily Security Observability Report, run §42849033654. This looks like a CLI per-field stdin handling problem rather than a description problem.
    • create_pull_request_review_comment was called with unknown params output_id and result in Matt Pocock Skills Reviewer, run §42880625191. It was a single occurrence and the error message already lists the supported parameters.

Change

  • pkg/workflow/tool_description_enhancer.go: new appendAllowedReviewEventsConstraint, used by submitPullRequestReviewConstraints. It lists the allowed events and names the excluded ones. Nothing is added when allowed-events is unset.
  • pkg/workflow/tool_description_enhancer_test.go: table test TestEnhanceToolDescriptionSubmitPullRequestReviewAllowedEvents (APPROVE excluded, only COMMENT, all allowed, unconfigured).

Validation

  • gofmt -l on both files: clean.
  • Not run: go test ./pkg/workflow, make fmt, make recompile. The sandbox cannot download Go modules (proxy.golang.org TLS failure), so the package does not build here. I did compile and run the new function in isolation (copied verbatim into a scratch module) and its output matches the strings the new test asserts:
    • [COMMENT, REQUEST_CHANGES] → Allowed review events: COMMENT, REQUEST_CHANGES. Events not listed (APPROVE) are rejected; do not use them.
    • [COMMENT] → ... Events not listed (APPROVE, REQUEST_CHANGES) are rejected; do not use them.
    • all three → no "rejected" text
    • unset → no constraint
  • A maintainer should run go test ./pkg/workflow -run TestEnhanceToolDescription and make recompile, then commit the regenerated .lock.yml files. Workflows using allowed-events (for example pr-code-quality-reviewer) will change.

🤖 Generated with [Claude Code]((claude.com/redacted)

Generated by ⚡ Daily Safe Output Tool Optimizer · claude · routed: sonnet55 high · ~63 AIC · ⌖ ~2 AIC · ⊞ 10K · ◷

  • expires on Nov 12, 2026, 1:10 PM UTC-08:00

@github-actions github-actions Bot added automated-analysis bug Something isn't working cookie Issue Monster Loves Cookies! safe-outputs tool-improvement labels Nov 10, 2026
@pelikhan
pelikhan marked this pull request as ready for review November 10, 2026 22:04
Copilot AI dynamic review requested due to automatic review settings November 10, 2026 22:04
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 8.5K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 7.2K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

Matt Pocock Skills Reviewer

Outcome: reported_action

The agent completed and reported one or more actions. The no-op output is a completion signal, not a claim that no action was taken.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Matt Pocock Skills Reviewer · routed: haiku55 xhigh · ~5 AIC · ⌖ ~2 AIC · ⊞ 6.6K · ◷

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · routed: gpt61 medium · ~5 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 181 · ◷

Generated by Ponytail Reviewer for #76610

@github-actions

Copy link
Copy Markdown
Contributor Author

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)
reviewed_at: 2026-11-10T22:08:40Z
review_event: REQUEST_CHANGES
top_themes: generated lock artifacts missing; duplicated review-event constants
files_reviewed: pkg/workflow/tool_description_enhancer.go, pkg/workflow/tool_description_enhancer_test.go
comment_count: 2

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K · ◷
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

Request changes: the source change is not accompanied by regenerated lock workflows, and the new event list duplicates canonical supported values.

Findings
  • Regenerate and commit the affected .lock.yml artifacts with make recompile.
  • Share one canonical review-event set across validation, schema generation, and description enhancement to prevent policy drift.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K
Comment /review to run again

appendMaxConstraint(constraints, config.Max, "Maximum %d review(s) can be submitted.")
appendTargetConstraint(constraints, config.Target)
appendStringConstraint(constraints, config.TargetRepoSlug, "Reviews will be submitted in repository %q.")
appendAllowedReviewEventsConstraint(constraints, config.AllowedEvents)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changes the compiled tool description without updating the checked-in generated lock workflows, so the repository can fail its recompile/drift check and affected workflows will continue shipping the old description.

💡 Regenerate the compiled artifacts

Run make recompile and commit the resulting .lock.yml changes alongside this source change; verify the generated diff is intentional.

}

// reviewEventValues lists every review event the submit_pull_request_review tool schema supports.
var reviewEventValues = []string{"APPROVE", "COMMENT", "REQUEST_CHANGES"}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reviewEventValues duplicates the supported-event set maintained elsewhere, so adding or renaming a review event can make this guidance claim a valid event is rejected (or omit a newly valid event).

💡 Share the canonical event set

Define one canonical supported-events list and reuse it for parsing/validation, schema generation, and this description constraint. That keeps the agent-facing policy synchronized with runtime enforcement as the enum evolves.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A moderate active finding remains because supported-event allowlists are duplicated, so future event additions could make generated descriptions incorrectly reject valid events.

1 open issue
What changed in this PR

This pull request improves agent guidance for submit_pull_request_review by exposing workflow-specific allowed-events constraints in generated tool descriptions.

Changes:

  • Adds guidance listing allowed and rejected review events when configured.
  • Integrates the constraint into submit-review tool descriptions.
  • Adds table-driven tests covering restricted, unrestricted, and unconfigured event lists.
File Description
pkg/​workflow/​tool_description_enhancer.go Generates explicit allowed and rejected review-event guidance for configured submit-review tools.
pkg/​workflow/​tool_description_enhancer_test.go Tests description output across configured, unrestricted, and unconfigured review-event policies.

Review effort: Lite (auto)

Comment on lines +394 to +395
// reviewEventValues lists every review event the submit_pull_request_review tool schema supports.
var reviewEventValues = []string{"APPROVE", "COMMENT", "REQUEST_CHANGES"}

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs and /codebase-design. Requesting changes: the new rule may be invisible on the CLI path where the failure occurred, and committed lock files will go stale.

📋 Key Themes & Highlights

Key Themes

  • Visibility: The new sentence is appended after the long preamble. --help cuts descriptions at 90 characters (TOOL_DESC_MAX_LEN in actions/setup/js/mcp_cli_bridge.cjs), so the rule is likely never shown to the agent on that path.
  • Stale lock files: Recompiling pr-code-quality-reviewer changed its embedded submit_pull_request_review text. ponytail-reviewer and pr-specification-reviewer embed the same text, so the check-workflow-drift job should fail until make recompile is run and the locks are committed.
  • Duplicated event list: The event values are repeated in submit_pr_review.go, safe_outputs_validation_config.go, and main_workflow_schema.json.

Validation Performed

  • go test -run TestEnhanceToolDescription ./pkg/workflow passes, including the new table test. This covers the "not run" gap in the PR description.
  • gofmt -l is clean on both files.
  • Building cmd/gh-aw and running compile pr-code-quality-reviewer reproduced the lock change. The lock file was restored afterward, and the tree is clean.

Positive Highlights

  • ✅ The helper is short, handles the unset case, and has a table test covering the excluded-event cases.
  • ✅ The schema enum and the description now describe the same narrowed event set.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~5 AIC · ⌖ ~2 AIC · ⊞ 6.6K
Comment /matt to run again

}

// reviewEventValues lists every review event the submit_pull_request_review tool schema supports.
var reviewEventValues = []string{"APPROVE", "COMMENT", "REQUEST_CHANGES"}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/codebase-design] The review event list is now hardcoded in several places. submit_pr_review.go (validEvents) and safe_outputs_validation_config.go (line 394) repeat the same values, and main_workflow_schema.json (line 9493) does too. A future event addition could leave this description out of sync.

💡 Suggested refactor

Define one package-level slice of review events and use it in the parser in submit_pr_review.go and in reviewEventValues. Keep the same order so the rendered text stays stable.

@copilot please address this.

appendMaxConstraint(constraints, config.Max, "Maximum %d review(s) can be submitted.")
appendTargetConstraint(constraints, config.Target)
appendStringConstraint(constraints, config.TargetRepoSlug, "Reviews will be submitted in repository %q.")
appendAllowedReviewEventsConstraint(constraints, config.AllowedEvents)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/codebase-design] This changes the compiled tool description, so committed lock files go stale. Recompiling pr-code-quality-reviewer changed its embedded submit_pull_request_review text, and ponytail-reviewer and pr-specification-reviewer embed the same text. The check-workflow-drift CI job should fail until the locks are regenerated.

💡 Suggested fix

Run make recompile in this PR and commit the regenerated .lock.yml files. Do not leave this to a maintainer, since the drift gate fails CI.

@copilot please address this.

}
}
if len(excluded) > 0 {
text += " Events not listed (" + strings.Join(excluded, ", ") + ") are rejected; do not use them."

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] The new sentence is appended after the long WRITE-ONCE preamble, so it falls outside the visible part of the description. safeoutputs submit_pull_request_review --help cuts the description at 90 characters (TOOL_DESC_MAX_LEN in actions/setup/js/mcp_cli_bridge.cjs, line 78). That is the CLI path where the reported call failed, so this fix may not reach it.

💡 Suggested fix

Put the allowed-events sentence before the preamble so its first 90 characters carry the rule. Add a regression test that asserts the rule appears within the first 90 characters. Then confirm the output of safeoutputs submit_pull_request_review --help after recompiling.

@copilot please address this.

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot run pr-finisher skill

@pelikhan
pelikhan merged commit f33f7dc into main Nov 11, 2026
47 of 48 checks passed
@pelikhan
pelikhan deleted the safeoutputs/review-allowed-events-description-c24f9fc5bc0ffe60 branch November 11, 2026 03:11
Copilot AI requested a review from pelikhan November 11, 2026 03:11
Copilot stopped work on behalf of pelikhan due to an error November 11, 2026 03:11
github-actions Bot added a commit that referenced this pull request Nov 11, 2026
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Peli de Halleux <pelikhan@users.noreply.github.com>
pelikhan added a commit that referenced this pull request Nov 11, 2026
* Apply filtered changes (#76610)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Peli de Halleux <pelikhan@users.noreply.github.com>

* eslint-factory: add require-null-check-with-typeof-object rule

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix typeof-object null guard analysis

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

* Apply remaining changes

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Peli de Halleux <pelikhan@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor Author

🎉 This pull request is included in a new release.

Release: v1.2.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automated-analysis bug Something isn't working cookie Issue Monster Loves Cookies! safe-outputs tool-improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants