Skip to content

Align CLI reference with command defaults and side effects - #76575

Merged
pelikhan merged 5 commits into
mainfrom
copilot/cli-consistency-fix-sides-effects
Nov 10, 2026
Merged

pelikhan merged 5 commits into
mainfrom
copilot/cli-consistency-fix-sides-effects

Conversation

Copilot AI commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

The CLI reference omitted fix --write file changes, left trial repository retention unclear, and blurred local and remote list behavior. Its introductory logs example also implied detailed artifacts were downloaded by default.

  • Document command behavior: Describe the files fix --write may update or delete, clarify that trial host repositories are retained unless --delete-host-repo-after is specified, and distinguish local and remote list modes, including the local-only --stale filter.
  • Clarify artifact selection: State that logs defaults to compact usage data and show how to request agent or firewall data:
    gh aw logs workflow --artifacts agent
    gh aw logs workflow --artifacts firewall
  • Guard against drift: Add focused help/reference consistency assertions for the four command sections.

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix CLI consistency issues with command side effects Align CLI reference with command defaults and side effects Nov 10, 2026
Copilot AI requested a review from gh-aw-bot November 10, 2026 18:48
@pelikhan
pelikhan marked this pull request as ready for review November 10, 2026 18:57
Copilot AI dynamic review requested due to automatic review settings November 10, 2026 18:57
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

Caution

agentic threat detected
Threat detection flagged this output in warn mode. Manual review is REQUIRED before any follow-up automation.

Details

Potential security threats were detected in the agent output. Detected threat types: prompt injection.

Review the workflow run logs for details.

PR Code Quality Reviewer

Outcome: reported_action

The agent completed and reported one or more actions. The no-op output is a completion signal, not a claim that no action was taken.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from PR Code Quality Reviewer · gpt56 · ~5 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 1.6K · ◷

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Specification Reviewer completed successfully!

PR Specification Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from PR Specification Reviewer · gpt56 · ~2 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 1.6K · ◷

Generated by PR Specification Reviewer for #76575

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 7.2K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · routed: gpt61 medium · ~4 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 181 · ◷

Generated by Ponytail Reviewer for #76575

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 8.5K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The documentation understates the files that fix --write can delete, omitting the .github/aw/ path and broader .github/agents/ scope.

1 open issue
What changed in this PR

This PR aligns the CLI reference with current command defaults and side effects while adding consistency checks to prevent documentation drift.

Changes:

  • Documents fix --write file updates and cleanup operations, trial repository retention, and local versus remote list behavior.
  • Clarifies compact default logs artifacts and adds agent/firewall artifact examples.
  • Adds focused help/reference consistency assertions for the four command sections.
File Description
pkg/​cli/​cli_consistency_help_test.go Adds assertions that CLI help and reference documentation remain aligned for the fix, trial, list, and logs sections.
docs/​src/​content/​docs/​setup/​cli.md Updates CLI reference documentation for command side effects, repository retention, listing modes, and artifact selection.

Review effort: Lite (auto)

Comment thread docs/src/content/docs/setup/cli.md Outdated
Comment on lines +375 to +376
workflow-specific `.agent.md` files from `.github/agents/`. It also refreshes
`.github/skills/agentic-workflows/SKILL.md` and `.github/agents/agentic-workflows.md`
@github-actions

Copy link
Copy Markdown
Contributor

Caution

agentic threat detected
Threat detection flagged this output in warn mode. Manual review is REQUIRED before any follow-up automation.

Details

Potential security threats were detected in the agent output. Detected threat types: prompt injection.

Review the workflow run logs for details.

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)
reviewed_at: 2026-11-10T19:01:01Z
review_event: REQUEST_CHANGES
top_themes: incomplete side-effect documentation; destructive trial cleanup semantics; environment-dependent defaults; fragile help assertions
files_reviewed: docs/src/content/docs/setup/cli.md, pkg/cli/cli_consistency_help_test.go, pkg/cli/fix_command.go, pkg/cli/trial_command.go, pkg/cli/list_workflows_command.go, pkg/cli/logs_command.go, pkg/cli/copilot_agents.go
comment_count: 4

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~5 AIC · ⌖ ~2 AIC · ⊞ 1.6K · ◷
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

agentic threat detected
Threat detection flagged this output in warn mode. Manual review is REQUIRED before any follow-up automation.

Details

Potential security threats were detected in the agent output. Detected threat types: prompt injection.

Review the workflow run logs for details.

Found four actionable issues in the documentation and consistency test:

  • fix --write documentation omits legacy cleanup paths and its skip condition.
  • --delete-host-repo-after can delete reused --host-repo repositories, but the docs imply only newly created hosts are affected.
  • The documented list path default ignores GH_AW_WORKFLOWS_DIR.
  • New flag assertions can panic on a missing flag instead of reporting a test failure.

Review was static; no tests were run per review instructions.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~5 AIC · ⌖ ~2 AIC · ⊞ 1.6K
Comment /review to run again

Comment thread pkg/cli/cli_consistency_help_test.go Outdated
trialDocs := section("trial", "run")
trialHelp := NewTrialCommand(func(string) error { return nil })
assert.Contains(t, trialHelp.Long, "retained by default")
assert.Contains(t, trialHelp.Flags().Lookup("delete-host-repo-after").Usage, "retained by default")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flags().Lookup(...) can return nil, so dereferencing .Usage makes a renamed or missing flag panic instead of producing a useful assertion failure. Please assert the lookup is non-nil before reading Usage (the same issue occurs for stale and path below).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added nil checks before reading flag usage and asserted the trial, list, and logs flag defaults. Commit: 3a9f9ef7.

Comment thread docs/src/content/docs/setup/cli.md Outdated
`list` enumerates workflows locally without GitHub API queries. For enabled/disabled state and run information, use `status` instead.
Without `--repo`, `list` enumerates workflows from the local checkout without GitHub API
queries. With `--repo`, it lists workflows from that remote repository; `--path` selects
the remote workflows directory (default: `.github/workflows`). The `--stale` filter is

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The --path default is constants.GetWorkflowDir(), which honors GH_AW_WORKFLOWS_DIR; .github/workflows is only the fallback. Please mention $GH_AW_WORKFLOWS_DIR here, otherwise this reference is inaccurate for configured repositories.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the remote --path default to document $GH_AW_WORKFLOWS_DIR, falling back to .github/workflows, and pinned the default in the consistency test. Commit: 3a9f9ef7.

Comment thread docs/src/content/docs/setup/cli.md Outdated

`--timeout` sets the execution timeout in integer minutes (default: `30`; `0` disables the timeout). For example, use `--timeout 10`, not `--timeout 10m`.

By default, a newly created private host repository is retained after the trial. Specify

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wording understates the destructive behavior of --delete-host-repo-after: the cleanup path is selected solely from the flag, so it also deletes a pre-existing repository supplied via --host-repo, not only a newly created trial repository. Please document that explicitly or change the cleanup logic to protect reused repositories (with a regression test).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified that --delete-host-repo-after deletes the host repository even when an existing repository was supplied via --host-repo. Commit: 3a9f9ef7.


**Options:** `--dir/-d`, `--disable-codemod`, `--list-codemods`, `--write`

Without `--write`, no files are modified. With `--write`, `fix` writes the updated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documented cleanup is incomplete and can mislead users about --write. The implementation also removes several legacy files under .github/agents/ and the singular .github/aw/upgrade-agentic-workflow.md, and these deletions are skipped when instruction-file updates are disabled. Please document the actual scope (or link to the command help), including the --skip-instruction-file-updates interaction.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expanded the help and reference to cover legacy .agent.md and .md files under .github/agents/ plus .github/aw/upgrade-agentic-workflow.md, and noted that cleanup/refresh are skipped when instruction-file updates are disabled. gh aw fix does not expose a --skip-instruction-file-updates flag, so the docs describe the behavior without advertising a nonexistent CLI flag. Commit: 3a9f9ef7.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /grill-with-docs to the docs and /tdd to the new consistency test. Requesting changes on one incorrect example.

📋 Key Themes & Highlights

Key Themes

  • Incorrect artifact example: --artifacts replaces its default instead of adding to it, so logs --artifacts agent drops the usage artifact. The new docs line "Include detailed agent data" says otherwise.
  • Guard checks prose, not behaviour: the new test asserts substrings only. It does not pin flag defaults, which the PR title and test name promise.

Verified

  • ✅ TestCLIDocsDocumentCommandSideEffectsAndDefaults passes on the PR head.
  • ✅ Red check: reverting the docs hunks makes the new test fail.
  • ✅ fix --write deletion targets, the trial retention default, list --stale (local only) and list --path match the code.
  • ✅ logs default of usage matches the flag default.

Notes

  • The pr-triage agent was not invoked. Classification was done directly from the diff.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~4 AIC · ⌖ ~2 AIC · ⊞ 6.6K
Comment /matt to run again

Comment thread docs/src/content/docs/setup/cli.md Outdated

```bash wrap
gh aw logs workflow # Download logs for workflow
gh aw logs workflow --artifacts agent # Include detailed agent data

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/grill-with-docs] --artifacts replaces its default rather than adding to it, so --artifacts agent downloads only agent data and drops the usage artifact. "Include detailed agent data" suggests the opposite. The command's own help already says "Download only agent logs" for this form.

💡 Suggested wording
gh aw logs workflow --artifacts agent          # Download only detailed agent data
gh aw logs workflow --artifacts usage,agent    # Keep usage data and add agent data

This matches the flag's replace-the-default behavior and the existing help text.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified that --artifacts replaces the default set: the agent and firewall examples download only those sets, and usage,agent preserves compact usage alongside agent data. Commit: 3a9f9ef7.

assert.Equal(t, 2, strings.Count(text, "`--repos`, `--interactive/-i`, `--dry-run`"))
}

func TestCLIDocsDocumentCommandSideEffectsAndDefaults(t *testing.T) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] The test name promises "Defaults", but nothing asserts a flag default. The substring checks pin prose, so they still pass when the docs are wrong. The --artifacts agent example in cli.md is one case this test misses.

💡 Suggested assertions
assert.Equal(t, "[usage]", NewLogsCommand().Flags().Lookup("artifacts").DefValue)
assert.Equal(t, "false", NewTrialCommand(func(string) error { return nil }).Flags().Lookup("delete-host-repo-after").DefValue)
assert.Equal(t, ".github/workflows", NewListCommand().Flags().Lookup("path").DefValue)

Checking DefValue pins the defaults the docs describe, so they cannot drift silently.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added explicit default assertions for artifacts, trial cleanup, and list path, with nil checks before using flag metadata. Commit: 3a9f9ef7.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot

🍳 A little prep to help this PR along! Copilot assignment requested for the following outstanding work:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Review (pkg/cli/cli_consistency_help_test.go:162): address unresolved feedback - Review comment
  3. Review (docs/src/content/docs/setup/cli.md:658): address unresolved feedback - Review comment
  4. Review (docs/src/content/docs/setup/cli.md:566): address unresolved feedback - Review comment
  5. Review (docs/src/content/docs/setup/cli.md:372): address unresolved feedback - Review comment
  6. Review (docs/src/content/docs/setup/cli.md:697): address unresolved feedback - Review comment
  7. Review (pkg/cli/cli_consistency_help_test.go:123): address unresolved feedback - Review comment

Keeping the changes focused; human review and CI get the final taste test. No merging or CI triggering.

Structured data
{
  "completed_task": 0,
  "head": "61e59fdeb0520689fb93100e05af0dc5b86c1dbf",
  "kind": "requested",
  "schema_version": "1",
  "state": "8079f229407c4dc1b94be90f953eb609789a8bf74a8cc7f55882364d2afae20c",
  "work": [
    "098765d6bf5fc594b3de513c852ae9cdabf9f84a08770ab2a12fcfd2c505d7e0",
    "21b73c503b6f269265cc9a0246d0ba5e76327382e43e1d5f6f3f1de41297f318",
    "5a32863c94d237c6cf6034e8e5a4f71c6c501f524cb85112132f0c7428ff2151",
    "8e6618d7e23e66ffeaea5c2cb9de8d790d01d04b7af32577b207558ae7e5ba4e",
    "9dd36a2c47469e2a3cfcaf70990283249b3921109c0440d69632a86bfb85398a",
    "c5db844f3700ef7443e236d5cb6861e205f6163935c08b8d9322476c35963b9a"
  ],
  "workflow": "pr-sous-chef"
}

Generated by 👨🍳 PR Sous Chef · pi · routed: haiku55 xhigh · ~2 AIC · ⌖ ~2 AIC · ⊞ 6.9K · ◷
Comment /souschef to run again

Copilot AI and others added 3 commits November 10, 2026 19:43
…cy-fix-sides-effects

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
@pelikhan
pelikhan merged commit eed4c37 into main Nov 10, 2026
49 checks passed
@pelikhan
pelikhan deleted the copilot/cli-consistency-fix-sides-effects branch November 10, 2026 21:46
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.2.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[cli-consistency] CLI Consistency Issues - 2026-11-10

4 participants