Skip to content

bug: Copilot CLI 1.0.21 added a startup model validation step: when COPILOT_MODEL is set #25593

Closed

Description

@jaroslawgajewski

Root Cause

Copilot CLI 1.0.21 added a startup model validation step: when COPILOT_MODEL
is set, the CLI calls GET /models to validate the requested model name before
execution. The /models endpoint does not accept classic PATs (ghp_*).

Workflows that do not set COPILOT_MODEL (i.e., use the default model)
are unaffected — they never trigger the validation call.

Affected Scope

  • Trigger: COPILOT_MODEL is set in the agent environment.
  • Token type: Classic PAT (ghp_*) used as COPILOT_GITHUB_TOKEN.
  • Unaffected: Workflows without COPILOT_MODEL; OAuth or fine-grained tokens.

Reproduction

  1. Use gh-aw v0.67.3 with COPILOT_GITHUB_TOKEN set to a classic PAT.
  2. Set COPILOT_MODEL=gpt-5.4 (or any model) in the agent environment.
  3. Trigger any workflow. Observe exit code 1 on the "Execute GitHub Copilot CLI" step.

Expected Behaviour

Model validation should either:

  • Accept classic PATs (parity with previous CLI versions), OR
  • Fail gracefully with a clear diagnostic pointing to the token type issue.

Prior Versions

gh-aw v0.67.2 (Copilot CLI 1.0.20) did not perform model validation on
startup. This regression was introduced silently in v0.67.3.

Environment

  • gh-aw CLI: v0.67.3
  • AWF: v0.25.16
  • Copilot CLI: 1.0.21
  • GitHub MCP Server: v0.32.0
  • MCP Gateway: v0.2.16
  • COPILOT_GITHUB_TOKEN type: classic PAT

Activity

  1. jaroslawgajewski commented on Apr 10, 2026

    @jaroslawgajewski
    Author

    possibly will address also #25494

  2. locked and limited conversation to collaborators on Apr 10, 2026
  3. unlocked this conversation on Apr 10, 2026
  4. locked and limited conversation to collaborators on Apr 10, 2026
  5. unlocked this conversation on Apr 10, 2026
  6. raye-deng commented on Apr 10, 2026

    @raye-deng

    This regression description is excellent, especially the way you narrowed it to the COPILOT_MODEL path plus classic PATs.

    We've run into the same class of failure in AI-assisted pipelines: everything looks healthy until one extra validation call hits a newer endpoint or auth assumption, and then the workflow dies before any useful work starts. The frustrating part is that traditional code-quality checks usually stay green because the break is in the runtime contract between tool, token type, and API behavior.

    What helped us was adding a PR-stage compatibility check for exactly these "API drift" edges, things like auth-mode assumptions, model-name validation, and package/version availability, alongside normal static analysis. It ends up complementing SonarQube well because Sonar catches code smells, but not these AI-tooling contract mismatches.

    If it's useful to compare approaches, we've been testing that with OCR here:

    npx @opencodereview/cli scan . --sla L1

    It won't fix this specific token regression by itself, but it has been useful for catching the same family of silent AI-pipeline breakages earlier.

  7. locked and limited conversation to collaborators on Apr 10, 2026
  8. unlocked this conversation on Apr 10, 2026
  9. lpcox commented on Apr 11, 2026

    @lpcox
    Collaborator
  10. lpcox commented on Apr 12, 2026

    @lpcox
    Collaborator
  11. lpcox commented on Apr 16, 2026

    @lpcox
    Collaborator
  12. lpcox commented on Apr 16, 2026

    @lpcox
    Collaborator

    Addressed in gh-aw-firewall by PRs #2006 (explicit BYOK model) and #2018 (config ingestion with COPILOT_MODEL fallback). Closing tracking issue github/gh-aw-firewall#2010.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions