Skip to content

Lockdown check fails on private cross-repo workflow_call: Contents API uses caller's GITHUB_TOKEN, not callee-accessible token #32312

Closed

Description

@norrietaylor

Version: gh-aw v0.72.1

Symptom

Every consumer workflow_call dispatch against a private gh-aw callee fails at the lock.yml's activation step with:

Cross-repo invocation detected: workflow source is "<callee>",
  current repo is "<caller>"
GET /repos/<callee>/contents/.github/workflows/<chore>.lock.yml
    ?ref=<callee-ref-sha> — 404 Not Found
Could not fetch content for .github/workflows/<chore>.lock.yml: Not Found
Unable to fetch lock file content for hash comparison via API,
  trying local filesystem fallback
Local lock file not found: …/work/<caller>/<caller>/.github/workflows/<chore>.lock.yml
##[warning]Could not compare frontmatter hashes — assuming lock file is outdated
##[error]ERR_CONFIG: Lock file '.github/workflows/<chore>.lock.yml' is outdated
  or unverifiable! Could not verify frontmatter hash for
  '.github/workflows/<chore>.md'. Run 'gh aw compile' to regenerate the lock file.

The check_workflow_timestamp_api.cjs activation step then sets stale_lock_file_failed=true, and the conclusion job marks the whole run failed before the agent ever runs.

Concrete reproducer: gominimal/min-ctl run 25906428806 calls gominimal/min-aw/.github/workflows/worker-fix.lock.yml@v0.6.3. Both repos private, same org. Fails as above on every dispatch.

Root cause

check_workflow_timestamp_api.cjs issues the Contents API call against the callee repo using GITHUB_TOKEN from the caller's runner. For a private callee:

  • GITHUB_TOKEN on a workflow run is repo-scoped to the running repo (the caller, e.g. gominimal/min-ctl).
  • The Contents API on a private repo requires read access to that specific repo.
  • gominimal/min-ctl's GITHUB_TOKEN has no scope on gominimal/min-aw → 404, even though both repos are in the same org and workflow_call itself was already authorized via repo-actions-access settings.

This works fine when the callee is public (elastic/ai-github-actions is public; their consumers don't see this) — public Contents API doesn't require auth. The gap is private-callee-private-caller.

Why the existing fallbacks don't help

  1. API path uses GITHUB_TOKEN → 404 for private callees.
  2. GH_AW_GITHUB_TOKEN fallback secret slot is declared optional in the lock.yml's workflow_call.secrets. Consumers can provide a token here, but the documentation/templates don't make this clear, and there's no practical way to mint a short-lived cross-repo token without long-lived PAT storage (security cost).
  3. Local filesystem fallback looks for the lock.yml on the caller's checkout. Consumers using the wrapper distribution pattern don't ship .lock.yml files locally (that's the point — thin wrappers).

Proposed fix

The lock.yml already receives APP_PRIVATE_KEY via workflow_call.secrets (used for safe-output writes via actions/create-github-app-token). If the corresponding GitHub App is installed on the callee repo as well, an App-minted installation token has cross-repo Contents read access.

check_workflow_timestamp_api.cjs should:

  1. Detect cross-repo invocation (it already does — emits "Cross-repo invocation detected" log).
  2. Detect that the initial Contents API call returned 404 / 401 / 403.
  3. Mint a fresh installation token using APP_PRIVATE_KEY + vars.APP_ID (both already available to the activation step), scoped to the callee repo.
  4. Retry the Contents API call with the minted token.
  5. If the mint fails (no APP_PRIVATE_KEY in secrets, App not installed on callee, etc.), fall through to today's behavior with a clearer error message: "Configure GH_AW_GITHUB_TOKEN with cross-repo Contents read access, or install the gh-aw App on the callee repo."

Alternative if internal mint is too invasive: emit a clearer error directing consumers to set up GH_AW_GITHUB_TOKEN. The current message ("outdated or unverifiable") sends users toward gh aw compile, which doesn't address the auth gap.

Workaround for current users

Ship .lock.yml mirrors on the consumer side alongside .md mirrors. The filesystem fallback then succeeds. Trade-off: ~80–90 KB per chore on the consumer (defeats the thin-wrapper goal). See gominimal/min-ctl#66 for the workaround in practice.

Related

Activity

  1. kkruel8100 commented on May 16, 2026

    @kkruel8100

    Experiencing same issue. Proposed workaround is not viable on central ops repo and large scale quantity of target repos. Please increase priority level and mark bug status.

  2. locked and limited conversation to collaborators on May 16, 2026
  3. unlocked this conversation on May 16, 2026
  4. pelikhan commented on May 16, 2026

    @pelikhan
    Collaborator

    /scout do a deep research the mentioned workflows and repos and provide a summary with snippets of the current situation

  5. github-actions commented on May 16, 2026

    @github-actions
    Contributor

    🏕️ Lost in the wilderness! Scout failed to deliver outputs. Sending search party...

  6. github-actions commented on May 16, 2026

    @github-actions
    Contributor

    🔍 Scout Research Report

    Triggered by @pelikhan via /scout comment

    Executive Summary

    The bug in #32312 is real, reproducible from the code in this repo, and the proposed fix path (mint an App-installation token in the activation step when cross-repo + 404/401/403) is sound. The root cause confirmed in code: the compiled Check workflow lock file step in every *.lock.yml runs actions/github-script without specifying a github-token, so it inherits the caller's repo-scoped GITHUB_TOKEN — which has no read access to the private callee. Prior fixes (#23935, #24422) taught the script which file to look for cross-repo but did not change which token it uses to fetch it. The same reporter (@norrietaylor) filed a separate cross-repo race in #32310, suggesting the wrapper-distribution pattern is exercising a class of gaps.

    Click to expand detailed findings

    Research Findings

    1. Root cause confirmed in scout.lock.yml (and every compiled lock.yml)

    The activation step that calls check_workflow_timestamp_api.cjs is generated without a github-token: override, so github inside github-script is authenticated with the default caller GITHUB_TOKEN:

    # .github/workflows/scout.lock.yml:223-234
    - name: Check workflow lock file
      id: check-lock-file
      uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
      env:
        GH_AW_WORKFLOW_FILE: "scout.lock.yml"
        GH_AW_CONTEXT_WORKFLOW_REF: "${{ github.workflow_ref }}"
      with:
        script: |
          const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs');
          setupGlobals(core, github, context, exec, io, getOctokit);
          const { main } = require('${{ runner.temp }}/gh-aw/actions/check_workflow_timestamp_api.cjs');
          await main();

    Compare to safe-output steps in the same file that do use the fallback chain (scout.lock.yml:1347+):

    github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }}

    The activation step is missing this. getFileContent (actions/setup/js/github_api_helpers.cjs:69-101) then calls github.rest.repos.getContent on the callee owner/repo with the caller's token, producing the 404 the issue describes.

    2. Cross-repo detection already exists — only the token slot is missing

    check_workflow_timestamp_api.cjs already does the right resolution (actions/setup/js/check_workflow_timestamp_api.cjs:86-160): it calls actions.getWorkflowRun to read referenced_workflows[], picks sha over ref, and logs Cross-repo invocation detected: workflow source is "<callee>", current repo is "<caller>" — that's the exact line in the issue's error block. So the script already knows it's cross-repo. It just doesn't switch tokens when it does.

    The inline comments name this gap explicitly:

    // Fallback: compare frontmatter hashes using local filesystem files.
    // Used when the GitHub API is inaccessible (e.g., cross-org reusable workflow where
    // the caller's GITHUB_TOKEN cannot read the source repo).
    

    That fallback is the local checkout, which doesn't exist on wrapper consumers — the issue is correct that filesystem fallback is structurally useless for the thin-wrapper distribution pattern.

    3. Prior fixes addressed the file, not the auth

    Both fixes ran on public callees where Contents API needs no auth. The private-callee case was never reached by those fixes because the test fixtures didn't exercise it.

    4. Related bug in same setup: #32310 (closed)

    Same reporter, same gominimal/min-aw ↔ gominimal/min-ctl repos. Bundle-apply race in safe_outputs because checkout uses fetch-depth: 1 and the bundle's base SHA falls out of reach when main advances between the agent and safe-outputs jobs. Different root cause, but it confirms the wrapper-distribution + private-callee pattern is being actively used and surfacing edge cases the existing test coverage misses.

    5. The lock.yml secrets contract already names the right slot

    Every compiled workflow_call: block already declares the optional slot (e.g. dependabot-worker.lock.yml):

    workflow_call:
      secrets:
        GH_AW_GITHUB_TOKEN:
          required: false
        GH_AW_GITHUB_MCP_SERVER_TOKEN:
          required: false

    So the consumer-side mechanism to pass a cross-repo-capable token already exists. The activation step just doesn't read it. Adding github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} to the compiled Check workflow lock file step (in compiler_activation_job_builder.go / compiler_activation_job.go) would let consumers who provision the secret pass it through without any new mechanism.

    6. App-token mint path is feasible and idiomatic

    The reporter's preferred fix (use APP_PRIVATE_KEY + vars.APP_ID already in scope to mint a fresh installation token) is the GitHub-recommended pattern for cross-repo private reads — see the [Cloud Chronicles guide]((cloudchronicles.blog/redacted) and GitHub's own authentication docs: "If you need a token that requires permissions that aren't available in the GITHUB_TOKEN, create a GitHub App and generate an installation access token within your workflow." actions/create-github-app-token accepts an explicit owner: parameter so the minted token can be scoped to the callee org/repo. The same action is already used elsewhere in safe-output jobs in this codebase, so introducing it into the activation job is consistent.

    Key constraint to verify before implementing: the App must be installed on the callee repo too, not just the caller. The fallback error message should make this explicit.

    7. The error message points users in the wrong direction

    From check_workflow_timestamp_api.cjs:343:

    Lock file '...' is outdated or unverifiable! Could not verify frontmatter hash for '...'.
    Run 'gh aw compile' to regenerate the lock file.
    

    When the actual failure is an auth 404 against the callee, gh aw compile does nothing useful — the lock file is fine, the script just can't read it. Even without changing the auth behavior, branching the message on the cross-repo + API-failure case would save users the wild-goose chase.

    Recommendations

    In priority order:

    1. Minimal fix (low risk, ships value immediately): In compiler_activation_job_builder.go, emit github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} on the Check workflow lock file step. Consumers who already provision GH_AW_GITHUB_TOKEN (PAT or App-minted) get the fix for free; no new contract.
    2. Sharper error message (independent of fix rejig docs #1): When workflowRepo !== currentRepo AND the API call returns 401/403/404 AND the local filesystem fallback finds no lock file, fail with: "Cross-repo lock check failed: the caller's GITHUB_TOKEN cannot read <callee>. Provision GH_AW_GITHUB_TOKEN (PAT or App installation token with Contents:read on the callee), or install the gh-aw GitHub App on the callee repo." Suppress the "Run gh aw compile" suggestion in this branch — it's misleading.
    3. Deeper fix (App-mint, what the issue proposes): Detect cross-repo + Contents-API-failure, then if APP_PRIVATE_KEY + APP_ID are present, mint an installation token scoped to the callee owner and retry. Falls back to message in (2) when the mint fails. This removes the consumer's burden of provisioning a separate PAT but adds App-install-on-callee as a prerequisite — document that clearly.
    4. Test coverage: Add a cross-repo + private-callee integration test fixture. The existing pkg/workflow/call_workflow_secrets_test.go validates the secrets list shape but doesn't exercise the activation-step auth path. A test that builds a fake referenced_workflows API response and asserts the step uses the fallback token (not GITHUB_TOKEN) would have caught this and would catch regressions in (1)/(3).
    5. Docs: Update docs/src/content/docs/reference/cross-repository.md with a "Private callee" section. Today it covers "reading from other private repositories" for MCP tooling but not the lock-file activation path.

    The minimum to unblock @norrietaylor and @kkruel8100 right now is #1 + #2. Those are a few-line change in the compiler and a conditional in the script; both are independent of the App-mint design and can ship in a patch release without breaking any existing consumer.

    Key Sources

    Suggested Next Steps

    1. Open a small PR adding github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} to the activation step generator and regenerating affected lock files (recommendation 1).
    2. In the same PR or a follow-up, branch the warning message on cross-repo + API-failure (recommendation 2).
    3. Track the App-mint path (recommendation 3) as a separate issue/PR — it has a larger surface (new dependency on the App being installed on callees, secret naming convention) and shouldn't block the quick wins.
    4. Add a compiler test ensuring the activation step's actions/github-script invocation carries the fallback github-token (recommendation 4).
    5. Note for the gominimal consumers: until the fix lands, the documented workaround in gominimal/min-ctl#66 (ship .lock.yml mirrors) works, but provisioning GH_AW_GITHUB_TOKEN as a workflow_call secret with Contents:read on the callee repo will work today without any gh-aw change — the secret slot is already declared in every compiled lock.yml.

    🔭 Intelligence gathered by Scout · ● 11.3M · ◷

  7. pelikhan commented on May 16, 2026

    @pelikhan
    Collaborator

    @copilot follow scout plan, ignore app mint for now.

  8. norrietaylor commented on May 16, 2026

    @norrietaylor
    Author

    Thanks Peli!

    I will try this out once it lands and report back.

  9. locked and limited conversation to collaborators on May 16, 2026
  10. unlocked this conversation on May 16, 2026
  11. norrietaylor commented on May 16, 2026

    @norrietaylor
    Author

    #32530 implements Scout's recommendation #2 (legible error) only. Its diff touches check_workflow_timestamp_api.cjs + test + changeset — no compiler change.

    Recommendation #1 — emit github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} on the compiled Check workflow lock file step (compiler_activation_job_builder.go) — is still open. That is the change that lets the activation step actually read a private callee.

    Without #1, #32530's new message is not actionable: it instructs operators to "Configure GH_AW_GITHUB_TOKEN", but the activation step has no github-token: slot, so a provisioned GH_AW_GITHUB_TOKEN is never consumed. The compiled workflow_call block already declares the optional secret (GH_AW_GITHUB_TOKEN: required: false), so #1 is a few-line generator change with no new contract.

    Request: keep this issue open after #32530 merges, or track #1 as a follow-up PR. #32530 makes the failure legible but does not unblock the private-to-private workflow_call path.

  12. locked and limited conversation to collaborators on May 16, 2026
  13. unlocked this conversation on May 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions