Repository navigation
Description
Version: gh-aw v0.72.1
Symptom
Every consumer workflow_call dispatch against a private gh-aw callee fails at the lock.yml's activation step with:
Cross-repo invocation detected: workflow source is "<callee>",
current repo is "<caller>"
GET /repos/<callee>/contents/.github/workflows/<chore>.lock.yml
?ref=<callee-ref-sha> — 404 Not Found
Could not fetch content for .github/workflows/<chore>.lock.yml: Not Found
Unable to fetch lock file content for hash comparison via API,
trying local filesystem fallback
Local lock file not found: …/work/<caller>/<caller>/.github/workflows/<chore>.lock.yml
##[warning]Could not compare frontmatter hashes — assuming lock file is outdated
##[error]ERR_CONFIG: Lock file '.github/workflows/<chore>.lock.yml' is outdated
or unverifiable! Could not verify frontmatter hash for
'.github/workflows/<chore>.md'. Run 'gh aw compile' to regenerate the lock file.
The check_workflow_timestamp_api.cjs activation step then sets stale_lock_file_failed=true, and the conclusion job marks the whole run failed before the agent ever runs.
Concrete reproducer: gominimal/min-ctl run 25906428806 calls gominimal/min-aw/.github/workflows/worker-fix.lock.yml@v0.6.3. Both repos private, same org. Fails as above on every dispatch.
Root cause
check_workflow_timestamp_api.cjs issues the Contents API call against the callee repo using GITHUB_TOKEN from the caller's runner. For a private callee:
GITHUB_TOKENon a workflow run is repo-scoped to the running repo (the caller, e.g.gominimal/min-ctl).- The Contents API on a private repo requires read access to that specific repo.
gominimal/min-ctl'sGITHUB_TOKENhas no scope ongominimal/min-aw→ 404, even though both repos are in the same org and workflow_call itself was already authorized via repo-actions-access settings.
This works fine when the callee is public (elastic/ai-github-actions is public; their consumers don't see this) — public Contents API doesn't require auth. The gap is private-callee-private-caller.
Why the existing fallbacks don't help
- API path uses
GITHUB_TOKEN→ 404 for private callees. GH_AW_GITHUB_TOKENfallback secret slot is declared optional in the lock.yml'sworkflow_call.secrets. Consumers can provide a token here, but the documentation/templates don't make this clear, and there's no practical way to mint a short-lived cross-repo token without long-lived PAT storage (security cost).- Local filesystem fallback looks for the lock.yml on the caller's checkout. Consumers using the wrapper distribution pattern don't ship
.lock.ymlfiles locally (that's the point — thin wrappers).
Proposed fix
The lock.yml already receives APP_PRIVATE_KEY via workflow_call.secrets (used for safe-output writes via actions/create-github-app-token). If the corresponding GitHub App is installed on the callee repo as well, an App-minted installation token has cross-repo Contents read access.
check_workflow_timestamp_api.cjs should:
- Detect cross-repo invocation (it already does — emits "Cross-repo invocation detected" log).
- Detect that the initial Contents API call returned 404 / 401 / 403.
- Mint a fresh installation token using
APP_PRIVATE_KEY+vars.APP_ID(both already available to the activation step), scoped to the callee repo. - Retry the Contents API call with the minted token.
- If the mint fails (no APP_PRIVATE_KEY in secrets, App not installed on callee, etc.), fall through to today's behavior with a clearer error message: "Configure GH_AW_GITHUB_TOKEN with cross-repo Contents read access, or install the gh-aw App on the callee repo."
Alternative if internal mint is too invasive: emit a clearer error directing consumers to set up GH_AW_GITHUB_TOKEN. The current message ("outdated or unverifiable") sends users toward gh aw compile, which doesn't address the auth gap.
Workaround for current users
Ship .lock.yml mirrors on the consumer side alongside .md mirrors. The filesystem fallback then succeeds. Trade-off: ~80–90 KB per chore on the consumer (defeats the thin-wrapper goal). See gominimal/min-ctl#66 for the workaround in practice.
Related
- Bundle-apply race: safe_outputs fetch-depth=1 loses ancestor commits between agent and safe_outputs jobs #32310 — bundle-apply race in safe_outputs (separate root cause, same setup)
Activity
kkruel8100 commentedon May 16, 2026
Experiencing same issue. Proposed workaround is not viable on central ops repo and large scale quantity of target repos. Please increase priority level and mark bug status.
pelikhan commentedon May 16, 2026
/scout do a deep research the mentioned workflows and repos and provide a summary with snippets of the current situation
github-actions commentedon May 16, 2026
🏕️ Lost in the wilderness! Scout failed to deliver outputs. Sending search party...
github-actions commentedon May 16, 2026
🔍 Scout Research Report
Triggered by
@pelikhanvia /scout commentExecutive Summary
The bug in #32312 is real, reproducible from the code in this repo, and the proposed fix path (mint an App-installation token in the activation step when cross-repo + 404/401/403) is sound. The root cause confirmed in code: the compiled
Check workflow lock filestep in every*.lock.ymlrunsactions/github-scriptwithout specifying agithub-token, so it inherits the caller's repo-scopedGITHUB_TOKEN— which has no read access to the private callee. Prior fixes (#23935, #24422) taught the script which file to look for cross-repo but did not change which token it uses to fetch it. The same reporter (@norrietaylor) filed a separate cross-repo race in #32310, suggesting the wrapper-distribution pattern is exercising a class of gaps.Click to expand detailed findings
Research Findings
1. Root cause confirmed in
scout.lock.yml(and every compiled lock.yml)The activation step that calls
check_workflow_timestamp_api.cjsis generated without agithub-token:override, sogithubinsidegithub-scriptis authenticated with the default callerGITHUB_TOKEN:Compare to safe-output steps in the same file that do use the fallback chain (
scout.lock.yml:1347+):The activation step is missing this.
getFileContent(actions/setup/js/github_api_helpers.cjs:69-101) then callsgithub.rest.repos.getContenton the callee owner/repo with the caller's token, producing the 404 the issue describes.2. Cross-repo detection already exists — only the token slot is missing
check_workflow_timestamp_api.cjsalready does the right resolution (actions/setup/js/check_workflow_timestamp_api.cjs:86-160): it callsactions.getWorkflowRunto readreferenced_workflows[], picksshaoverref, and logsCross-repo invocation detected: workflow source is "<callee>", current repo is "<caller>"— that's the exact line in the issue's error block. So the script already knows it's cross-repo. It just doesn't switch tokens when it does.The inline comments name this gap explicitly:
That fallback is the local checkout, which doesn't exist on wrapper consumers — the issue is correct that filesystem fallback is structurally useless for the thin-wrapper distribution pattern.
3. Prior fixes addressed the file, not the auth
GITHUB_WORKFLOW_REFpointing to caller instead of callee forworkflow_call. Same script, same symptom name ("outdated or unverifiable") but a different root cause — wrong path lookup.referenced_workflowsAPI lookup the script uses today.Both fixes ran on public callees where Contents API needs no auth. The private-callee case was never reached by those fixes because the test fixtures didn't exercise it.
4. Related bug in same setup: #32310 (closed)
Same reporter, same
gominimal/min-aw↔gominimal/min-ctlrepos. Bundle-apply race insafe_outputsbecause checkout usesfetch-depth: 1and the bundle's base SHA falls out of reach whenmainadvances between the agent and safe-outputs jobs. Different root cause, but it confirms the wrapper-distribution + private-callee pattern is being actively used and surfacing edge cases the existing test coverage misses.5. The lock.yml secrets contract already names the right slot
Every compiled
workflow_call:block already declares the optional slot (e.g.dependabot-worker.lock.yml):So the consumer-side mechanism to pass a cross-repo-capable token already exists. The activation step just doesn't read it. Adding
github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }}to the compiledCheck workflow lock filestep (incompiler_activation_job_builder.go/compiler_activation_job.go) would let consumers who provision the secret pass it through without any new mechanism.6. App-token mint path is feasible and idiomatic
The reporter's preferred fix (use
APP_PRIVATE_KEY+vars.APP_IDalready in scope to mint a fresh installation token) is the GitHub-recommended pattern for cross-repo private reads — see the [Cloud Chronicles guide]((cloudchronicles.blog/redacted) and GitHub's own authentication docs: "If you need a token that requires permissions that aren't available in theGITHUB_TOKEN, create a GitHub App and generate an installation access token within your workflow."actions/create-github-app-tokenaccepts an explicitowner:parameter so the minted token can be scoped to the callee org/repo. The same action is already used elsewhere in safe-output jobs in this codebase, so introducing it into the activation job is consistent.Key constraint to verify before implementing: the App must be installed on the callee repo too, not just the caller. The fallback error message should make this explicit.
7. The error message points users in the wrong direction
From
check_workflow_timestamp_api.cjs:343:When the actual failure is an auth 404 against the callee,
gh aw compiledoes nothing useful — the lock file is fine, the script just can't read it. Even without changing the auth behavior, branching the message on the cross-repo + API-failure case would save users the wild-goose chase.Recommendations
In priority order:
compiler_activation_job_builder.go, emitgithub-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }}on theCheck workflow lock filestep. Consumers who already provisionGH_AW_GITHUB_TOKEN(PAT or App-minted) get the fix for free; no new contract.workflowRepo !== currentRepoAND the API call returns 401/403/404 AND the local filesystem fallback finds no lock file, fail with: "Cross-repo lock check failed: the caller'sGITHUB_TOKENcannot read<callee>. ProvisionGH_AW_GITHUB_TOKEN(PAT or App installation token with Contents:read on the callee), or install the gh-aw GitHub App on the callee repo." Suppress the "Rungh aw compile" suggestion in this branch — it's misleading.APP_PRIVATE_KEY+APP_IDare present, mint an installation token scoped to the callee owner and retry. Falls back to message in (2) when the mint fails. This removes the consumer's burden of provisioning a separate PAT but adds App-install-on-callee as a prerequisite — document that clearly.pkg/workflow/call_workflow_secrets_test.govalidates the secrets list shape but doesn't exercise the activation-step auth path. A test that builds a fakereferenced_workflowsAPI response and asserts the step uses the fallback token (notGITHUB_TOKEN) would have caught this and would catch regressions in (1)/(3).docs/src/content/docs/reference/cross-repository.mdwith a "Private callee" section. Today it covers "reading from other private repositories" for MCP tooling but not the lock-file activation path.The minimum to unblock
@norrietaylorand@kkruel8100right now is #1 + #2. Those are a few-line change in the compiler and a conditional in the script; both are independent of the App-mint design and can ship in a patch release without breaking any existing consumer.Key Sources
actions/setup/js/check_workflow_timestamp_api.cjs— lines 86–160 (cross-repo detection), 246–282 (compareFrontmatterHashes), 343 (misleading error)actions/setup/js/github_api_helpers.cjs:69-101—getFileContentusing caller-scopedgithubclient.github/workflows/scout.lock.yml:223-234— activation step with nogithub-token:GITHUB_TOKENis repo-scoped by design: community/discussions/46566Suggested Next Steps
github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }}to the activation step generator and regenerating affected lock files (recommendation 1).actions/github-scriptinvocation carries the fallbackgithub-token(recommendation 4)..lock.ymlmirrors) works, but provisioningGH_AW_GITHUB_TOKENas a workflow_call secret with Contents:read on the callee repo will work today without any gh-aw change — the secret slot is already declared in every compiled lock.yml.pelikhan commentedon May 16, 2026
@copilot follow scout plan, ignore app mint for now.
norrietaylor commentedon May 16, 2026
Thanks Peli!
I will try this out once it lands and report back.
norrietaylor commentedon May 16, 2026
#32530 implements Scout's recommendation #2 (legible error) only. Its diff touches
check_workflow_timestamp_api.cjs+ test + changeset — no compiler change.Recommendation #1 — emit
github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }}on the compiledCheck workflow lock filestep (compiler_activation_job_builder.go) — is still open. That is the change that lets the activation step actually read a private callee.Without #1, #32530's new message is not actionable: it instructs operators to "Configure
GH_AW_GITHUB_TOKEN", but the activation step has nogithub-token:slot, so a provisionedGH_AW_GITHUB_TOKENis never consumed. The compiledworkflow_callblock already declares the optional secret (GH_AW_GITHUB_TOKEN: required: false), so #1 is a few-line generator change with no new contract.Request: keep this issue open after #32530 merges, or track #1 as a follow-up PR. #32530 makes the failure legible but does not unblock the private-to-private
workflow_callpath.