Rotate REBUILD_API_KEY to unblock rebuild-content workflow (401 since 2026-07-01) #903

Context

Rebuild-content workflow has been failing since 2026-07-01 with HTTP 401 on the Push content to AppRouter step:

• 2026-07-01 16:22 → HTTP 503 REBUILD_API_KEY not configured (pre-fix(approuter): resolve REBUILD_API_KEY from credstore, unblock rebuild-content.yml (#867) #871: process.env.REBUILD_API_KEY was empty)
• 2026-07-02 03:47 → HTTP 401 Unauthorized (post-fix(approuter): resolve REBUILD_API_KEY from credstore, unblock rebuild-content.yml (#867) #871: credstore returned a value, but it doesn't match the GH Actions secret)

Diagnosis: the credstore alias was seeded per #871's rollout step 2, but the seeded value drifted from secrets.REBUILD_API_KEY in GitHub. GitHub Actions secrets are write-only, so the "canonical" value can't be recovered — the fix is to rotate both stores from a single terminal in one sitting so they're guaranteed to match byte-for-byte.

Blast radius: none beyond the rebuild workflow. This bearer token gates only POST /admin/rebuild (the approuter's static-content hot-reload endpoint). Rotating it invalidates no user sessions and no other automations.

Prerequisites

• Riley online to approve the credstore write via /admin-ui/#secrets-display
• gh CLI authenticated against sap-tutorials/tutorials-ims
• Windows Git Bash or any POSIX shell — commands work on both

Rotation procedure

Do all steps in a single terminal session. Do not close the shell between steps 1 and 2 — $NEW_KEY must be identical on both writes.

1. Generate a fresh secret

NEW_KEY=$(node -e "console.log(require('crypto').randomBytes(32).toString('base64url'))")
echo "New key length: ${#NEW_KEY}"   # expect 43
# DO NOT echo $NEW_KEY itself into a shared terminal / screen share

2. Write it into the GitHub Actions secret

# NOTE: --body "$NEW_KEY" — NOT --body-file - or a stdin pipe.
# Piping via stdin appends a trailing \n (see memory
# feedback_gh_secret_set_trailing_newline.md) which is one of the
# exact failure modes that produced the current 401.
gh secret set REBUILD_API_KEY --repo sap-tutorials/tutorials-ims --body "$NEW_KEY"

Verify with gh secret list --repo sap-tutorials/tutorials-ims | grep REBUILD_API_KEY — the Updated column should reflect the current time.

3. Write the same value into the credstore

Two paths — Path A (admin UI) requires Riley's approval; Path B is scriptable.

Path A — admin UI (matches #871's rollout step 2):

1. Copy the key to the clipboard without a trailing newline — printf '%s' "$NEW_KEY" | clip. Regular echo "$NEW_KEY" | clip will append \n which will fail the constant-time compare in isAuthorizedBearer.
2. Open /admin-ui/#secrets-display
3. Find the REBUILD_API_KEY row → "Rotate" / "Set Value"
4. Paste $NEW_KEY. Take one look at the input field: no leading/trailing whitespace, no visible extra characters.
5. Save.

Path B — direct credstore write (bypasses the admin UI; only if Path A is unavailable):

# TODO: fill in the credstore CLI incantation once the runbook exists.
# The srv side uses srv/lib/credstore.js write helpers; there is no
# top-level script for direct writes today. Path A is the recommended
# path for the foreseeable future.

4. Flush the approuter's TTL cache

The credstore reader (approuter/lib/credstore-secret.js resolveSecret) caches values for 5 minutes. Either:

• Wait 5 minutes, or
• cf restart tutorials-approuter (immediate)

For DEV, restart is preferred — one downtime blip beats a 5-minute guessing game.

5. Verify

gh workflow run rebuild-content.yml --repo sap-tutorials/tutorials-ims --ref main \
  -f mode=catalog-only
gh run watch --repo sap-tutorials/tutorials-ims  # pick the run just triggered

The Push content to AppRouter step should return HTTP 200. If it still 401s, either:

• The values still differ — re-check the credstore value against $NEW_KEY (length + first 4 chars is usually enough to spot drift)
• The TTL cache hasn't expired and we didn't restart the app → cf restart tutorials-approuter

Failure modes to watch for

Each of these has a memory that describes it in more detail:

• Trailing \r\n from Windows clipboard paste — printf '%s' in step 3, look at the input field before saving
• Trailing \n from gh secret set — --body "$VAR", never --body-file - or stdin pipe (feedback_gh_secret_set_trailing_newline.md)
• TTL cache serving stale value after credstore write — restart or wait 5 min
• Wrote to a different environment's credstore — DEV/QA/PROD each have their own credstore instance; verify cf target before triggering rebuild

Related

• Root cause: PR fix(approuter): resolve REBUILD_API_KEY from credstore, unblock rebuild-content.yml (#867) #871 rollout step 2 (credstore seed) drifted from GH Actions secret
• Follow-up PR: closes fix(approuter): resolve REBUILD_API_KEY from credstore, unblock rebuild-content.yml (#867) #871's rollout step 3 by stripping REBUILD_API_KEY from deploy/{dev,qa,prod}.mtaext + the envsubst allowlist so DEV/QA/PROD can only read REBUILD_API_KEY from the credstore, eliminating drift-between-two-stores as a class of failure
• Rollout doc: github-dispatch-pat-rotation.md will be updated in the follow-up PR to reflect credstore-only rotation

Acceptance

• gh workflow run rebuild-content.yml -f mode=catalog-only → all steps green
• gh run list --workflow rebuild-content.yml --limit 5 shows the next successful run