fix(qa): slug-scope QA content publish + resilient commit + lock release - #2069
Merged
Merged
Conversation
The Rebuild Content (QA) workflow intermittently failed at the publish commit with gorouter 502→503 (run 33164190492). Root cause: every QA rebuild — even a single-tutorial tutorial-qa-updated dispatch — republished all 2211 slugs in force mode, making the commit a ~35s single request that is fragile to a transient routing blip on the single srv-qa instance; the client's 3 retries fell inside a ~4s window and couldn't ride it out. #2062 was a half-done version of this fix: it added a force-publish input and wired `--channel qa ${force-publish && '--force'}` intending delta-by-default, but never removed the force:true hardcode in resolvePublishConfig, so the config override still forced a full publish and no PR ever wired PUBLISH_SLUG. Option B is now fully active on QA (DB-config delta flags default ON, #2061/#2064), so a slug-scoped delta QA publish is safe. Fix #1 — slug-scoped QA publish: - publish-content.ts resolvePublishConfig qa: force now honors --force (mirrors prod) instead of hardcoded true. - rebuild-content-qa.yml publish step: forward the dispatched slug as PUBLISH_SLUG (empty on force-publish re-seed). Commit-triggered rebuild → one slug, sub-second commit; full rebuild → whole-catalog delta; force-publish → full re-seed. - fetchRemoteHashes/fetchRemoteSourceHashes: send Bearer when apiKey is set — srv-qa gates /content/hashes (verified 401 unauthenticated), unlike public prod route; without this, QA delta would 401. Fix #2 — resilient commit retry: - withRetry: optional ±jitter via computeBackoff (default 0, callers unchanged). - commit call: attempts 3→5, backoff [1s,3s,9s]→[2s,5s,10s,20s]+20% jitter so a brief blip is ridden out instead of failing in ~4s. Fix #3 — release publish lock on commit failure: - On permanent commit failure the client now aborts the session (mirrors the append-failure path) which marks the manifest FAILED and releases the lock, instead of stranding it for the 30-min TTL (which 409'd every QA rebuild in that window). Abort gets its own transient retry; reaper remains the backstop. (Server-side auto-abort deliberately omitted: it would turn the client's commit retries into 409s and defeat retry of a transient internal error.) Docs: correct "QA always --force" in build.md + qa-endpoint-design.md. Tests: publish-content-qa force/delta + workflow-wiring guard; withRetry jitter; fetchRemoteHashes auth header. 50 publish-layer tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Rebuild Content (QA)intermittently failed at the publish commit with gorouter502 → 503(run 33164190492). Not OOM/not a crash —tutorials-srv-qa(2G, single instance) never restarted. Root cause: every QA rebuild republishes all 2211 slugs in force mode, so the commit is a ~35s single request that is fragile to a transient routing blip on the single instance; the client's 3 retries fell inside a ~4s window and couldn't ride it out.Why it was still full-force despite two prior fixes:
#2057made only the fetch/build phase slug-aware.#2062was a half-done version of this fix — it added aforce-publishinput and wired--channel qa ${{ inputs.force-publish && '--force' || '' }}intending delta-by-default, but never removed theforce:truehardcode inresolvePublishConfig, so the config override still forced a full publish and no PR ever wiredPUBLISH_SLUG. Option B is now fully active on QA (DB-config delta flags default ON,#2061/#2064), so a slug-scoped delta QA publish is safe (mutableContentCurrent, server carries unchanged slugs — no page wipe).Changes
#1 — slug-scoped QA publish (completes #2062)
resolvePublishConfigqa branch:forcenow honors--force(mirrors prod) instead of hardcodedtrue.rebuild-content-qa.yml: forward the dispatched slug asPUBLISH_SLUG(empty on aforce-publishre-seed). Commit-triggered → one slug, sub-second commit; full rebuild → whole-catalog delta;force-publish=true→ full re-seed.fetchRemoteHashes/fetchRemoteSourceHashes: sendBearerwhenapiKeyis set — srv-qa gates/content/hashes(verified 401 unauthenticated), unlike the public prod route; without this QA delta would 401.#2 — resilient commit retry
withRetry: optional ±jitter viacomputeBackoff(default 0 → existing callers unchanged).[1s,3s,9s]→[2s,5s,10s,20s]+ 20% jitter (the old schedule only ever waited ~4s total; the9000was dead).#3 — release publish lock on commit failure
abortSessions (mirrors the append-failure path) → marks manifestFAILED+ releases the lock, instead of stranding it for the 30-min TTL (which409 Another publish in progress'd every QA rebuild in that window). Abort has its own transient retry; the stuck-manifest reaper remains the backstop.Docs: corrected "QA always --force" in
build.md+ the QA endpoint design spec.Testing
npm testhas 132 pre-existing failures unrelated to this change (CAP@cap-js/db-serviceOData service tests) — confirmed identical on cleanorigin/DEV(stash-verified); nothing in the touched publish layer fails.Verify after merge / on DEV
workflow_dispatch(slug=<one>): expect the publish log to show a small changed count + sub-second commit, and the verify-publish watchdog still reporting the full 2211 slugs served (carry-forward intact).force-publish=truestill re-seeds.🤖 Generated with Claude Code