Skip to content

fix(qa): slug-scope QA content publish + resilient commit + lock release - #2069

Merged
jung-thomas merged 1 commit into
DEVfrom
fix/qa-slug-scoped-publish
Aug 28, 2026
Merged

jung-thomas merged 1 commit into
DEVfrom
fix/qa-slug-scoped-publish

Conversation

@jung-thomas

Copy link
Copy Markdown
Contributor

Why

Rebuild Content (QA) intermittently failed at the publish commit with gorouter 502 → 503 (run 33164190492). Not OOM/not a crash — tutorials-srv-qa (2G, single instance) never restarted. Root cause: every QA rebuild republishes all 2211 slugs in force mode, so the commit is a ~35s single request that is fragile to a transient routing blip on the single instance; the client's 3 retries fell inside a ~4s window and couldn't ride it out.

Why it was still full-force despite two prior fixes: #2057 made only the fetch/build phase slug-aware. #2062 was a half-done version of this fix — it added a force-publish input and wired --channel qa ${{ inputs.force-publish && '--force' || '' }} intending delta-by-default, but never removed the force:true hardcode in resolvePublishConfig, so the config override still forced a full publish and no PR ever wired PUBLISH_SLUG. Option B is now fully active on QA (DB-config delta flags default ON, #2061/#2064), so a slug-scoped delta QA publish is safe (mutable ContentCurrent, server carries unchanged slugs — no page wipe).

Changes

#1 — slug-scoped QA publish (completes #2062)

  • resolvePublishConfig qa branch: force now honors --force (mirrors prod) instead of hardcoded true.
  • rebuild-content-qa.yml: forward the dispatched slug as PUBLISH_SLUG (empty on a force-publish re-seed). Commit-triggered → one slug, sub-second commit; full rebuild → whole-catalog delta; force-publish=true → full re-seed.
  • fetchRemoteHashes/fetchRemoteSourceHashes: send Bearer when apiKey is set — srv-qa gates /content/hashes (verified 401 unauthenticated), unlike the public prod route; without this QA delta would 401.

#2 — resilient commit retry

  • withRetry: optional ±jitter via computeBackoff (default 0 → existing callers unchanged).
  • commit call: attempts 3→5, backoff [1s,3s,9s]→[2s,5s,10s,20s] + 20% jitter (the old schedule only ever waited ~4s total; the 9000 was dead).

#3 — release publish lock on commit failure

  • On permanent commit failure the client now abortSessions (mirrors the append-failure path) → marks manifest FAILED + releases the lock, instead of stranding it for the 30-min TTL (which 409 Another publish in progress'd every QA rebuild in that window). Abort has its own transient retry; the stuck-manifest reaper remains the backstop.
  • Server-side auto-abort deliberately omitted: it would turn the client's commit retries into 409s and defeat retry of a transient internal error; the client owns that decision.

Docs: corrected "QA always --force" in build.md + the QA endpoint design spec.

Testing

  • 50 publish-layer unit tests pass (publish-content-qa force/delta + workflow-wiring guard, withRetry jitter, fetchRemoteHashes auth header).
  • Full npm test has 132 pre-existing failures unrelated to this change (CAP @cap-js/db-service OData service tests) — confirmed identical on clean origin/DEV (stash-verified); nothing in the touched publish layer fails.

Verify after merge / on DEV

  • Trigger a single-slug QA workflow_dispatch (slug=<one>): expect the publish log to show a small changed count + sub-second commit, and the verify-publish watchdog still reporting the full 2211 slugs served (carry-forward intact).
  • Confirm a no-slug rebuild still lands the catalog and force-publish=true still re-seeds.

🤖 Generated with Claude Code

The Rebuild Content (QA) workflow intermittently failed at the publish commit
with gorouter 502→503 (run 33164190492). Root cause: every QA rebuild — even a
single-tutorial tutorial-qa-updated dispatch — republished all 2211 slugs in
force mode, making the commit a ~35s single request that is fragile to a
transient routing blip on the single srv-qa instance; the client's 3 retries
fell inside a ~4s window and couldn't ride it out.

#2062 was a half-done version of this fix: it added a force-publish input and
wired `--channel qa ${force-publish && '--force'}` intending delta-by-default,
but never removed the force:true hardcode in resolvePublishConfig, so the config
override still forced a full publish and no PR ever wired PUBLISH_SLUG. Option B
is now fully active on QA (DB-config delta flags default ON, #2061/#2064), so a
slug-scoped delta QA publish is safe.

Fix #1 — slug-scoped QA publish:
- publish-content.ts resolvePublishConfig qa: force now honors --force (mirrors
  prod) instead of hardcoded true.
- rebuild-content-qa.yml publish step: forward the dispatched slug as
  PUBLISH_SLUG (empty on force-publish re-seed). Commit-triggered rebuild → one
  slug, sub-second commit; full rebuild → whole-catalog delta; force-publish →
  full re-seed.
- fetchRemoteHashes/fetchRemoteSourceHashes: send Bearer when apiKey is set —
  srv-qa gates /content/hashes (verified 401 unauthenticated), unlike public
  prod route; without this, QA delta would 401.

Fix #2 — resilient commit retry:
- withRetry: optional ±jitter via computeBackoff (default 0, callers unchanged).
- commit call: attempts 3→5, backoff [1s,3s,9s]→[2s,5s,10s,20s]+20% jitter so a
  brief blip is ridden out instead of failing in ~4s.

Fix #3 — release publish lock on commit failure:
- On permanent commit failure the client now aborts the session (mirrors the
  append-failure path) which marks the manifest FAILED and releases the lock,
  instead of stranding it for the 30-min TTL (which 409'd every QA rebuild in
  that window). Abort gets its own transient retry; reaper remains the backstop.
  (Server-side auto-abort deliberately omitted: it would turn the client's
  commit retries into 409s and defeat retry of a transient internal error.)

Docs: correct "QA always --force" in build.md + qa-endpoint-design.md.
Tests: publish-content-qa force/delta + workflow-wiring guard; withRetry jitter;
fetchRemoteHashes auth header. 50 publish-layer tests pass.
@jung-thomas
jung-thomas merged commit 2bec9d9 into DEV Aug 28, 2026
2 of 3 checks passed
@jung-thomas
jung-thomas deleted the fix/qa-slug-scoped-publish branch August 28, 2026 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant