AI-quizzes: 2-week rollout window before dropping AI_AUTHOR_ENABLED flag (follow-up to #275)

[jung-thomas]

Background

#275 graduates the AI-authored quizzes spike. The spec proposes "drop the AI_AUTHOR_ENABLED flag, ship to all tutorials." Counter-proposal: keep the flag for a soft rollout window, then drop.

Proposal

1. Now: Set AI_AUTHOR_ENABLED=true as a default in rebuild-content.yml (and the QA equivalent rebuild-content-qa.yml).
2. For 2 weeks: Monitor:
   • CI quiz-cache build failures (any uncaught exceptions during expansion).
   • Author-side feedback — surfaced via Joule chat or direct ping.
   • Cost telemetry — token usage per build via Analytics Explorer.
3. After 2 weeks of clean signal: Open a PR that removes the flag check entirely, making AI-quiz expansion always-on.

Why this and not a hard cutover

The flag costs us one CI env var line. The upside is a fast kill-switch if a model regression sneaks in (say, a model deployment swap that changes generation quality silently). After 2 weeks of production data we'll know if the kill-switch is needed; until then it's cheap insurance.

Acceptance

• AI_AUTHOR_ENABLED=true set in rebuild-content.yml and rebuild-content-qa.yml workflow env.
• First post-graduation rebuild produces caches for >100 tutorials cleanly.
• After 2 weeks: review failure / cost / feedback telemetry. If clean, open a follow-up PR removing the flag.

Refs

• AI-authored quizzes: evaluation + decision (follow-up to #208) #275 (pilot decision)
• AI-authored quizzes: generate [VALIDATE_N] candidates from tutorial body + rules.vr (follow-up to #171) #208 (original spike)
• PR feat: AI-authored quizzes spike (closes #208) #260 (impl + flag)

[jung-thomas]

Soak clock starts: 2026-06-13

PR #320 merged today, flipping ai-author-enabled default to true on both rebuild-content.yml and rebuild-content-qa.yml for workflow_dispatch triggers. The 2-week soak window starts now.

The companion empty-step guard (#311) shipped in the same PR — protects against the abap-create-project-step-5 hallucination shape that drove the only no verdicts in #275's pilot. Catches steps with fewer than 50 substantive words and skips generation rather than letting the LLM confabulate.

What's being soaked

• Default-on AI-quiz authoring on every workflow_dispatch rebuild (full or per-slug).
• Empty-step guard skipping LLM calls for placeholder/empty steps.
• Build summary line gained an <n> empty-step skipped counter visible in the GH Actions log.
• repository_dispatch (admin-write debounce rebuilds from PR feat: admin-write rebuild trigger for /browse/ freshness (#174) #220) deliberately stays at the empty-string fallback — admin tweaks don't trigger LLM calls.

What to monitor (until 2026-06-27)

Signal	Where	Threshold for concern
Workflow failures	Actions → Rebuild Content	Any failed step in the AI-quiz block (VCAP construction, fetch-tutorials AI section, build summary missing)
Cost telemetry	Analytics Explorer → CodeCheckSubmissions table is for code-check; AI-quiz cost lives in the build summary line	>$5/build sustained, or >2× the #275 pilot cost (~$1.10 / 6 tutorials = ~$0.18/tutorial)
Empty-step skip rate	Build summary <n> empty-step skipped token	>20% of total directives skipped (could indicate either a parser regression OR many tutorials with placeholder steps that need author cleanup)
Author feedback	Joule chat / direct ping / community comments	Any quality regression complaint pointing at AI-authored content
Regression on first big rebuild	First post-merge run with > 100 tutorials	Total errors > 0 in summary line

Decision points

• Clean signal across 2 weeks (~4 rebuilds at typical cadence) → open a follow-up PR removing the AI_AUTHOR_ENABLED flag check entirely from scripts/fetch-tutorials.ts:686-690 and scripts/lib/expand-ai-authored.ts:32. Close this issue at that point.
• Quality regression → flip ai-author-enabled=false on the relevant workflow as a kill-switch, file a follow-up issue describing the regression, leave this open.
• Empty-step skip rate too high → investigate whether the rate reflects real author-side gaps (file a tutorial-cleanup issue) vs a parser regression (file a AI-quizzes: skip generation on empty/placeholder steps (follow-up to #275) #311 follow-up).

Cost expectation

The #275 pilot ran 6 tutorials → 90 LLM calls → ~$1.10. CI rebuilds running over the full 1397-tutorial catalog with sparse [AUTOAUTHOR_*] directives will produce far fewer calls (only tutorials whose authors have opted in). Expected cost per full rebuild: $5–15 once authoring uptake reaches double-digit tutorials, with the per-build hard cap (AI_AUTHOR_BUILD_CAP=200) as the worst-case ceiling.

Refs

• PR feat(311+313): AI-quizzes graduation tail — empty-step guard + CI default-on #320 (this work)
• AI-authored quizzes: evaluation + decision (follow-up to #208) #275 (graduation), AI-quizzes: skip generation on empty/placeholder steps (follow-up to #275) #311 (empty-step guard, closed in feat(311+313): AI-quizzes graduation tail — empty-step guard + CI default-on #320), AI-quizzes: wire AI_AUTHOR_ENABLED + CHAT_DEPLOYMENT_ID into rebuild-content.yml (follow-up to #275) #313 (CI wiring, closed in feat(311+313): AI-quizzes graduation tail — empty-step guard + CI default-on #320)
• AI-authored quizzes: generate [VALIDATE_N] candidates from tutorial body + rules.vr (follow-up to #171) #208 (original spike), feat: AI-authored quizzes spike (closes #208) #260 (impl), fix + polish: #208 spike LLM bugs + follow-up nits #261 (pilot + initial CI wiring)

[jung-thomas]

First soak rebuild — AI-author block ✓, publish step ✗ (pre-existing)

Run: 27471972865 (workflow_dispatch, DEV target, ai-author-enabled=true, build-cap=200). Triggered 2026-06-13T16:09Z.

What this run validated

1. Two missing secrets surfaced and were set:

   • AI_AUTHOR_AICORE_SERVICE_KEY (the tutorials-aicore service-key credentials JSON)
   • CHAT_DEPLOYMENT_ID = da6d9c5e3fb50c3d
     Both were absent before this rebuild — PR feat(311+313): AI-quizzes graduation tail — empty-step guard + CI default-on #320's flag-flip made the workflow default to needing them but didn't set them. PR feat(311+313): AI-quizzes graduation tail — empty-step guard + CI default-on #320 had a latent bug: with default-on flag and no secrets, every default-input rebuild would fail-fast at VCAP construction. Now resolved.

2. AI-author block end-to-end success:

   [ai-author] expanded directives across all tutorials: 8 cache miss (LLM call), 0 cache hit, 0 errors, 0 empty-step skipped. Build cap: 200.
   

   Eight tutorials had [AUTOAUTHOR_*] directives. All eight succeeded; zero errors; zero empty-step skips (none of the AI-author tutorials have placeholder steps). The <n> empty-step skipped token from AI-quizzes: skip generation on empty/placeholder steps (follow-up to #275) #311 is present in the summary, confirming the post-AI-quizzes: skip generation on empty/placeholder steps (follow-up to #275) #311 line shape is what CI emits and matches the no-upstream-errors invariant regex shipped in PR feat(311+313): AI-quizzes graduation tail — empty-step guard + CI default-on #320.

3. Cost estimate (this rebuild): 8 calls × ~$0.014/call avg ≈ $0.10–0.15. At the current authoring uptake, even a daily rebuild stays well under $5/day.

4. The new code paths from PRs fix(314+315): unblock Phase 4 substrate — route shadowing + harness HTTP refactor #316, feat(311+313): AI-quizzes graduation tail — empty-step guard + CI default-on #320, fix(318+319): chat-settings resolver + telemetry HANA SQL #321 all worked end-to-end through the full 1397-tutorial fetch:

   • AI-quizzes: skip generation on empty/placeholder steps (follow-up to #275) #311 empty-step guard: 0 false positives on the 8 AI-author tutorials.
   • AI-quizzes: wire AI_AUTHOR_ENABLED + CHAT_DEPLOYMENT_ID into rebuild-content.yml (follow-up to #275) #313 default-on flag: workflow dispatched with default ai-author-enabled=true.
   • ChatSettings.deploymentId should auto-populate from CHAT_DEPLOYMENT_ID env var (follow-up to #210) #318 chat-settings resolver: deploymentId resolved cleanly via env-var path; no throw.

What failed (orthogonal to #312)

The Publish tutorial content to HANA step failed with:

Error: No API key. Set CONTENT_API_KEY env var or pass --api-key

Cause: the CONTENT_API_KEY repo secret is not set. Only CONTENT_API_KEY_QA exists. This is unrelated to #312/#311/#313 — it's a pre-existing CI gap (the 3 prior May runs of this workflow all failed too, before AI authoring shipped). Filing a separate issue. The fix is one gh secret set CONTENT_API_KEY command per the value documented in CLAUDE.md under "Content Publishing".

The publish failure does not block #312 because:

• The AI-author block runs before publish (Fetch tutorials step). It completed successfully with full numbers above.
• The 2-week soak is about the AI-author quality + reliability, not whether the rebuild reaches publish.
• Publishing is independently verified each time Tom runs his canonical local deploy (mbt build && cf deploy), which is the actual production path — CI publish has been broken since before AI authoring existed and content has shipped fine.

Soak status

Clock starts NOW. First validation point is this run. Decision point ~2026-06-27 (2 weeks from soak-clock-start in PR #320, less the 4-hour gap before the first run).

Monitoring criteria from the prior #312 comment still hold. Observations after this run:

Signal	Status
Workflow failures (AI block)	0 errors out of 8 LLM calls
Cost telemetry	$0.10–0.15 / 8 directives — well under $5 threshold
Empty-step skip rate	0/8 (= 0%) — well under 20% threshold
Author feedback	None yet
Regression on first big rebuild	AI block: pass. Publish: pre-existing failure (see above).

Follow-up filed

• (separate from AI-quizzes: 2-week rollout window before dropping AI_AUTHOR_ENABLED flag (follow-up to #275) #312) CONTENT_API_KEY repo secret missing — blocking the publish step in rebuild-content.yml. Fix is a one-line gh secret set. Filing in a moment.

Next checkpoints

• Day 7 (~2026-06-20): if any author-write or workflow_dispatch rebuild fires in the interim, summarize signal here.
• Day 14 (~2026-06-27): decision point. If clean signal across whatever runs occurred, open a follow-up PR removing the AI_AUTHOR_ENABLED flag check from scripts/fetch-tutorials.ts:690 and scripts/lib/expand-ai-authored.ts. Closes this issue.
• If quality regression: flip ai-author-enabled=false on the workflow as a kill-switch, file the regression as a follow-up, leave this open.

[jung-thomas]

Soak window complete — flag-removal PR opened

Soak clock: 2026-06-13 → 2026-06-27 (14 days, 2 weeks). Decision point hit today.

Aggregated signal across the window

~20 successful rebuild-content.yml runs. Spot-checked the AI-author summary line across runs at every interval — every single one came back identical in shape: 0 errors, 0 empty-step skipped. Cache warmed early in the window (after the seed run on 2026-06-13), so most runs hit cache rather than firing fresh LLM calls.

Run	Date	Telemetry
27598642246	2026-06-16	8 cache miss (LLM call), 0 errors, 0 empty-step skipped
27631866181	2026-06-16	8 cache hit, 0 errors, 0 empty-step skipped
27910294128	2026-06-21	10 cache hit, 0 errors, 0 empty-step skipped
28047420373	2026-06-23	10 cache hit, 0 errors, 0 empty-step skipped
28292054065	2026-06-27	10 cache hit, 0 errors, 0 empty-step skipped

The recent failures during the window (28304515829, 28322396467) are orthogonal — upstream-source 404s and a publish-step verify mismatch respectively. The AI-author block completed cleanly in both before those downstream steps failed.

Decision criteria — all green

Signal	Threshold	Observed
Workflow failures in the AI block	Any	0
Cost per build	>$5 sustained	~$0.10-0.15 (8 calls × ~$0.014 once cache warmed)
Empty-step skip rate	>20%	0%
Author feedback / quality regressions	Any complaint	None filed
First big rebuild	Errors > 0	0 errors across 1397-tutorial catalog

Action taken

Opened #697 to drop the AI_AUTHOR_ENABLED flag:

• scripts/fetch-tutorials.ts — flag gates around expandAiAuthoredQuestions() and the telemetry summary line removed; summary line now always emitted so the CI no-upstream-errors invariant regex keeps matching.
• rebuild-content.yml + rebuild-content-qa.yml — ai-author-enabled workflow_dispatch input removed; VCAP_SERVICES setup runs for all non-catalog-only modes.
• scripts/preflight-ai-quiz-smoke.ts + package.json — redundant AI_AUTHOR_ENABLED=true env injections removed.
• Operational docs (CLAUDE.md, docs/launch.md, docs/developers/architecture/ai-authored-quizzes.md, docs/developers/operations/ai-author-ci-setup.md, docs/developers/reference/cap-cds-gotchas.md) — prose updated to describe always-on + the new kill-switch (empty out AI_AUTHOR_AICORE_SERVICE_KEY repo secret → SDK fails fast).
• Spec/plan docs under docs/superpowers/ deliberately untouched — they're point-in-time records.

Cost remains bounded by author opt-in: only tutorials with [AUTOAUTHOR_*] in their rules.vr ever produce LLM calls. The parser emits zero placeholders for everything else, and expandAiAuthoredQuestions() fast-exits as a no-op. The AI_AUTHOR_BUILD_CAP=200 per-build ceiling stays as the worst-case backstop.

Will close this issue when #697 merges.

cc @jung-thomas