Quizzes: explore AI-authored quizzes, AI-graded answers, and AI code-check

[jung-thomas]

Daniel asks whether we could:

1. Have AI generate a quiz for a tutorial (from the tutorial body and rules.vr material), rather than relying solely on author-written validation rules.
2. Have AI grade quiz answers — including free-text — instead of strict rule matching.
3. Have the user paste code and have AI check whether the code accomplishes the tutorial's goal.

This dovetails with the existing Joule / chat / RAG infrastructure and embeddings store.

Action

• Spike: prototype an AI-generated quiz for one tutorial; compare to the hand-authored rules.vr quiz.
• Spike: AI code-check tool that takes user code + tutorial step context and returns pass/fail with feedback.
• Decide which (if any) become first-class features and what the author opt-in / fallback story is.

---

Source: feedback from Daniel Wroblewski, 2026-06-01

[jung-thomas]

Brainstorm output: scoped spike to AI code-check (capability #3)

Spent some design time on this and wrote up the result as a spec. Branch + diff: spec/171-ai-code-check → docs/superpowers/specs/2026-06-02-ai-code-check-spike-design.md.

Why one capability, not three

The issue bundles three subsystems that share infrastructure but have very different shapes:

1. AI-authored quizzes — generate questions from tutorial body. Build-time. Hardest to evaluate (needs the renderer + grader to compare against rules.vr).
2. AI free-text grader — replaces strict-match on existing [VALIDATE_N] text questions. Runtime. Needs the missing client-side validation widget either first or alongside.
3. AI code-check — paste box per step, AI says pass/partial/fail. Net-new surface. Smallest scope.

Spec'ing all three at once produces a spec that's vague about each. We start with #3 because it has zero existing-surface coupling and the clearest learner value if it lands. Capabilities #1 and #2 get follow-up specs once we have evaluation data from this spike.

Shape of the code-check spike

• Authoring: authors extend their existing rules.vr in the -Contribution repo with [CODECHECK_N] blocks (###Goal, optional ###Language, ###Hints, ###ReferenceSolution). Same fetch/parse/cache pipeline as [VALIDATE_N].
• Two channels for the parsed data: trimmed (no reference solution) goes to Hugo frontmatter; full spec goes to a new CodeCheckSpecs entity in HANA via the publish pipeline. Reference solutions never reach HTML.
• Runtime: registered as a checkCode Joule chat tool in chat-orchestrator.js. Force-called via tool_choice from a new /api/codecheck endpoint that the inline UI uses (returns the structured tool-call args directly, no streamed assistant reply). Also organically callable when a learner pastes code in chat.
• Grader: one OrchestrationClient chat completion with response_format: json_schema enforcing { verdict, summary, suggestions[], correctAspects[] }. temperature: 0.1. Model from ChatSettings.modelName (defaults gpt-4o). Belt-and-braces post-process redaction if the verdict text contains 30+ chars of the reference solution.
• UI: UI5 panel with goal, hints, paste box, structured verdict, "Try again", "Ask Joule about this". Plain <textarea> for the spike (Monaco is too heavy and CSP-gated).
• Auth: XSUAA-required. Per-user 30 checks/hour and per-step 5/5min rate limits. Anonymous learners see a sign-in CTA in place of the paste box.
• Persistence: every submission lands in CodeCheckSubmissions with @PersonalData annotations and full token/latency telemetry. Anonymization cascades. @analytics.exposed so Analytics Explorer can drive the eval.
• Cost: ~$0.007 per check at gpt-4o ($0.21/user/hour at the rate cap).
• Feature flag: ChatSettings.codeCheckEnabled (default off).

Spike exit criteria

The point of the spike is to answer "should this be first-class," and we're committing to evidence:

• Phase 3: authors curate 30 sample submissions per pilot tutorial; we run scripts/evaluate-code-check.js; authors rate verdicts as agree / disagree / partial.
• Phase 4 decisions:
  • ≥80% agreement on the pass-vs-fail boundary → propose graduating with the gaps from eval addressed.
  • <80% but salvageable → propose Approach C (RAG-then-grade) as iteration 2.
  • <60% → spike negative; close without graduation; retain code behind the flag.

Cost & latency telemetry from CodeCheckSubmissions plus Cloud Logging answers the affordability side independently.

Explicitly out of scope

• AI free-text grader for [VALIDATE_N] (capability feat: add slug mapping endpoint and getEventProgress fallback #2 — separate spec).
• AI-authored quiz generation (capability feat: implement Admin UI (Fiori Elements + freestyle SAPUI5) #1 — separate spec).
• A shipped client-side validation widget for existing rules.vr questions — hugo/layouts/shortcodes/tutorial-step.html:17 mount point still has nothing on it; that gap remains.
• Monaco/CodeMirror; multi-file submissions; lint hints.
• Retrieve-related-steps via embeddings (Approach C, conditional on Phase 4).
• Author admin UI for browsing submissions (Analytics Explorer covers ad-hoc).

Next step

If you sign off on the design, I'll invoke the writing-plans skill to turn it into an executable plan. Spec branch is spec/171-ai-code-check; happy to take comments inline on the file if anything wants tightening.