[richshaw]

Fixes #570, the last of five stacked PRs. It's based on #576 (the questions screen). This is the PR that turns the questions on.

The problem

A Magic Compose description that's only an event's name, like "Bake sale", gives the model nothing to build slots from. It either makes three "TBD" rows or invents jobs and counts, and the organizer still uses one of the hour's drafts.

The fix

The prompt

Rule 4 (the drafter's never-invent text, NEVER_INVENT_MIDDLE.drafter in signup-rules.ts) now splits two cases:

If the prompt doesn't say what people sign up for (e.g. "make a signup for my kid's soccer team") and carries no answers to your questions, don't draft: ask, with the QUESTIONS reply. If it says what but not when or how many (e.g. "snack duty for the soccer team"), draft anyway: where the slots need dates or times it doesn't give, produce 1-3 placeholder slots with labels like "TBD: game 1"…

RULES_IN_BOTH and the AI connector's instructions are unchanged, and their tests prove it.

A QUESTIONS section after the output schema gives the reply shape and its rules:

• list only the topics the prompt leaves out, always with "what";
• offer 2 to 4 options, in the prompt's language, each at most 40 characters;
• never offer a date, a time or a count;
• never ask when a message carries "Answers to your questions:".

Capacity may be null for no limit, which the follow-up message's "no limit" relies on.

The worked examples:

• Example C, the vague soccer prompt, now asks: {"questions":{"topics":["what","when","how_many"],"whatOptions":["Bring snacks","Drive to away games","Help at practice"]}}.
• A new Example D ("Ushers for the school play, two parents per performance") says what but not when, so it still drafts TBD: performance 1–3 with 2 spots each. Find fixable problems in a draft and offer one-tap fixes #503 relies on those "TBD:" rows.

The golden file is regenerated; its diff is only these changes. With #526 now on main, the prompt carries both #526's CAPACITY section and this QUESTIONS section, and the golden-file test passes. A test parses every worked example with the server's schema.

Evals

The harness (scripts/lib/eval-harness.ts, scripts/eval-magic-compose.ts):

• A questions result arm.
• New outcomes:
  • unexpected_questions;
  • expected_questions_but_drafted;
  • asked_again: questions in reply to the answers, in any shape, read with the route's own asksQuestions.
• New checks: questions.must_ask (true or false), topics_must_include, topics_must_not_include, topics_must_equal, options_count_range, options_must_not_contain_date_or_number (English and French number words, months and weekdays), options_must_mention_any and options_must_not_mention_any.
• A case with answers (a FollowUp) is checked with FollowUpSchema before any model call. The first call must ask. Then the answers go through buildFollowUpMessages, as in the route, and the draft is scored.
  • Only the asked topics are sent: when only if the first call asked "when", howMany only if it asked "how_many".
  • A failed second call is reported as "second call: …", with the first call's checks still shown.
• answers is a new key because followUps already means the MCP runner's organizer turns.
• scripts/lib/eval-harness.test.ts tests the harness's own logic. For it to run, vitest.config.ts now includes scripts/**/*.test.ts.

pnpm eval:mcp loads the same file:

• On a case where Magic Compose asks, an assistant that asks instead of creating passes as asked_before_creating, but only with a non-empty reply containing a "?". A refusal or an empty reply fails, and a create fails.
• answers cases are skipped.

Cases. #526 added ev_027 (the classroom starter), so this PR's new cases are ev_028–ev_037, and ev_008 flips from "drafts 1–6 slots" to "asks".

id	prompt	expected
ev_028	"Bake sale"	asks
ev_029	"Class picnic"	asks
ev_030	"Team party on Saturday, June 5"	asks, but not about the date
ev_031	the French prompt	asks, with options in French
ev_032	a dish list	drafts
ev_033	a full potluck	drafts
ev_034	the design's snack duty	drafts "TBD:" rows
ev_035	conferences	drafts "TBD:" rows
ev_036	drafts from answers	those items, that date and those counts, with null for no limit
ev_037	Draft with placeholders	drafts "TBD:" rows

Quota

magicComposePerOrganizer goes from 10 to 12 an hour. A thin description now takes two AI calls: the first, and the one with the answers. The free retry doesn't count. No docs or pages named the old number.

Privacy and changelog

The privacy page already says the answers go to the AI provider (#573). CHANGELOG has a line under Added.

Your two open questions

I took the issue's defaults:

• a missing date alone doesn't ask;
• the hourly limit goes to 12.

The evals haven't run against a real model

This session has no LLM endpoint. Against a local fake OpenAI-compatible endpoint:

• every case runs (37, or 11 with --tag=questions);
• with the harness fixes, ev_001–ev_026 score line for line as before.

Those scores say nothing about the prompt itself. Before merging, please run, on this branch and on main, and compare:

pnpm eval:magic-compose
pnpm eval:magic-compose --tag=questions
pnpm eval:mcp

Watch:

• ev_037 (Draft with placeholders): the prompt's rule 4 now exempts a message that carries answers, so it shouldn't ask again;
• ev_030 (it mustn't ask for the date);
• ev_031 (French options).

Reviews

• The first review, fixed in e4b58cd:
  • rule 4 had no exception for a message carrying answers;
  • Example D's prompt was word for word an eval case's.
• A second review (/code-review), fixed in 7b78492. It found eight harness issues:
  • a partial re-ask was scored as a parse failure;
  • must_ask: false meant the opposite;
  • the MCP runner passed any reply on an ask-case;
  • French counts got through;
  • bad case answers were checked only after a paid call;
  • unasked topics were still sent;
  • a dead branch;
  • a second-call failure hid the first call's checks.
• Left as is: the eval doesn't repeat the route's firmer retry. That's deliberate: it measures whether the first reply to the answers drafts.
• Formatting: Prettier also re-wrapped some existing lines in eval-harness.ts, eval-mcp.ts and the README, files this PR already rewrites.

Tophat

On this branch alone, against the fake endpoint:

• the server sends the merged system prompt, with both the QUESTIONS and CAPACITY sections;
• the full flow (description → questions → answers → draft in the builder) works with no console errors;
• the 12th draft in an hour goes through and the 13th gets the friendly limit message.

Tests

pnpm lint && pnpm typecheck && pnpm knip && pnpm test (1488) and pnpm test:db (569) pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UW9ZpEu4MWkeqTv2Qe14ks