Ask a few questions when a Magic Compose description doesn't say what people sign up for

CHANGELOG.md

@@ -5,6 +5,7 @@ All notable changes to OpenSignup are documented here. Format follows [Keep a Ch
 ## [Unreleased]
 
 ### Added
+- Magic Compose asks a few questions when a description doesn't say what people sign up for (#570). A description that is only an event's name, like "Bake sale", used to get three "TBD" rows or made-up jobs. Now the AI answers with the topics the description leaves out (what, when, how many) and two to four suggestions for what people could sign up for, in the description's language; it never suggests dates or counts, which the page asks for with its own controls. The organizer's answers go back with the description, and that second call always drafts or refuses, with "TBD:" rows for anything left blank and for **Draft with placeholders**. A description that says what but not when or how many still drafts straight away, with "TBD:" rows where the slots need dates or times, and a list of items, roles or tasks, or a schedule of dated practices or shifts, counts as saying what. A first reply with questions the page can't show (no "what", or one usable option) is tried once more on the same quota, and extra options are cut to four. A slot gets no limit only when the description or the answers say there's none. Both calls count toward the hour's AI drafts, so the limit goes from 10 to 12 an hour. The privacy page now says the answers go to the AI provider along with the description. `pnpm eval:magic-compose` scores questions: a case can expect them (`questions.*` checks), or give `answers` to send after a first call that must ask, and the draft made from them is scored. `ev_008`, the vague team prompt, now expects questions instead of 1-6 placeholder slots (and is about a swim club, not the soccer team of the prompt's own example, so it scores the rule rather than recall), and ten new made-up cases cover prompts that must ask, near misses that must still draft, and drafts from answers. `pnpm eval:mcp` passes an assistant that asks the organizer on those cases and skips the ones with answers.
 - Activity events for Magic Compose's questions (#570), so drafts that stop there can be counted. `magic_compose.asked` (`{ topics, options }`) is written when a first draft comes back with questions instead, and one `magic_compose.answered` for each call that carried the organizer's answers, however it ended, with `{ answered, skipped, picked, added, limits, placeholders, retried, gaveUp }`: which of "what" and "when" were answered or left blank, how many of the AI's options were picked and how many things of their own were added (as the browser reports them), how many were given a number of people, whether Draft with placeholders was pressed with nothing answered, and whether the AI asked again and was retried or given up on. Both are written outside any transaction, with no signup. Payloads hold codes and counts only, never the description, the options or the answers. `docs/telemetry.md` lists them, and the privacy page says the log records them.
 - After **Publish and share**, a panel helps the organizer send the link, in place of the "Signup published" banner (#505). It says "Published" and "Now send people the link", and offers **Copy link**; a message to send with it, built from the signup's own data, such as "Fall book fair volunteers / Thu, Nov 12 to Sat, Nov 14 · 21 spots open / Sign up here, no account needed: <link>", which the organizer can change and which stays changed while the page is open; **Copy message** and **Email it**, a `mailto:` link with the title as its subject and the message as its body, less its first line when that is the title; and a QR code to **Download** as a PNG for a flyer or a sign-up table, with the signup's title under the code in up to two lines, shown only where the browser can draw it. The message's dates come from the date field reminders use. Days in a row read as a range, "Thu, Nov 12 to Sat, Nov 14"; two or three days with a gap are named, "Sat, Nov 7 and Sat, Nov 14" or "Sat, Nov 7, Sat, Nov 14 and Sat, Nov 21"; four or more read as a range again. When every dated slot has the same value in the time field reminders pair with that date, the message gives it: "Sat, Nov 21, 11:00 AM · 19 spots open" for one day, "Thu, Nov 12 to Sat, Nov 14 · 11:00 AM · 21 spots open" for several. The days and the time come from the slots people can still take, as the spots do (open, with a spot left or no limit), so a closed or full slot adds no day and hides no time; when none has room they come from the open slots, and when none is open from every slot. It counts spots, not slots, since readers don't know the organizer's word "slot": "21 spots open" before anyone signs up, "18 spots left" after, and no count when an open slot has no limit or no spot is left. A signup with no dates and no limits gets only its title and the link. The description stays out, since a link preview shows it. On a phone that has a share menu, **Share…** comes first and opens it with the message and link, in place of Copy message and Email it. On a phone, **QR code** shows the code large in the sheet, under the Copy link row, and then looks pressed and says **Hide QR code**, since the code pushes the message below the screen; **Save image** is under the code: where the phone can share a file it opens the share menu with the PNG, so it can go to Photos, and the menu says where it went; elsewhere, or if the phone refuses, it downloads and says "Saved". The message box is labelled "A message to send with it" on a phone and a computer, and on a computer grows with the message up to 12 rows. Copy message keeps its width while it says "Copied", so Email it doesn't move. On a computer the panel opens under the header's buttons; on a phone it is a sheet. An open signup's header now has a **Share** button that opens the panel again, headed "Share the link": on a computer in the blue spot right of **Preview**, before ⋯, and on a phone in the bar at the bottom of the screen. Publish now stays on the tab it was pressed on instead of going to Build with `?published=1`, so a reload or a shared address never opens the panel again. Escape or the panel's ✕ closes it and puts focus back on Share, as does a tap outside the sheet on a phone; on a computer a click outside closes it and leaves focus where it landed. Each copy is announced, and a copy the clipboard refuses (a site served over plain HTTP, a denied permission) says "Couldn’t copy. Select the link and copy it." and selects the link. When Publish on this page has published the signup, screen readers hear "Published. Anyone with the link can sign up now."; a signup published elsewhere, such as by an AI assistant, doesn't open the panel or say so. The QR code is drawn in the browser by `lean-qr` 2.7.4 (MIT, no dependencies); it and the rest of the panel load once a draft's or an open signup's page is idle (in Safari, which can't say when, 200 ms after it's shown), so no page's first load carries them and the panel usually opens at its full size, not on a spinner that then grows; a load that fails is tried again when the panel opens, and if that fails too the panel still shows the public link with its copy button, and "Couldn’t load the other ways to share. Reload to try again." Email it opens the organizer's own email app, and Share… the phone's own menu; the QR code goes to no outside service. Copies and shares from the panel are recorded through the link-copy beacon from #525: Copy link and Copy message as `signup.link_copied` with the surfaces `share_panel` and `share_message` (the fallback's chip counts as `share_panel`), and, as a new `signup.link_shared` event (`share_menu`, `email`, `qr_code`), Share… once the menu says it sent, Email it on its click, and the QR code once each time the panel opens, the first time a phone shows it large or it's first downloaded or saved; a refused copy or a share menu closed without sending records nothing. Each row holds only the surface and the signup's status: never the message, and nothing about who the link went to. `POST /api/signups/[id]/link-events` takes `event=shared` and refuses a surface that isn't its event's, `docs/telemetry.md` lists the new event and surfaces, and the privacy page says the log records these. The help page "Create and publish your first signup" says what the panel offers.
 - The AI connector can reopen a closed signup, with a new `reopen_signup` tool (#20). It calls the same `reopenSignup` as `POST /api/signups/[id]/reopen`, and its result shows `closesAt` as null when a closing time that had passed was removed. `close_signup` no longer says closing can't be undone, and the help page "Connect an AI assistant" lists reopening among what an assistant can do. The assistant isn't told to ask before reopening, as it is before closing: a reopen only follows the organizer asking for one, and closing undoes it.

evals/magic-compose/README.md

@@ -4,7 +4,7 @@ Developer evals for OpenSignup's Magic Compose feature (natural-language to sign
 
 ## Files
 
-- `dev-evals.json` — 27-case developer eval suite, shared by the Magic Compose and MCP runners, covering each `fieldType`, cross-product/groupBy correctness, capacity instead of duplicate rows, edge cases, multilingual input, hostile prompts, and the server invariants that hold regardless of model output.
+- `dev-evals.json` — 37-case developer eval suite, shared by the Magic Compose and MCP runners, covering each `fieldType`, cross-product/groupBy correctness, capacity instead of duplicate rows, edge cases, multilingual input, hostile prompts, when Magic Compose asks questions instead of drafting, and the server invariants that hold regardless of model output.
 
 The user-facing starter prompts (School, Sports, Volunteering) shipped in the UI as click-to-try chips live in the Magic Compose UI code, not as a JSON file in this directory.
 
@@ -33,21 +33,28 @@ These evals are written against the shape Magic Compose actually produces — no
 
 Value formats per `fieldType`:
 
-| fieldType | shape in `slot.values[ref]` |
-|-----------|-----------------------------|
-| `date`    | `"YYYY-MM-DD"`              |
-| `time`    | `"HH:MM"` (24h)             |
-| `number`  | finite number               |
+| fieldType | shape in `slot.values[ref]`  |
+| --------- | ---------------------------- |
+| `date`    | `"YYYY-MM-DD"`               |
+| `time`    | `"HH:MM"` (24h)              |
+| `number`  | finite number                |
 | `enum`    | one of the field's `choices` |
-| `text`    | trimmed string, ≤200 chars  |
+| `text`    | trimmed string, ≤200 chars   |
 
 Limits: max 20 fields, max 200 slots.
 
+When the prompt doesn't say what people sign up for, the reply is questions instead of a draft (`QuestionsSchema` in `prompt.ts`): the topics the prompt leaves out, always including `what`, and two to four options for "what". The page asks in its own words and sends the answers back with the prompt (`buildFollowUpMessages` in `follow-up.ts`); that second call drafts or refuses, never asks.
+
+```ts
+{ questions: { topics: Array<'what' | 'when' | 'how_many'>, whatOptions: string[] } }
+```
+
 ## Core principles (load-bearing)
 
-- **Slots are the atom, not questions.** The generator drafts the *things participants sign up for*, not free-form questions. No `essay`, `favourite-colour`, `goals` text fields.
+- **Slots are the atom, not questions.** The generator drafts the _things participants sign up for_, not free-form questions. No `essay`, `favourite-colour`, `goals` text fields.
 - **No PII fields.** Never `ssn`, `dob`, `home-address`, `driver-license`, `bank-account`, etc., even on request. The prompt-level refusal pattern returns a structured refusal payload `{ refusalReason: string }` — no title, fields, or slots — which the server surfaces to the user without persisting anything.
 - **Don't invent specifics.** No dates, locations, capacities, or opponents the prompt doesn't mention.
+- **Ask only when "what" is missing.** A prompt that doesn't say what people sign up for ("Bake sale") gets questions, and only about what it leaves out. One that says what but not when or how many still drafts, with "TBD:" rows where the slots need dates or times.
 - **Cross-product slots.** A grid like "3 classes × 12 times" is 36 slots, each with a non-empty `values` object — not 3+12 slots and not 1 slot.
 - **groupBy is the scan axis.** With a small named dimension (class, station, teacher), set `groupBy` to that field's ref. For a pure date list, set `groupBy: null` (the converter also falls back to the sole enum field if the model omits it).
 
@@ -76,6 +83,23 @@ Expectations are structural, not literal-text:
 - `*.must_not_match_any` — case-insensitive substring; any match fails.
 - `slots.count_range: [min, max]` — inclusive integer bounds.
 - `must_express_target_via_one_of` — accepts either `capacity` or a dedicated number field. The generator gets to choose; the assertion accepts both.
+- `slots.capacity_for_label` — `capacity: null` asks for no limit.
+
+### Questions
+
+A case expects a questions reply when it says `"questions.must_ask": true`, or, without that key, when it has any other `questions.*` key. `"questions.must_ask": false` expects a draft, whatever other keys the case has. A draft where questions are expected fails as `expected_questions_but_drafted`; questions where they aren't fail as `unexpected_questions`, as a refusal fails as `unexpected_refusal`. The keys:
+
+- `questions.must_ask: true` — the reply is questions. `false` — it isn't.
+- `questions.topics_must_include` / `questions.topics_must_not_include` / `questions.topics_must_equal` — the topics asked (`what`, `when`, `how_many`); `must_equal` ignores order.
+- `questions.options_count_range: [min, max]` — how many options for "what".
+- `questions.options_must_not_contain_date_or_number: true` — no option holds a digit, or a month, a weekday or a number word in English or French, accents ignored (the app asks for dates and counts itself). Not the French "un" and "une", which are also articles.
+- `questions.options_must_mention_any` / `questions.options_must_not_mention_any` — whole words, case and accents ignored, in some option / in no option. The French case uses both to check the options are in French and about its bake sale: articles, verbs that fit any signup and words that are English too don't count.
+
+### Answers
+
+A case with `answers` (a `FollowUp`, as the page sends it: `what.picked`, `what.added`, `when`, `howMany`, `placeholders`) runs both calls. The answers are checked with `FollowUpSchema`, as the route checks them, before either call: a case whose answers fail it is reported as a broken case, and nothing is sent. The first call gets the prompt alone and must ask; its checks are reported as `first:…`, with the case's `questions.*` keys scored there (so `questions.must_ask` can only be `true`). Then the prompt goes back with the answers, through `buildFollowUpMessages` as in the route, and every other key scores that reply. As on the page, `what` always goes back, `when` only if the first call asked about `when`, and `howMany` only if it asked about `how_many`. When the first call didn't ask (it drafted or refused), the answers go back as written, since they test the second prompt on its own. A reply that asks again, in any shape (`asksQuestions`, as the route reads it, so also a partial one such as `{"questions":{"topics":["when"]}}`), fails as `asked_again`. The route retries once with a firmer line when the model asks again; the eval doesn't, so any `asked_again` is worth a look. When the second call fails, its error starts `second call:` and the `first:` checks are still printed. `answers` is not `followUps`, which the MCP runner uses for organizer messages.
+
+Run only the cases about asking with `pnpm eval:magic-compose --tag=questions`.
 
 ## Running
 
@@ -98,7 +122,10 @@ Per-run JSON snapshots are written to `evals/magic-compose/snapshots/` for local
 - The model gets a system message with today's date and the server instructions (`src/mcp/instructions.ts`), every tool as `tools/list` describes it, and the case's prompt as the organizer's one message.
 - The arguments of its `create_signup` call are the draft that gets scored. Arguments `create_signup` would refuse go back to the model as the same `invalid_input` error the server sends, so it can retry; the snapshot's `trace.rejectedCreates` counts them.
 - `list_workspaces` and `list_signups` answer as a new account with one workspace and no signups; any other tool says nothing matches. Nothing touches a database.
-- A reply with no `create_signup` call is scored as the refusal arm, so `ev_013` passes when the assistant declines. On any other case it fails as `unexpected_refusal`, with the reply as its detail: an assistant that asks a question first did not build anything.
+- A reply with no `create_signup` call is scored as the refusal arm, so `ev_013` passes when the assistant declines. On most other cases it fails as `unexpected_refusal`, with the reply as its detail: an assistant that asks a question first did not build anything.
+- Where Magic Compose would ask (a case that expects questions, as above), asking the organizer is right for an assistant too, so a reply that asks passes as `asked_before_creating` and a create fails as `expected_questions_but_drafted`. A reply counts as asking when it holds a `?` or asks for details in so many words ("Let me know the dates", "Please share…", "I'll need the dates"; `askedBeforeCreating` in `scripts/lib/eval-harness.ts`), a deliberately simple heuristic: an empty reply, or one that asks for nothing (a refusal, say, even one ending "Let me know if there's anything else"), fails `asked_before_creating` with a detail saying which. A case can also say `"assistant_may_ask": true` where Magic Compose drafts "TBD:" rows but an assistant, told to ask for what the slots need, may ask for the dates instead (`ev_034`, `ev_035`).
+- Cases with `answers` are skipped: they answer Magic Compose's own questions, which an assistant never asks.
+
 ### Follow-up cases
 
 `evals/mcp/followup-evals.json` holds cases where the organizer says more after the first message. `prompt` is the first message and `followUps` lists the ones after it, each sent once the model has replied in words to the one before. Run only these with `pnpm eval:mcp --tag=follow_up`.
@@ -116,6 +143,7 @@ Snapshots go to `evals/mcp/snapshots/` (gitignored).
 When adding a case:
 
 - Write the prompt the way a real organizer would type it. Slightly run-on sentences and parentheticals are fine; clean marketing prose makes the generator look better than it is.
+- Don't reuse a worked example's prompt from `prompt.ts`: a case that matches one scores recall of the example, not the rules. `prompt.test.ts` fails when a case's prompt is, or holds, an example's.
 - Cover at least one `fieldType`, edge case, or principle that existing cases don't.
 - Prefer machine-checkable expectations (`slots.count`, `fields.field_types_subset_of`, regex patterns) over prose. A `groupBy_should_be_null` boolean is easier to assert than "the grouping should be reasonable".
 - Include the regression context in `rationale` when the case targets a specific past bug.
@@ -128,4 +156,5 @@ When adding a case:
 - Empty prompt rejected with `invalid_input` before LLM call (`ev_014`).
 - No PII fields, no tenant IDs in model output (`ev_012`, `ev_020`).
 - Refusal pattern on hostile prompts (`ev_013`).
+- Questions, not placeholder rows, for a prompt that doesn't say what people sign up for (`ev_008`, which drafted 1-6 slots until #570, and `ev_028`–`ev_031`); drafts for near misses (`ev_032`–`ev_035`); drafts from the answers (`ev_036`, `ev_037`).
 - Several people for the same thing is one slot with a capacity, not a row per person: shifts that each need several volunteers, several parents per game, and the "Classroom or school" starter prompt word for word, whose potluck is one slot per dish type (`ev_005`, `ev_006`, `ev_026`, `ev_027`). `ev_025` is the counter-case, one parent per game at capacity 1. Run them together with `--tag=capacity_not_duplicate_rows`.