Ask when, not what, when every Magic Compose slot would be a TBD row

CHANGELOG.md

@@ -33,6 +33,7 @@ All notable changes to OpenSignup are documented here. Format follows [Keep a Ch
 - Organizers can let participants sign up without an email (#378). The signup's `requireEmail` setting, on by default, is set on the Settings tab, in a new "Sign-up form" section: "Ask for an email address", Required or Optional, then Save. The AI connector's `update_signup` tool and `PATCH /api/signups/[id]` can set it too. A line under the choices says what the one picked means, and changes as soon as the radio does: with Required, everyone is emailed their link to change or cancel; with Optional, people who leave it blank get no emails, not even reminders, so they need to save their link, and someone who loses it can sign up again while the organizer removes the old one on the Responses tab. A viewer is refused with "You don’t have edit access.", the words the Build tab now uses too in place of "You no longer have edit access.", and a refused Delete on the same tab is put in the same plain words instead of the service's own message. With it off, the sign-up form's Email box says "(optional)" and starts blank, even for someone this browser remembers. Its help says what an email brings (the link, and a reminder only where that slot will get one) and to save the link without one. Someone who leaves it blank gets no confirmation or reminder emails. Their link to change or cancel is not emailed: the confirmation screen shows it, says to save it and offers Copy link in place of Share link, which on a phone invited them to post it, and the `os_commit` cookie keeps it in that browser. If the organizer turns it back on after the page loaded, the refused blank email turns the box required. In `POST /api/slots/[id]/commitments`, a blank, whitespace-only, null or missing `email` now means none: a signup that requires one refuses it with `invalid_input` on `email`, and one that doesn't always adds a new participant, as there is nothing to find them by. Turning the setting back on asks only new sign-ups: anyone already signed up without an email keeps their spot and can still change or move it. `get_signup` and the other connector tools now show `requireEmail`; `update_signup` advertises no default for any setting, so a client that fills in defaults cannot turn it back on with an unrelated change. The privacy and cookies pages and the help pages say what happens without an email, and the help page "Create and publish your first signup" explains the setting and what to do when someone loses their link. The "Sorry, this link isn’t working" page tells everyone to contact the organizer, and to try their confirmation email's link only if they got one.
 
 ### Changed
+- Magic Compose asks when, instead of drafting nothing but "TBD:" rows, for a description that says what people sign up for but gives nothing to make a real slot from (#687). "Snack sign up" used to get three "TBD: Event" rows; now it gets the questions screen, without "What do people sign up for?". Its first section is headed "What people sign up for", with what the AI read from the description ("Bring snacks") already picked, to remove or add to, and "How many" works from those as before. The AI answers with the topics left open, `when` always among them, and one to four things it read; a reply that asks neither what nor when is still tried once more. A description that lists items or roles still drafts straight away, with or without dates, and **Draft with placeholders** and "Not decided" still make "TBD:" rows. A day of the week counts as when, so "Saturday cleanup crew" still drafts one slot. `pnpm eval:magic-compose`: `ev_034` (snack duty with no dates) and `ev_035` (conferences with no times) now expect questions about when instead of "TBD:" rows, and two new made-up cases, `ev_040` ("Snack sign up") and `ev_041`, cover a name or a role with no dates. `magic_compose.asked` can now list `when` without `what`, with the number of things read rather than options offered.
 - **Publish and share** no longer stops for a slot that says TBD, TBC or TBA (#670). Organizers often mean it ("Cashier TBD", "Location TBA"), so the "Before people see it" question, with **Name it first** and **Publish anyway**, is gone, and the signup publishes straight away. Publish still stops when there's nothing to sign up for: no slots, no fields, or every slot empty. The Build tab no longer marks a slot that says TBD either: the "Still says “TBD”" and "People can see “TBD”" tags, the row's warm tint and the tag's words in the row's accessible name are gone, for drafts and published signups alike, and the help page no longer mentions them. New `signup.published` and `signup.publish_refused` activity rows no longer carry `acknowledged: 'placeholder'` or `{ reason: 'placeholder', warning: true }`; older rows still can (`docs/telemetry.md`).
 - Times are picked from a list instead of typed into the browser's time box (#686). On a computer that box is filled in segment by segment (hour, minutes, AM/PM) and doesn't take "9am". The Build tab's Time cell and the "One day" times on Magic Compose's questions screen now use one `TimeSelect`: a list of times every 15 minutes, read on the 12-hour clock as on the public page, with "No time" first. Typing "9" jumps to 9:00 AM, and phones keep their own picker. A time that isn't on the 15-minute grid (09:10, say, from an AI assistant) is listed as itself, so opening a slot never changes it. On the questions screen the two times are now labelled **Starts** and **Ends**; Ends lists only times after the start, each with its length ("1:00 PM (4 hr)"), and clears when the start is moved to or past it. Times are still stored as 24-hour HH:MM, so nothing changes on the server, in emails or in reminders.
 - The help page "Close or reopen a signup" now mentions pausing sign-ups in its summary on `/help`, its description and its first paragraph (#653). Organizers looking for a way to pause a signup searched for "pause" and found nothing, because the page only said "close". The page's title, link and the **Close signup** button are unchanged.

docs/telemetry.md

@@ -97,15 +97,21 @@ Signup lifecycle above. These two count the questions round.
 
 | event | actor | payload | fired from |
 |---|---|---|---|
-| `magic_compose.asked` | organizer | `{ topics, options }`: the topics the model asked about (`what`, `when`, `how_many`; always `what`) and how many options it offered for "what" (2 to 4), never the options themselves | `POST /api/signups/magic-compose`, on a first call the model answered with questions instead of a draft, unless the organizer had already left |
+| `magic_compose.asked` | organizer | `{ topics, options }`: the topics the model asked about (`what`, `when`, `how_many`; `what` or `when` or both) and how many options it offered for "what" (2 to 4), or, without `what`, how many things it read from the description (1 to 4), which the page shows already picked; never the options themselves | `POST /api/signups/magic-compose`, on a first call the model answered with questions instead of a draft, unless the organizer had already left |
 | `magic_compose.answered` | organizer | `{ answered, skipped, picked, added, limits, placeholders, retried, gaveUp }`, described below | `POST /api/signups/magic-compose`, once a follow-up (a call carrying the organizer's answers) has ended, however it ended: a draft, a refusal, an error or `asked_again`. One row per follow-up the hourly quota was charged for |
 
 In `magic_compose.answered`:
 
 - `answered` and `skipped` list which of `what` and `when` the organizer
-  answered or left blank, for the ones they were asked. "How many" has no
-  blank state (an empty box means no limit), so it is in neither.
-- `picked` is how many of the model's options the organizer picked, and
+  answered or left blank. `when` is there only if it was asked. `what` is
+  always there: when it wasn't asked, it is in `answered` if any of the
+  things read from the description (or the organizer's own) went back, and
+  in `skipped` if all were removed, so a count of `what` answers has to be
+  read alongside `magic_compose.asked`'s topics. "How many" has no blank
+  state (an empty box means no limit), so it is in neither.
+- `picked` is how many of the model's options the organizer picked (when
+  "what" wasn't asked, how many of the things read from the description they
+  kept), and
   `added` how many things of their own they added. These are the browser's
   claim: it sends the two lists apart, and the server can't check which items
   came from the options.

evals/magic-compose/README.md

@@ -43,7 +43,7 @@ Value formats per `fieldType`:
 
 Limits: max 20 fields, max 200 slots.
 
-When the prompt doesn't say what people sign up for, the reply is questions instead of a draft (`QuestionsSchema` in `prompt.ts`): the topics the prompt leaves out, always including `what`, and two to four options for "what". The page asks in its own words and sends the answers back with the prompt (`buildFollowUpMessages` in `follow-up.ts`); that second call drafts or refuses, never asks.
+When the prompt doesn't say what people sign up for, the reply is questions instead of a draft (`QuestionsSchema` in `prompt.ts`): the topics the prompt leaves out, including `what`, and two to four options for "what". When it says what but every slot would be a "TBD:" placeholder ("Snack sign up"), the reply asks `when` without `what`, and `whatOptions` holds one to four things read from the prompt, which the page shows already picked. The page asks in its own words and sends the answers back with the prompt (`buildFollowUpMessages` in `follow-up.ts`); that second call drafts or refuses, never asks.
 
 ```ts
 { questions: { topics: Array<'what' | 'when' | 'how_many'>, whatOptions: string[] } }
@@ -123,7 +123,7 @@ Per-run JSON snapshots are written to `evals/magic-compose/snapshots/` for local
 - The arguments of its `create_signup` call are the draft that gets scored. Arguments `create_signup` would refuse go back to the model as the same `invalid_input` error the server sends, so it can retry; the snapshot's `trace.rejectedCreates` counts them.
 - `list_workspaces` and `list_signups` answer as a new account with one workspace and no signups; any other tool says nothing matches. Nothing touches a database.
 - A reply with no `create_signup` call is scored as the refusal arm, so `ev_013` passes when the assistant declines. On most other cases it fails as `unexpected_refusal`, with the reply as its detail: an assistant that asks a question first did not build anything.
-- Where Magic Compose would ask (a case that expects questions, as above), asking the organizer is right for an assistant too, so a reply that asks passes as `asked_before_creating` and a create fails as `expected_questions_but_drafted`. A reply counts as asking when it holds a `?` or asks for details in so many words ("Let me know the dates", "Please share…", "I'll need the dates"; `askedBeforeCreating` in `scripts/lib/eval-harness.ts`), a deliberately simple heuristic: an empty reply, or one that asks for nothing (a refusal, say, even one ending "Let me know if there's anything else"), fails `asked_before_creating` with a detail saying which. A case can also say `"assistant_may_ask": true` where Magic Compose drafts "TBD:" rows but an assistant, told to ask for what the slots need, may ask for the dates instead (`ev_034`, `ev_035`).
+- Where Magic Compose would ask (a case that expects questions, as above), asking the organizer is right for an assistant too, so a reply that asks passes as `asked_before_creating` and a create fails as `expected_questions_but_drafted`. A reply counts as asking when it holds a `?` or asks for details in so many words ("Let me know the dates", "Please share…", "I'll need the dates"; `askedBeforeCreating` in `scripts/lib/eval-harness.ts`), a deliberately simple heuristic: an empty reply, or one that asks for nothing (a refusal, say, even one ending "Let me know if there's anything else"), fails `asked_before_creating` with a detail saying which. A case can also say `"assistant_may_ask": true` where Magic Compose drafts "TBD:" rows but an assistant, told to ask for what the slots need, may ask for the dates instead.
 - Cases with `answers` are skipped: they answer Magic Compose's own questions, which an assistant never asks.
 
 ### Follow-up cases
@@ -156,5 +156,5 @@ When adding a case:
 - Empty prompt rejected with `invalid_input` before LLM call (`ev_014`).
 - No PII fields, no tenant IDs in model output (`ev_012`, `ev_020`).
 - Refusal pattern on hostile prompts (`ev_013`).
-- Questions, not placeholder rows, for a prompt that doesn't say what people sign up for (`ev_008`, which drafted 1-6 slots until #570, and `ev_028`–`ev_031`); drafts for near misses (`ev_032`–`ev_035`); drafts from the answers (`ev_036`, `ev_037`).
+- Questions, not placeholder rows, for a prompt that doesn't say what people sign up for (`ev_008`, which drafted 1-6 slots until #570, and `ev_028`–`ev_031`); drafts for near misses (`ev_032`, `ev_033`); questions about when, without "what", for a prompt that says what but whose every slot would be a "TBD:" placeholder (`ev_034`, `ev_035`, which drafted those rows until #687, and `ev_040`, `ev_041`); drafts from the answers (`ev_036`, `ev_037`).
 - Several people for the same thing is one slot with a capacity, not a row per person: shifts that each need several volunteers, several parents per game, and the "Classroom or school" starter prompt word for word, whose potluck is one slot per dish type (`ev_005`, `ev_006`, `ev_026`, `ev_027`). `ev_025` is the counter-case, one parent per game at capacity 1. Run them together with `--tag=capacity_not_duplicate_rows`.

evals/magic-compose/dev-evals.json

@@ -576,30 +576,33 @@
       }
     },
     {
-      "id": "ev_034_what_without_when_drafts_placeholders",
+      "id": "ev_034_what_without_when_asks_when",
       "tags": ["questions", "near_miss", "no_invention", "edge:capacity_gt_1"],
       "runner": "llm",
       "prompt": "Snack duty for the basketball team, two families per game",
       "expect": {
-        "slots.count_range": [1, 3],
-        "slots.all_capacities_equal": 2,
-        "slot_values_must_cover_all": ["tbd"],
-        "slot_dates.set_must_equal": [],
-        "assistant_may_ask": true,
-        "rationale": "#570: it says what but not when, so it still drafts \"TBD: game 1\"… with 2 spots each, and invents no dates. An assistant may ask for the dates instead, as its own instructions say."
+        "questions.must_ask": true,
+        "questions.topics_must_include": ["when"],
+        "questions.topics_must_not_include": ["what", "how_many"],
+        "questions.options_count_range": [1, 4],
+        "questions.options_must_not_contain_date_or_number": true,
+        "questions.options_must_mention_any": ["snack", "snacks"],
+        "rationale": "#687: it says what (snacks) and how many (two families) but not when, and nothing else in it makes a real slot, so every slot would be \"TBD: game 1\". It asks when, with the snacks read from the prompt as the list, rather than drafting placeholder rows. Until #687 it drafted 1-3 \"TBD:\" rows with 2 spots each."
       }
     },
     {
-      "id": "ev_035_conference_slots_without_times_drafts_placeholders",
+      "id": "ev_035_conference_slots_without_times_asks_when",
       "tags": ["questions", "near_miss", "no_invention"],
       "runner": "llm",
       "prompt": "Parent-teacher conferences, 15-minute slots",
       "expect": {
-        "slots.count_range": [1, 3],
-        "slot_values_must_cover_all": ["tbd"],
-        "slot_dates.set_must_equal": [],
-        "assistant_may_ask": true,
-        "rationale": "#570: it says what but not when, so it drafts \"TBD:\" times rather than asking or inventing a schedule."
+        "questions.must_ask": true,
+        "questions.topics_must_include": ["when"],
+        "questions.topics_must_not_include": ["what"],
+        "questions.options_count_range": [1, 4],
+        "questions.options_must_not_contain_date_or_number": true,
+        "questions.options_must_mention_any": ["conference", "conferences", "meeting", "meet"],
+        "rationale": "#687: it says what (a conference slot) but gives no day or times, so every slot would be a \"TBD:\" time. It asks when, not what. Until #687 it drafted 1-3 \"TBD:\" rows."
       }
     },
     {
@@ -671,6 +674,43 @@
         "slot_values_must_cover_all": ["salad", "burger", "dessert", "lemonade"],
         "rationale": "#685: an item list for one day gets a date field with that date in every slot, not the date in the description only."
       }
+    },
+    {
+      "id": "ev_040_what_only_name_asks_when",
+      "tags": ["questions", "edge:vague", "no_invention"],
+      "runner": "llm",
+      "prompt": "Snack sign up",
+      "expect": {
+        "questions.must_ask": true,
+        "questions.topics_must_include": ["when", "how_many"],
+        "questions.topics_must_not_include": ["what"],
+        "questions.options_count_range": [1, 4],
+        "questions.options_must_not_contain_date_or_number": true,
+        "questions.options_must_mention_any": ["snack", "snacks"],
+        "rationale": "#687: a name that says what (snacks) and nothing else. Every slot would be a \"TBD:\" row, so it asks when and how many, with the snacks read from the prompt as the list, not options to pick. Until #687 it drafted three \"TBD: Event\" rows. Numbered from ev_040 so it can't collide with the ev_038 and ev_039 that #693 adds."
+      }
+    },
+    {
+      "id": "ev_041_role_without_when_asks_when",
+      "tags": ["questions", "no_invention"],
+      "runner": "llm",
+      "prompt": "Need drivers for the choir's away concerts",
+      "expect": {
+        "questions.must_ask": true,
+        "questions.topics_must_include": ["when"],
+        "questions.topics_must_not_include": ["what"],
+        "questions.options_count_range": [1, 4],
+        "questions.options_must_not_contain_date_or_number": true,
+        "questions.options_must_mention_any": [
+          "drive",
+          "driver",
+          "drivers",
+          "driving",
+          "carpool",
+          "ride"
+        ],
+        "rationale": "#687: a role (drivers) with no concert dates. Every slot would be \"TBD: concert 1\", so it asks when, reading the role from the prompt, rather than drafting placeholders or asking what."
+      }
     }
   ]
 }