[richshaw]

Fixes #687.

The problem

Since #577, Magic Compose asks questions only when a description doesn't say what people sign up for. A description that says what but nothing else, like "Snack sign up", went straight to a draft: rule 4 told the model to make 1–3 "TBD:" rows, so the organizer landed in the builder with three "TBD: Event" rows and nothing usable. QuestionsSchema also required what and 2–4 options, so a model that tried to ask about when only (several did in #577's evals) got "AI draft did not match expected shape".

The fix

• Rule 4 (NEVER_INVENT_MIDDLE.drafter in src/lib/signup-rules.ts): when the prompt says what but its slots need dates or times it doesn't give and nothing in it could make a real slot, so every slot would be a "TBD:" placeholder, ask about "when", not "what". A prompt that lists items or roles drafts, with or without dates. Placeholder rows are now only for answers that leave the dates out ("Not decided", Draft with placeholders). The old "named items … already say what: draft, don't ask" now reads "…: don't ask about it", since a named task like snack duty can now lead to asking when. The assistant's half of the rule and RULES_IN_BOTH are unchanged, so the MCP instructions are unchanged.
• The QUESTIONS section of the prompt: topics includes "what" when the prompt doesn't say it. Otherwise it leaves "what" out and includes "when". A new line says that without "what", whatOptions holds 1–4 things read from the prompt.
• Example D (school play ushers, two parents per performance) flips from three "TBD: performance" rows to {"questions":{"topics":["when"],"whatOptions":["Usher at the play"]}}. The golden prompt is regenerated, and its diff is only these changes.
• QuestionsSchema: topics must include what or when. whatOptions needs 2 or more when what is asked and 1 or more when it isn't. Both levels stay .strict(). Unknown topics, blanks, duplicates and over-long options are dropped as before, and a list over four is cut to four, as it already is when what is asked. A reply that asks neither what nor when still gets the route's one firmer retry.
• The questions screen (Questions.tsx, answers.ts): when what wasn't asked, the screen starts with the model's items picked (startingAnswers). They're shown as removable chips under What people sign up for, with "From your description. Remove any that don't fit, or add your own." There are no option pills. Typing a removed item back picks it again, through the existing addPending. removeAdded became removeItem, which takes an item off either list. Focus moves to the next chip's ×, as before. "How many" rows and the follow-up are unchanged: the chips go in what.picked.
• Evals (evals/magic-compose/dev-evals.json):
  • ev_034 (snack duty, no dates) and ev_035 (conferences, no times) now expect questions with when and without what, and options that mention the prompt's own thing.
  • New made-up cases ev_040 ("Snack sign up": asks when and how many, options mention snacks) and ev_041 (drivers for away concerts).
  • I numbered them from 040 so they can't collide with the ev_038/ev_039 that Give one-day Magic Compose drafts a Date column, so their reminders go out #693 adds.
  • The item-list cases (ev_001, ev_016, ev_024, ev_032) are unchanged and still expect a draft.
  • The README is updated to match.
• Docs:
  • docs/telemetry.md: magic_compose.asked can list when without what, and options then counts the items read.
  • The help article's questions line now says "remove what doesn't".
  • CHANGELOG under Changed.

Choices I made that the issue didn't

• I edited the CHANGELOG and JSON by splicing, not by running Prettier over them, because main isn't Prettier-clean and a whole-file format would bury the change.
• The magic_compose.answered payload code is unchanged. With what not asked, answered/skipped still says whether any item went back, and picked counts the description's items the organizer kept. The telemetry doc says so.

Tests

• Route (route.questions.db.test.ts): a stubbed ["when","how_many"] reply with one item gives 200 { questions }, makes one call, charges one quota and creates no signup. The "can't ask" fixture is now a reply with neither what nor when, and its retry tests still pass.
• Schema (prompt.test.ts):
  • Rejected: no what and no when; no what with no items, only a blank one, or no whatOptions key; an extra key; mixed draft keys with a when-only questions object.
  • Accepted: one tidied item; five items cut to four.
  • Example D parses as the new questions reply.
• Rule (signup-rules.test.ts): rule 4 asks when for the all-placeholder case, keeps item lists drafting, and keeps placeholders for answers that leave dates out.
• Component (Questions.test.tsx):
  • Without what, the item shows as a picked chip with the new heading and no option pill, and gets a "How many" row.
  • Draft my signup sends it in what.picked.
  • Removing it and adding your own works, typing it back re-picks it, and the follow-up carries both.
  • Focus moves from a removed chip to the next one.
• Unit (answers.test.ts): startingAnswers, removeItem, and buildFollowUp without what.
• Ran in this session, all passing: pnpm lint && pnpm typecheck && pnpm knip && pnpm test (2008 tests), pnpm test:db (629), and pnpm build && pnpm budgets (/app/signups/new at 115.3 of 116 kB).

Before merging

#687's "Done when" asks for a model-panel eval run, PR vs main, and this session has no LLM endpoint. Please run, on this branch and on main (3 runs each, on the OpenRouter panel):

pnpm eval:magic-compose
pnpm eval:magic-compose --tag=questions

Check that:

• "Snack sign up" (ev_040) and ev_034/ev_035/ev_041 now ask with when and without what;
• the ask rate on the item lists (ev_001–ev_027, ev_032, ev_033) doesn't rise;
• ev_036/ev_037 still draft from answers.

---

Generated by Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

[richshaw]

Tophat: passed

Ran 2fdbaa7 locally with pnpm dev, Postgres 16, and Chromium at 1280×900 and 390×844 (phone with touch on). I didn't run the worker, since nothing here touches jobs or email. There's no model in this session, so LLM_BASE_URL pointed at a small local mock of the Chat Completions API. The mock saved each request and answered from a queue I set for each check. That means this tophat shows what the drafter is now told, and what the page does with a when-only questions reply. It can't show how often a real model now asks when, which is the model-panel eval the PR's "Before merging" asks for. All data was made up: pat@example.com, "Snack sign up", "Drivers for the band’s away concerts". For "before" I ran the same script on the PR's base, main at e8ab9b1. Between runs I cleared the per-organizer magic.compose rate-limit row so the checks could repeat.

Check	Result	What I saw
Before (main): a {"topics":["when","how_many"],"whatOptions":["Bring snacks"]} reply	✅ gap confirmed	No questions screen. The route makes its firmer retry and goes straight to drafting (696-before-desktop-result). When the retry asks the same thing again, the page shows "AI draft did not match expected shape — try again or adjust your prompt" (HTTP 500)
What the drafter now reads	✅	The first request's system prompt has rule 4's new sentence (…every slot would be a "TBD:" placeholder…, ask about "when", not "what") and Example D's "topics": ["when"] reply
The same reply on the PR	✅	One call (200), then "A few details first". The screen shows What people sign up for with "From your description. Remove any that don't fit, or add your own.", Bring snacks as a picked chip with ×, no option pills, When is it?, and a "People for Bring snacks" row under How many (696-after-desktop-questions)
Draft my signup	✅	The second request sent what: {picked: ["Bring snacks"], added: []}, when: not_decided, howMany: [null], and the model was told What people sign up for: "Bring snacks". The draft opened in Build (696-after-desktop-after-draft)
Remove, add your own, type it back	✅	Removing Bring snacks dropped its How many row and moved focus to the add box. Adding "Bring drinks" and then typing "bring snacks" re-picked the original. Sent: picked ["Bring snacks"], added ["Bring drinks"], howMany [null, 2] (696-after-desktop-edited)
Focus after removing a chip	✅	With two model items plus "Load the van", removing a chip moved focus to the next chip's ×, and removing the last one moved it to the add box. A when-only reply with no how_many shows no How many section
Every chip removed, then Draft	✅	Sends picked: [], added: []. The drafting screen reads "What: Left blank. The draft will use placeholder slots…", the same as skipping "what" on main
Neighbour: a reply that asks what	✅ unchanged	"What do people sign up for?" / "Pick any that apply, or add your own." with three option pills (aria-pressed=false). Picking one sent picked: ["Drive to an away game"]
Phone, 390×844	✅	Same chip, heading and How many row. Nothing overflows, and Draft went through to Build (696-after-phone-questions, 696-after-phone-after-draft)
Unit tests, console, server log	✅	pnpm test src/help src/components/magic-compose src/lib/magic-compose src/lib/signup-rules: 15 files, 293 tests pass, including the golden prompt and RULES_IN_BOTH. No browser console errors on the PR. The server log has only the health check's pgboss.job probe, because no worker was running

For the owner (UX): when a description says what people sign up for but nothing a real slot could be made from ("Snack sign up"), Magic Compose should now ask When is it? (and how many) instead of making three "TBD" rows. On that screen, the "what" part changes from a question to a statement, What people sign up for, with the items read from the description already picked as removable chips, and "From your description. Remove any that don't fit, or add your own." Descriptions that don't say what still get the old question with option pills. How often a real model takes the new path is still open: please run the model-panel eval the PR asks for. See 696-after-desktop-questions and 696-after-phone-questions beside 696-before-desktop-result, plus 696-after-desktop-edited.

Screenshots are in https://claude.ai/code/session_016YZ9SVx3UwtY8voe3mmB3S.

---

Generated by Claude Code

[richshaw]

Review: changes needed (round 1)

This PR lets Magic Compose ask "when" without asking "what" when a description names the thing but would only make "TBD:" rows. It covers rule 4, the QUESTIONS section, Example D, QuestionsSchema (still .strict() at both levels), the questions screen's pre-picked chips, the evals, the help line, telemetry docs and CHANGELOG. The schema and the screen are done well and well tested. The route, schema, rule, component and unit tests cover each "Done when" item except the model-panel eval, which the PR rightly leaves to the owner. Two leftovers keep it from passing: one line of the prompt still sends the new case to "draft", and one line of the telemetry doc still describes the old payload.

Blocking (2):

• src/lib/magic-compose/prompt.ts:208: the "asking what" bullet still says "If the options would be the prompt's own items or roles, it already says what: draft instead". That sends "Snack sign up" back to TBD rows and contradicts the new rule 4. Also __golden__/system-prompt.txt:22.
• docs/telemetry.md:106: it says answered/skipped cover only the topics asked, but answeredPayload always records what, including in a when-only round. The PR body says the doc covers this, but it doesn't.

Optional (5):

• Questions.tsx:176 (Back): appendAnswers gets the pre-picked items even when nothing was touched. Back straight away turns "Snack sign up" into "Snack sign up\nWhat: Bring snacks (no limit)", text the organizer never chose. When it matches startingAnswers(questions), you could pass the description unchanged.
• Questions.tsx:137 (Draft with placeholders): with the chips untouched, hasAnswers is true. So the button drafts from the read items, with each count sent as "no limit", rather than sending placeholders: true, and magic_compose.answered logs placeholders: false. That's consistent with "empty count = no limit", but it may not be what the button's name suggests. Worth a look when you try the screen.
• src/lib/signup-rules.ts:55: "A prompt that lists items or roles drafts, with or without dates" sits beside Example D and ev_041. Both name a single role and are now expected to ask. The line reads fine as "lists" (plural), but the eval run will show whether models agree.
• route.questions.db.test.ts:88: the old JSDoc (""what" was already known, so it's left out") was left above CANT_ASK's new one.
• The issue's "Done when" says a five-option reply is rejected. The PR cuts it to four, matching what already happens when "what" is asked. That's a reasonable choice, and the description states it.

Earlier threads: none. Copilot couldn't review (quota).
Checks: CI green on 2fdbaa7 (lint-type-test, e2e, CodeQL). The tophat passed on 2fdbaa7, and no runtime code has changed since. Locally I read the diff and ran the built-in code review at high effort; I verified each finding above against the code.

Gate: owner (UX). The questions screen, the prompt and when Magic Compose asks all change. After the fix, look at the tophat's 696-after-desktop-questions / 696-after-phone-questions screenshots, and run the model-panel eval the PR's "Before merging" asks for (ev_040, ev_034, ev_035, ev_041, and the item-list ask rate).

Session: https://claude.ai/code/session_016YZ9SRngohvtRKSKMaRVum

---

Generated by Claude Code

[richshaw]

Addressed review round 1 in fc72b8c. Both blocking threads are answered and resolved above.

The optional points:

• Stale JSDoc above CANT_ASK: taken in fc72b8c. It's a test-only change.
• Back appends the pre-picked items: left as is. It changes how the screen behaves, which is the owner's call at the pipeline:owner gate.
• Draft with placeholders when the chips are untouched: left as is, for the same reason. It's consistent with "an empty count means no limit".
• The "lists items or roles" line beside Example D and ev_041: left as is. The eval run the PR already asks for will show whether models read it that way.
• Five options cut to four: no change. The description already states this choice.

The prompt wording changed, so the PR goes back to tophat.

---

Generated by Claude Code

[richshaw]

Tophat: passed

I ran fc72b8c locally with pnpm dev, Postgres 16, and Chromium at 1280×900 and 390×844. I didn't run the worker, since nothing here touches jobs or email. This session has no real model, so I pointed LLM_BASE_URL at a small local mock of the Chat Completions API. The mock logged every request and replied by prompt: "Snack sign up" got {"topics":["when","how_many"],"whatOptions":["Bring snacks"]}, "Drivers for the away concerts" got when only with two items, a third prompt got the old what + when shape, a fourth asked neither what nor when, and any follow-up got a fixed draft. So this tophat shows how the page and the route handle each reply shape. It can't show how often a real model now asks when. That's the eval the PR's "Before merging" asks for. All data is made up (pat@example.com).

Check	Result	What I saw
Before (main at e8ab9b1): the same when-only reply	✅ gap reproduced	The route rejected the reply, sent the one firmer retry ("Reply with a draft or a refusal now…"), and the organizer landed straight in Build on the mock's draft. No questions screen
After: "Snack sign up" shows the questions	✅	What people sign up for / "From your description. Remove any that don't fit, or add your own." shows Bring snacks as a picked chip with a ×. There are no option pills, then When is it? and How many people do you need? with a row for Bring snacks (696-after-desktop-questions). A single route call, no retry
Remove, add your own, type it back	✅	× on Bring snacks removed it (gone from the page). Then I typed "Bring drinks" + Enter and "Bring snacks" + Enter. The chips are Remove Bring snacks, Remove Bring drinks, and How many has a row for each (696-after-desktop-removed-readded)
Answers reach the follow-up	✅	With date 2030-11-14 and 3 for Bring snacks, Draft my signup sent What people sign up for: "Bring snacks", "Bring drinks", When: 2030-11-14, How many…: "Bring snacks": 3, "Bring drinks": no limit, and opened Build on the draft (696-after-desktop-answered, 696-after-desktop-build)
Draft with placeholders	✅	Sent the kept item and When: not given, use a TBD: placeholder, then opened Build
Phone, 390×844, two items	✅	"Drivers for the away concerts" shows both chips, with no sideways scroll (width 390) (696-after-phone-questions). Removing the first moved focus to Remove Drive to Riverside. Removing the last moved it to the "Add your own" box, and the How many section went away with no items left (696-after-phone-all-removed). Draft my signup then sent What…: not given, use a TBD: placeholder and started drafting
Neighbour: description that doesn't say what	✅ unchanged	The old screen: "What do people sign up for?" / "Pick any that apply, or add your own." with three unpressed option pills (696-neighbour-desktop-what-asked)
Neighbour: a reply asking neither what nor when	✅	The route made its one firmer retry and opened Build on the draft, as before
Unit tests, console, server log	✅	pnpm test src/components/magic-compose src/lib/magic-compose src/lib/signup-rules src/help: 15 files, 293 tests pass. There were no browser console errors. The only server-log errors are the health check's pgboss.job probe, which fails because the worker wasn't running

For the owner (UX): a description that names the job but gives nothing to build slots from ("Snack sign up") used to go straight to a draft of "TBD" rows. With this PR the model is told to ask when instead, and the questions screen can show that. The heading reads "What people sign up for", with the description's things already picked as removable chips, followed by the usual When and How many questions. Descriptions that don't say what still get the old pick-from-options screen. Two things to know:

• If the organizer removes every chip and adds nothing, Draft my signup still drafts. It tells the model "what" wasn't given, so the model uses a placeholder.
• How often real models now ask when, and whether item lists start asking, is down to the eval run the PR's "Before merging" asks for.

See 696-after-desktop-questions, 696-after-phone-questions and 696-after-desktop-removed-readded.

Screenshots are in https://claude.ai/code/session_016YZ9SUnN3jrznFjkUTJvNf.

---

Generated by Claude Code

[richshaw]

Review: passed (round 2)

Round 1's two blocking findings are fixed in fc72b8c, and nothing new came in with the fix. The "asking what" bullet in prompt.ts no longer says "draft instead". It now sends the case where the options would be the prompt's own items to rule 4: ask "when" if every slot would be a "TBD:" placeholder, otherwise draft. The golden prompt changes on that one line only. docs/telemetry.md now says what is always in answered or skipped, even in a round where it wasn't asked, and that matches answeredPayload (telemetry.ts: what goes in answered when any item went back, and when only when a when answer exists). The stale JSDoc above CANT_ASK is gone. The rest of the PR is the same code round 1 reviewed. I went through the repo checklist again: no services, schema, MCP tools, instructions.ts or RULES_IN_BOTH change, and the assistant's half of rule 4 is untouched.

Blocking (0)
Optional (0) new. Round 1's optional points (Back appends the pre-picked items, Draft with placeholders with untouched chips, the "lists items or roles" wording) were answered and left for the owner's call. They're repeated below.
Earlier threads: all resolved (2).
Checks: CI is green on fc72b8c (lint-type-test, e2e, CodeQL). The tophat passed on fc72b8c (17:29Z), on the same head. Locally I read the fix commit and checked the telemetry doc against answeredPayload. Copilot couldn't review (quota).

Gate: owner (UX). This changes when Magic Compose asks questions, the drafter prompt, and the questions screen: "What people sign up for" now shows pre-picked chips taken from the description. Please:

• look at the tophat's 696-after-desktop-questions, 696-after-phone-questions and 696-after-desktop-removed-readded screenshots;
• run the model-panel eval the PR's "Before merging" asks for (ev_040, ev_034, ev_035, ev_041 should ask when without what, the item-list ask rate shouldn't rise, and ev_036/ev_037 should still draft);
• decide on two behaviours round 1 left to you. Pressing Back right away appends the pre-picked items to the description. Draft with placeholders with the chips untouched drafts from the items, with "no limit" counts.

Session: https://claude.ai/code/session_016YZ9SV6f7zJ31MfLwmVv2j

---

Generated by Claude Code