Skip to content

feat(home): flag-driven getting-started hero A/B/C test - #163

Draft
NetanelBaruch wants to merge 4 commits into
home-page-updatefrom
test/getting-started-flow-abtest
Draft

feat(home): flag-driven getting-started hero A/B/C test#163
NetanelBaruch wants to merge 4 commits into
home-page-updatefrom
test/getting-started-flow-abtest

Conversation

@NetanelBaruch

@NetanelBaruch NetanelBaruch commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

What

Implements the getting-started flow A/B/C test on the home hero: each visitor sees one committed primary CTA instead of every option at once, driven by the Mixpanel feature flag.

Test plan: 02-campaigns/2026-08-04-website-flow-ab-test-plan.md (Novu-Marketing repo). Mixpanel experiment: Website getting-started flow: UI vs CLI vs Prompt (currently draft, flag disabled).

Arm Flag value Primary CTA Preview
Control ui "Get started free" → agents sign-up ui
B cli npx novu connect copy command cli
C prompt copyable coding-agent prompt prompt

Flag key (exact, do not rename): website-getting-started-flow-ui-vs-cli-vs-prompt-SJw6Uc, splits 34/33/33.

How

  • Enable Mixpanel feature flags in src/lib/mixpanel.ts init (flags: true, autofetch on load).
  • src/lib/experiments.ts (new): resolves the arm from the flag, registers getting_started_flow as a super property (so every downstream event carries the arm), and exports the flag key + event-name constants. Reading the variant fires the exposure event that assigns the visitor to the experiment.
  • useGettingStartedFlow hook (new): renders the control (ui) on the server and first paint, then swaps to the assigned arm once the flag resolves. All arms share the same block height, so the swap does not shift layout.
  • Two website-side copy eventsWebsite CLI Command Copied and Website Prompt Copied — fired via a new optional onCopy callback on CopyCommand and CopyPromptButton. (The existing Run Novu Connect Command / AI prompt copied events fire in the terminal / post-signup dashboard, not on the site.)

QA

Force any arm locally with a query param, no Mixpanel assignment involved:

  • /?gsf=ui · /?gsf=cli · /?gsf=prompt

Confirm in Mixpanel Live View that reading the flag emits the exposure event and that Website CLI Command Copied / Website Prompt Copied arrive with the getting_started_flow property.

Launch (after merge + QA)

  1. Merge; verify each arm renders and both copy events fire.
  2. Enable the flag in Mixpanel, then launch the experiment (draft → active).
  3. Check SRM + exposure counts at 48h; read results at day 28 or on a sequential-testing call.

Open decision for review

The plan defines the control as a signup-primary CTA, but the current hero shows the CLI command + prompt side by side (no signup). So this PR makes ui a signup button rather than "the current hero". If you'd rather the control mirror today's hero exactly, say so and I'll adjust the control arm (the flag/experiment already treats ui as control either way). Signup destination used: ROUTE.dashboardV2AgentsSignUp.

Base branch is home-page-update (where the home hero lives); retarget if Pixel Point works from a different integration branch.

🤖 Generated with Claude Code

Wire the home hero getting-started CTA to the Mixpanel feature flag
"website-getting-started-flow-ui-vs-cli-vs-prompt-SJw6Uc" so each visitor
sees one committed flow instead of every option at once:

- ui (control): primary "Get started free" signup CTA
- cli: primary "npx novu connect" copy command, signup as secondary
- prompt: primary copyable coding-agent prompt, signup as secondary

Implementation:
- Enable Mixpanel feature flags in init (autofetch on load).
- lib/experiments.ts resolves the arm, registers "getting_started_flow" as a
  super property, and exposes the flag key + event name constants. Reading the
  variant fires the exposure event that assigns the visitor to the experiment.
- useGettingStartedFlow hook renders the control on server/first paint, then
  swaps to the assigned arm. ?gsf=ui|cli|prompt forces an arm for QA.
- All arms share the same block height to avoid layout shift on swap.
- Add website-side copy events "Website CLI Command Copied" and
  "Website Prompt Copied" via an onCopy callback on CopyCommand and
  CopyPromptButton.

Experiment stays in draft/flag disabled until this ships and each arm is QA'd.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@vercel

vercel Bot commented Aug 15, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
website-new Ready Ready Preview Aug 17, 2026 4:53pm

Request Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants