US Launch A/B Test Plan

Issue: BUY-3127
Owner: Wave (Growth Specialist)
Status: Draft v1 — US Launch Week

---

Overview

Three independent A/B tests run during US launch week to optimize hero copy, CTA button text, and onboarding flow. All tests use a cookie-based 50/50 split. Results feed the next iteration of the US landing page and onboarding funnel.

---

Test 1 — Hero Headline

Implementation:

• Cookie key: ab_hero, values: A or B
• Set on first visit to / or /us
• Persists for 30 days

Metrics:

• Primary: conversion to first search (event: search_performed)
• Secondary: bounce rate on landing page, time on page

---

Test 2 — CTA Button

Implementation:

• Cookie key: ab_cta, values: A or B
• Independent of hero test — same user may see A/B hero + A/B CTA in any combination
• Set on first page load

Metrics:

• Primary: CTA click-through rate (event: cta_clicked)
• Secondary: conversion to first search after CTA click

---

Test 3 — Onboarding Flow

Implementation:

• Cookie key: ab_onboard, values: A or B
• Variant A renders a <OnboardingSavingsExample /> component on first login / first visit post-signup
• Variant B renders <SearchBar /> immediately — no interstitial
• Both groups see the same core experience after their respective onboarding moment

Metrics:

• Primary: first search completion rate (event: search_performed)
• Secondary: signup rate (event: user_signed_up), return visit rate at day 7

---

Cookie-Based Split Logic

function getVariant(cookieKey) {
  const existing = getCookie(cookieKey);
  if (existing) return existing; // preserve variant across sessions

  const variant = Math.random() < 0.5 ? 'A' : 'B';
  setCookie(cookieKey, variant, { maxAge: 60 * 60 * 24 * 30 }); // 30 days
  return variant;
}

All three tests are independent — a user can be in variant B for hero, A for CTA, and B for onboarding simultaneously.

---

Event Tracking

All events include a session ID and timestamp. Events fire to the analytics pipeline (/api/events or Segment/PostHog endpoint TBD).

---

Minimum Sample Size

For a baseline conversion rate of ~5% and a target MDE of +20% relative lift:

• Per variant: ~3,100 visitors
• Total (3 tests × 2 variants): ~18,600 visitors minimum across launch week
• If US launch targets 10k visitors/week, extend test to 2 weeks or reduce to 80/20 split

---

Analysis Plan

• Check results at day 7 and day 14
• Use a two-proportion z-test for each metric; significance threshold p < 0.05
• Report: lift per variant, confidence interval, p-value, and recommendation
• Winner declared only if: (a) p < 0.05, (b) effect direction is consistent across primary + secondary metrics, (c) no cross-test interactions

---

Technical Tasks

1. Frontend: Implement getVariant(cookieKey) utility — 0.5 day
2. Frontend: Wire hero headline, CTA button, and onboarding components to variant cookie — 0.5 day
3. Analytics: Confirm event tracking for all 5 events above — 0.5 day
4. QA: Smoke-test all 8 combinations (2³) manually before launch — 0.5 day
5. Launch: Deploy behind feature flag; confirm 50/50 split in analytics — launch day

---

Rollout Plan

---

Success Criteria

• All three tests reach minimum sample size within 14 days
• At least 1 of 3 tests shows statistically significant lift (p < 0.05) on primary metric
• No test shows a significant decline in signup rate or return visit rate
• Documented recommendation for permanent page state for each of the three elements