Spotify Analytics

Email Campaign
A/B Test Results

Two-proportion z-test · 10,000 users · Free + Premium segments

Variant B wins
n = 10,000 α = 0.05
3 metrics significant
Open rate — B
32.1%
▲ +27.6% lift vs A
Variant A baseline: 25.2%
Click rate — B
3.71%
▲ +118% lift vs A
Variant A baseline: 1.70%
Conversion — B
0.34%
▲ +467% lift vs A
Variant A baseline: 0.06%
Annual revenue lift
$83.9M
▲ Projected at scale
250M free users @ $9.99/mo
Metric comparison

Variant A vs Variant B — All Metrics

Personalized subject line (B) outperforms generic recap (A) across the full funnel

Variant A — "Your Weekly Music Recap is Here"
Variant B — "We picked these songs just for you"
Open rate
Variant A
25.17%
25.2%
Variant B
32.10%
32.1%
Click rate
Variant A
1.70%
1.70%
Variant B
3.71%
3.71%
Conversion rate (Free → Premium)
Variant A
0.06%
0.06%
Variant B
0.34%
0.34%
Statistical significance
Open rate
25.17% → 32.10%
p = 0.0000 · significant at α = 0.05 · CI: [5.07%, 8.79%]
+27.6% lift
Click rate
1.70% → 3.71%
p = 0.0000 · significant at α = 0.05 · CI: [1.42%, 2.60%]
+118% lift
Conversion rate
0.06% → 0.34%
p = 0.0000 · significant at α = 0.05 · CI: [0.16%, 0.39%]
+467% lift
Segmentation analysis

Open Rate by Age Group

Variant B outperforms in every cohort — 25–34 shows strongest absolute lift

Age group Variant A Variant B Lift
18–24 24.9%
30.7%
+23.3%
25–34 25.7%
34.4%
+33.8%
35–44 26.0%
30.5%
+17.3%
45+ 23.7%
32.5%
+37.1%

Conversion Rate by User Tier

Variant B drives near-identical uplift for both Free and Premium users

Free users
70% of base
+467%
Variant A
0.06%
Variant B
0.34%
Premium users
30% of base
+386%
Variant A
0.07%
Variant B
0.34%
Business impact
Projected annual revenue lift — full rollout
$83,916,000

If Variant B is rolled out to all 250 million free Spotify users, the projected uplift in Premium conversions generates an additional ~$83.9M annually at the $9.99/month price point — based on the 0.28 percentage point conversion rate improvement observed in this experiment.

700K
additional conversions
250M
free users at scale
$9.99
premium / month

Recommendation: Roll out Variant B

All three funnel metrics show statistically significant improvement (p < 0.0001) with Variant B — the personalized subject line. The lift holds consistently across every age group and user tier, indicating the effect is not segment-specific. Given the magnitude of the conversion rate improvement (+0.28pp) and the projected $83.9M annual revenue impact, a full rollout is recommended with no further holdout required. Consider A/B testing send time next (Morning/Afternoon/Evening) to compound gains.