Experimentation & A/B Testing

Because the cost of doing nothing, is very high

This is one a series of posts trying to explain marketing jargon to my engineer friends. I will explain as many terms as possible without the marketing jargon so its easy to understand.

A/B testing (also called split testing) is an experiment where you show two versions of something to different groups of users simultaneously and measure which version performs better against a defined metric. Version A is the control (the current thing). Version B is the variant (the new thing you’re testing).

Marketers A/B test all the time:

• Email subject lines: Does “You’re leaving money on the table” outperform “5 ways to grow your revenue”?
• Landing page headlines: Short and punchy vs. long and descriptive?
• Button colors and CTAs: “Start Free Trial” vs. “Get Started” vs. “Try It Free”
• Ad creatives: Photo of a person vs. product screenshot vs. illustrated graphic
• Showing a relevant set of “You may also like”, how do you know what the best performing set it?
• Pricing page layout: 3-tier vs. 2-tier, monthly toggle default vs. annual
• Onboarding flows: 5-step wizard vs. single form vs. progressive disclosure

Where Engineers and Marketers Disagree on A/B Testing

Engineers often get frustrated with how marketers run A/B tests. Here’s the tension:

• Statistical significance: Marketers sometimes call a test “done” when they see a 60% win rate after 200 users. Engineers know that’s nowhere near statistically significant. You need enough sample size for the result to be trustworthy — usually thousands of users per variant, depending on the baseline conversion rate.
• Peeking problem: Marketers often check results daily and stop the test when they see a result they like. This is a classic statistical mistake called optional stopping — it inflates false positive rates dramatically.
• Novelty effect: A new button color might perform better simply because it’s different, not because it’s better. The effect often fades after a week.

Multivariate Testing (MVT) vs. A/B Testing

If A/B testing is testing one variable at a time, multivariate testing (MVT) tests multiple variables simultaneously. For example: does changing the headline AND the image AND the button color together produce a better result? MVT requires much larger sample sizes because you’re testing the interaction of multiple variables. Most marketing teams stick to A/B because they don’t have enough traffic for MVT to be statistically valid.

Experimentation is a Culture

The best growth teams don’t just A/B test — they build a culture of experimentation. This means:

• Every initiative has a hypothesis: “We believe that [change] will result in [outcome] because [reasoning]”
• Tests are documented, win or lose — learnings are shared across teams
• Failure is expected and valued (a test that disproves a hypothesis is still useful data)
• There’s an experimentation platform that handles randomization, bucketing, and analysis (e.g., Optimizely, LaunchDarkly, Statsig, or a homegrown solution)

The Engineering Overlap

Feature flags (which engineers love) are the infrastructure layer that enables marketing experimentation. When you implement a feature behind a flag with a percentage rollout, you’ve just built the foundation for an A/B test. Tools like LaunchDarkly, Split.io, Statsig, and GrowthBook and many more unify feature flagging and experimentation into one platform which is why engineering and growth teams often share the same tooling.

Simply because, doing nothing costs a lot of time, and money. You are probably leaving money on the table if you are not constantly testing.

[All opinions expressed are my own and have no relation with my employers – past or present. I use Huffl.AI to structure my thoughts. ]