{"id":79,"date":"2026-02-17T05:22:57","date_gmt":"2026-02-17T05:22:57","guid":{"rendered":"https:\/\/pptx.wtf\/?p=79"},"modified":"2026-04-17T05:36:16","modified_gmt":"2026-04-17T05:36:16","slug":"experimentation-a-b-testing","status":"publish","type":"post","link":"https:\/\/pptx.wtf\/?p=79","title":{"rendered":"Experimentation &amp; A\/B Testing"},"content":{"rendered":"\n<h3 class=\"wp-block-heading\">Because the cost of doing nothing, is very high<\/h3>\n\n\n\n<p>This is one a series of posts trying to explain marketing jargon to my engineer friends. I will explain as many terms as possible without the marketing jargon so its easy to understand.<\/p>\n\n\n\n<p><strong>A\/B testing<\/strong> (also called <strong>split testing<\/strong>) is an experiment where you show two versions of something to different groups of users simultaneously and measure which version performs better against a defined metric. Version A is the control (the current thing). Version B is the variant (the new thing you\u2019re testing).<\/p>\n\n\n\n<p>Marketers A\/B test all the time:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Email subject lines: Does &#8220;You\u2019re leaving money on the table&#8221; outperform &#8220;5 ways to grow your revenue&#8221;?<\/li>\n\n\n\n<li>Landing page headlines: Short and punchy vs. long and descriptive?<\/li>\n\n\n\n<li>Button colors and CTAs: &#8220;Start Free Trial&#8221; vs. &#8220;Get Started&#8221; vs. &#8220;Try It Free&#8221;<\/li>\n\n\n\n<li>Ad creatives: Photo of a person vs. product screenshot vs. illustrated graphic<\/li>\n\n\n\n<li>Showing a relevant set of &#8220;You may also like&#8221;, how do you know what the best performing set it? <\/li>\n\n\n\n<li>Pricing page layout: 3-tier vs. 2-tier, monthly toggle default vs. annual<\/li>\n\n\n\n<li>Onboarding flows: 5-step wizard vs. single form vs. progressive disclosure<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Where Engineers and Marketers Disagree on A\/B Testing<\/h3>\n\n\n\n<p>Engineers often get frustrated with how marketers run A\/B tests. Here\u2019s the tension:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Statistical significance:<\/strong> Marketers sometimes call a test &#8220;done&#8221; when they see a 60% win rate after 200 users. Engineers know that\u2019s nowhere near statistically significant. You need enough sample size for the result to be trustworthy \u2014 usually thousands of users per variant, depending on the baseline conversion rate.<\/li>\n\n\n\n<li><strong>Peeking problem:<\/strong> Marketers often check results daily and stop the test when they see a result they like. This is a classic statistical mistake called <em>optional stopping<\/em> \u2014 it inflates false positive rates dramatically.<\/li>\n\n\n\n<li><strong>Novelty effect:<\/strong> A new button color might perform better simply because it\u2019s <em>different<\/em>, not because it\u2019s <em>better<\/em>. The effect often fades after a week.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Multivariate Testing (MVT) vs. A\/B Testing<\/h3>\n\n\n\n<p>If A\/B testing is testing one variable at a time, <strong>multivariate testing (MVT)<\/strong> tests multiple variables simultaneously. For example: does changing the headline AND the image AND the button color together produce a better result? MVT requires much larger sample sizes because you\u2019re testing the interaction of multiple variables. Most marketing teams stick to A\/B because they don\u2019t have enough traffic for MVT to be statistically valid.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Experimentation is a Culture <\/h3>\n\n\n\n<p>The best growth teams don\u2019t just A\/B test \u2014 they build a <strong>culture of experimentation<\/strong>. This means:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Every initiative has a hypothesis: &#8220;We believe that [change] will result in [outcome] because [reasoning]&#8221;<\/li>\n\n\n\n<li>Tests are documented, win or lose \u2014 learnings are shared across teams<\/li>\n\n\n\n<li>Failure is expected and valued (a test that disproves a hypothesis is still useful data)<\/li>\n\n\n\n<li>There\u2019s an experimentation platform that handles randomization, bucketing, and analysis (e.g., Optimizely, LaunchDarkly, Statsig, or a homegrown solution)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">The Engineering Overlap<\/h3>\n\n\n\n<p>Feature flags (which engineers love) are the infrastructure layer that enables marketing experimentation. When you implement a feature behind a flag with a percentage rollout, you\u2019ve just built the foundation for an A\/B test. Tools like <strong>LaunchDarkly<\/strong>, <strong>Split.io<\/strong>, <strong>Statsig<\/strong>, and <strong>GrowthBook<\/strong> and many more unify feature flagging and experimentation into one platform which is why engineering and growth teams often share the same tooling.<\/p>\n\n\n\n<p>Simply because, doing nothing costs a lot of time, and money. You are probably leaving money on the table if you are not constantly testing.<\/p>\n\n\n\n<p>[All opinions expressed are my own and have no relation with my employers &#8211; past or present. I use <a href=\"https:\/\/huffl.ai\" data-type=\"link\" data-id=\"https:\/\/huffl.ai\">Huffl.AI<\/a> to structure my thoughts. ]<\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Because the cost of doing nothing, is very high This is one a series of posts trying to explain marketing jargon to my engineer friends. I will explain as many terms as possible without the marketing jargon so its easy to understand. A\/B testing (also called split testing) is an experiment where you show two [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":80,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[],"class_list":["post-79","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-marketing"],"_links":{"self":[{"href":"https:\/\/pptx.wtf\/index.php?rest_route=\/wp\/v2\/posts\/79","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pptx.wtf\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pptx.wtf\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pptx.wtf\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/pptx.wtf\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=79"}],"version-history":[{"count":3,"href":"https:\/\/pptx.wtf\/index.php?rest_route=\/wp\/v2\/posts\/79\/revisions"}],"predecessor-version":[{"id":89,"href":"https:\/\/pptx.wtf\/index.php?rest_route=\/wp\/v2\/posts\/79\/revisions\/89"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/pptx.wtf\/index.php?rest_route=\/wp\/v2\/media\/80"}],"wp:attachment":[{"href":"https:\/\/pptx.wtf\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=79"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pptx.wtf\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=79"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pptx.wtf\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=79"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}