A/B Math: Defeating Randomness
Learn why a 10% increase in conversions is completely meaningless until it passes a 95% Confidence Interval Z-Score test.
The Law of Small Numbers
If you flip a coin 4 times, you might get 3 Heads and 1 Tail. That doesn't mean the coin is rigged to produce 75% Heads—it's just a random anomaly because the sample size is microscopic. You must flip the coin 10,000 times before it mathematically settles perfectly at 50%.
In web design, if you change a button from Blue to Red, and the Red button gets 5 clicks while the Blue gets 3... your boss might claim the Red button converts 66% better. This is a catastrophic misinterpretation of statistics.
The P-Value (Null Hypothesis)
The P-Value calculates the strict probability that your "winning" test was just entirely random luck.
If your P-Value is 0.20, there is a 20% chance that the Red button isn't actually better, and you just got lucky with the traffic today. The global scientific standard declares that a test is only valid ("Significant") if the P-Value drops below 0.05 (Meaning: less than a 5% chance of random luck, yielding a 95% Confidence Interval).
When to stop an A/B Test?
Most novice marketers stop an A/B test the minute one variant pulls slightly ahead. You must mathematically wait for the Z-Score to cross `1.96` (for 95% confidence). If you lock in a winner prematurely, you will suffer "Imaginary Growth"—where your analytics say conversions are up 10%, but your bank account hasn't moved a single dollar, because the test was never statistically legitimate.