What Is the Math Behind SmartStats?

SmartStats calculations are far more accurate than our old ones. Our old method used a common trick- the Central Limit Theorem and approximated the true statistical distribution with a normal distribution. This approximation works well if you have a lot of samples. That’s not possible for all customers, so it became clear that we needed to find a better way to get more accurate answers.

Let’s look at a couple of graphs that illustrate the problem. We’ll consider an A/B test with a 5% conversion rate. When we have 10,000 samples, true distribution, and normal approximation line up perfectly:

That’s great for anyone who has tens of thousands of visitors reaching the end of their funnel every week. In contrast, when we have only 100 samples, the green and blue lines differ significantly.

Our old method used the approximation (the green line) in our calculations, but our new method uses the true distribution (blue line). It’s more work for us; the true calculations are more difficult for both our math geeks and our computers, but these give more accurate and faster results.

A lot of statistical techniques tell you something like “this method is accurate only if the number of conversions > 3 sqrt (num_conversions x ctr x (1-ctr))” or something along those lines. This kind of “rule of thumb” ensures that the green and blue lines agree with each other. As a result of using exact calculations, SmartStats does not require any such rules.