- Accounts on the non-Data360 framework, and
- Campaigns created before an account was migrated from non-Data360 to Data360.
For new campaigns (created after migration) or all campaigns in accounts created directly on the Data360 framework, refer to Enhanced Reports in Data360, VWO’s upgraded statistical engine that offers deeper insights and improved control on statistics.
SmartStats calculations are far more accurate than our old ones. Our old method used a common trick- the Central Limit Theorem and approximated the true statistical distribution with a normal distribution. This approximation works well if you have a lot of samples. That’s not possible for all customers, so it became clear that we needed to find a better way to get more accurate answers.
The bias that can be introduced if all days are not evenly included in the test is known as the day of the week effect.
You can achieve this by scheduling your tests to end at a certain time and date. To know more about scheduling a test, refer to How to Schedule Your VWO Test.
Let’s look at a couple of graphs that illustrate the problem. We’ll consider an A/B test with a 5% conversion rate. When we have 10,000 samples, true distribution and normal approximation line up perfectly:
That’s great for anyone who has tens of thousands of visitors reaching the end of their funnel every week. In contrast, when we have only 100 samples, the green and blue lines differ significantly.
Our old method used the approximation (the green line) in our calculations, but our new method uses the true distribution (blue line). It’s more work for us; the true calculations are more difficult for our math geeks and our computers, but they give more accurate and faster results.
A lot of statistical techniques tell you something like “this method is accurate only if the number of conversions > 3 sqrt (num_conversions x ctr x (1-ctr))” or something along those lines. This kind of “rule of thumb” ensures that the green and blue lines agree with each other. As a result of using exact calculations, SmartStats does not require any such rules.