|
In this article, you’ll learn the following: |
Overview
Imagine you're experimenting with comparing different design options for a website. Instead of just one alternative, you're testing several variations to see which one performs best. However, with more variations comes a challenge: the chance of mistakenly declaring a winner by random chance increases.
The problem of multiple comparisons: A typical false positive rate (FPR) is applied to each variation individually. However, when considering all variations in a single experiment, the overall error rate (family-wise error rate) can become inflated.
Bonferroni Correction to the rescue: This technique addresses this issue by adjusting the statistical threshold for declaring a winner. It does this by considering the number of variations and dividing the individual FPR by that number. Essentially, it makes it harder to declare a winner statistically, ensuring any observed improvements are more likely to be genuine.
How Does VWO Handle Bonferroni Correction?
While traditional Bonferroni correction simply adjusts the decision boundary, VWO takes a more nuanced approach. It incorporates the uncertainty directly into the statistical significance values, making them more informative for the experimenter.
This means you get a single, decisive number that reflects the uncertainty associated with each variation's improvement. No need to interpret complex probabilities and change decision boundaries.
Impact on Expected Improvement
One consequence of Bonferroni correction is that the expected improvement for all variations tends to be "flattened down". This means that very high or low improvement values become less likely.
Important Considerations
- Changing Variations: Adding or removing variations during an experiment will change the statistical significance values for all variations due to the correction recalculating the uncertainty.
- Toggling Correction On/Off: Enabling or disabling the correction can lead to different conclusions about the results.
By understanding Bonferroni Correction, you can make informed decisions based on your experiment results, ensuring that any identified winners are truly effective and not just random fluctuations.