|
In this article, you’ll learn the following: |
Overview
Imagine you're experimenting to compare two different approaches. You want to know when you've gathered enough data to confidently declare one approach to be more effective. Traditional statistical tests require waiting until all data is collected, which can be inefficient.
However, the modern approach demands continuous monitoring of results throughout the experiment to identify the winner (or stop the experiment) sooner. But, this speed comes at a potential cost: an increased chance of falsely declaring a winner (false positive rate).
Sequential Testing Mode to the Rescue: This mode adjusts probabilities to account for the ongoing nature of the experiment and the maximum number of participants planned. By doing so, it helps maintain a desired false positive rate.
Working Methodology
VWO utilizes a variation of a statistical approach called Alpha-spending. This method adjusts the decision threshold for declaring a winner based on the planned sample size. The benefit of this approach is that it's easy to understand and integrates well with sample size calculations.
Impact on Expected Improvement
Sequential Testing Correction also impacts the expected improvement interval for all variations. Here's what you might observe:
- Wider Expected Improvement Interval: This interval reflects the uncertainty in the final expected improvement and widens with the correction.
Understanding the Values in Reports
Your VWO test report provides two key values to help you interpret results:
- Expected Improvement: The median value of the statistically derived and corrected improvement (considering the ongoing nature of the experiment).
- Expected Improvement Interval: The range within which the final expected improvement might fall, considering statistical uncertainty.
The width of this interval is tied to your configured False Positive Rate (FPR). For example, with an FPR of 10%, the interval would represent a 90% confidence range.
As the experiment progresses and reaches the planned sample size, the correction's influence diminishes, and the expected improvement values approach the observed improvement.
Additional Considerations
- Changing Maximum Participants: If you adjust the maximum number of participants, the Probability of Improvement values will also change due to the correction.
- Toggling Correction On/Off: Enabling or disabling the correction can lead to different conclusions about the results.
- Correction After Maximum Visitors: The correction is not applied once the maximum visitor requirement is reached. However, extending the experiment or stopping it at this point is recommended.
By understanding Sequential Testing Correction, you can make informed decisions based on your experiment results, ensuring reliable insights for optimizing your approaches and campaigns.