Estimate the Duration to Get Statistically Significant Results From Your Test in VWO

In this article, you’ll learn about the Estimation Duration Calculator and its capabilities in VWO.

In a typical business scenario, before you create an A/B test on your website, you form a hypothesis that you want to test. It helps you design your test, set up goals you want to optimize and approximate the impact the variations will have on the goals.

In VWO, you can use this approximate impact you care about to estimate how long a test will take to generate statistically significant results. This can be done using the duration calculator, which exposes all variables (goal, current conversion rate, current revenue per visitor, the improvement you wish to detect, and the statistical significance threshold) that can impact the time it takes for the test to reach statistical significance.

The duration calculator helps you judge by when you can expect a significant result from your test. Based on the output, you can also decide whether it is worth waiting for the test to conclude.

To estimate the time to reach significance, all you need to do is specify the variable values in the duration calculator. Post this, VWO gives you the timelines for which you must run your test. However, if you have not configured the variables, we’ll wait for the following before we automatically use the data collected in the test to show the estimated test duration:

• A minimum of 100 visitors or a minimum of four hours since the test started, whichever is earlier, to get estimated daily visitors,   
• At least ten conversions in the base variation to estimate the average conversion rate and average revenue per visitor (if applicable).

Applications of Duration Calculator 

• Prioritize your hypothesis by estimating how long a test will take to reach statistical significance. The test that can create a bigger impact in less time can be given a higher priority than others.
  
  
• Estimate the time it will take for the test to conclude based on your criteria. We have also included an option to customize the statistical significance threshold for the test you plan to run. This can help you estimate the time needed for the test, which you would like to conclude when the probability to beat control is, say 80% instead of 95% (default).
  
  
• Understand what to do when the estimated duration has passed, but the test has not reached statistical significance. This calculator can present to you an extreme case scenario. You can wait until then, and if you don’t get significant results in that period too, it urges you to conclude the test.

Using the Duration Calculator in VWO

To learn how to use the duration calculator and how to configure the different fields, follow this:

Procedure

1. Login to your VWO account.
2. From the panel on the left, go to TESTING and create a new campaign (A/B or Split).
3. Once you finalize your campaign, you’ll see the Estimate Duration option. Clicking on this displays the Estimate time to reach significance dialog where you can specify the variable values to estimate the test duration.
   
   NOTE: This is required for the first time (when there’s no data in the test). Once your test starts collecting data, you can use the “automatic update” option to get this field’s value. Or this will pick-up data from the test. Thus, you do not have to go through this entire process to estimate the test duration.
4. For any existing campaign, go to the REPORT tab and click on the hourglass icon to estimate your test duration.
5. To view information about visitors included so far, remaining visitors, and the number of days remaining to arrive at statistical significance, hover on the hourglass icon and click on the See how it is calculated option.
6. If you wish to modify the variables on which the duration is calculated, click on the Change estimation inputs option. Here, you can modify the following:
   
   
   • • What goal do you want to estimate duration for?: Choose the goal for which you want to estimate the time to reach statistical significance.
       
       
     • What is the existing average conversion rate of this goal?: It is the current conversion rate of your website (the control version). To determine this, we recommend using either Google Analytics or running VWO Goals. If your test is targeted to a segment, you should consider the same segments when calculating the conversion rate. If the automatic update option is enabled, this will be automatically picked up from the data collected in the test.
       
       
     • What is the minimum improvement you wish to detect for this goal?: It is the percentage of improvement in conversion rate you wish to detect against the existing conversion rate. By default, it is set to 20%. This is a relative improvement and not an absolute improvement. If the actual improvement is less than the specified improvement, the test will take longer than the estimated duration. We recommend you to look at the hypothesis and the changes you are testing and decide an estimated value based on that.
       
       
     • What is the average existing revenue per visitor of this goal?: It is the current revenue per visitor on your website (the control version). When the automatic update option is enabled in the duration calculator, the revenue per visitor is automatically obtained from the test's data. You can copy the revenue that you used while creating this goal.
       
       NOTE: This field only appears for revenue goals.
     • The statistical threshold for concluding the test: Changing this doesn’t mean that the VWO will calculate the results based on the new threshold. VWO continues to use the same set of the threshold; however, you can estimate the test duration based on something else. Thus, if you want to estimate the time required to reach  80% probability, you can do that.

What are Average and Extreme Cases?

To calculate the Average and Extreme cases, we simulate 1000 tests to derive the number of visitors required to conclude 95% of the 1000 tests. We use this visitor count to calculate the number of days required for the Average and Extreme case. 

In the Average case, we look for the number of visitors it takes to conclude at least 50% (500) tests, whereas, for the Extreme case, we look for the number of visitors it takes to conclude at least 95% (950) tests.