You can use our Test Duration Calculator to find out the statistical significance required for your business needs. Alternatively, you can download this excel spreadsheet to calculate the data yourself.
NOTE: The Test Duration Calculator and the excel spreadsheet uses
the Frequentist Method to calculate data. However, the statistical
significance used in Frequentist Method does not apply to A/B and
Split tests in VWO, which are based on Bayesian statistics.
You may observe that the ‘significance value’ is different in the calculator and excel spreadsheet. The reason is that the excel spreadsheet shows the exact ‘p-value’, while the significance calculator will give ‘p-value’ between 1-p if the p-value comes out to be more than 0.5. If the p-value is less than 0.5, both the duration calculator as well as excel sheet will show the same value.
USE CASE 1: IF THE P-VALUE IS THE SAME IN BOTH CALCULATORS
USE CASE 2: IF THE P-VALUE IS DIFFERENT IN BOTH CALCULATORS
Refer to our KB article on Bayesian statistics and the mathematics behind calculating SmartStats. For revenue goals, download this excel file to understand the statistical significance.