How VWO Calculates Statistical Significance

You can use our Test Duration Calculator to find out the statistical significance required for your business needs. Alternatively, you can download this excel spreadsheet to calculate the data yourself.

Note that the Test Duration Calculator and the excel spreadsheet uses the Frequentist Method to calculate data. However, the statistical significance used in Frequentist Method does not apply to A/B and Split tests in VWO, which are based on Bayesian statistics.

You may observe that the ‘significance value’ is different in the calculator and excel spreadsheet. The reason is that the excel spreadsheet shows the exact ‘p-value’, while the significance calculator will give ‘p-value’ between 1-p if the p-value comes out to be more than 0.5. If the p-value is less than 0.5, both the duration calculator as well as excel sheet will show the same value.

USE CASE 1: IF THE P-VALUE IS THE SAME IN BOTH CALCULATORS

USE CASE 2: IF THE P-VALUE IS DIFFERENT IN BOTH CALCULATORS

Refer to our KB article on Bayesian statistics and the mathematics behind calculating SmartStats. For revenue goals, download this excel file to understand the statistical significance.