What is a Winner in VWO?

VWO calls a variation a Winner when its potential loss is below the threshold of caring and its Probability to be best is higher than 95%.  The probability to be best is typically the probability of a variation outperforming all other variations to be the best. VWO declares only one Winner.

Note: To declare a variation as a winner, VWO requires a minimum of 25 conversions per variation, 1500 visitors for the test, and the test must be running for at least a week.

How VWO Calculates the Threshold of Caring
The threshold of caring is the magnitude that decides whether to consider a variation to declare a Smart Decision/Winner or not.

The threshold of caring = Current conversion rate * multiplier * 100
where the multiplier is 0.1*0.01 for revenue goal and 0.1*0.075 for non-revenue goal

Note: We do not ask for these details, we compute them based on the first week's data.

In the current reporting system, the Conversion Rate of a variation is taken from the test report. Say it is 2.5%, the threshold of caring will be: 0.025 * 0.1 * 0.01 * 100 = 0.0025. Hence for the variation to be a winner, it’s potential loss must be below 0.0025, and the Probability to be best is above 95%

Note: If the threshold of caring for any test is > 1%, we will set it to 1%, and if the threshold is < 0.001%, we will set it to 0.001%. 

Note: The calculations of the threshold of caring was changed in September 2019. The formula earlier was-The threshold of caring = Current conversion rate * Expected lift in conversion rate * multiplier * 100 where multiplier = 0.01 for High Certainty mode; 0.075 for Balanced mode; and 0.2 for Quick Learnings mode and current conversion rate and expected lift in conversion rate were accepted from the user.

When is a Winner Retracted in VWO?
Once a variation has been declared as a winner, the decision will be reverted only if the absolute potential loss for that variation becomes more than 1%. This helps in cases where the winner is declared and then due to change in visitor behavior patterns, the variation underperforms.