VWO calls a variation a Winner when its potential loss is below the threshold of caring and its Probability to be best is greater than or equal to 95%. The probability to be best is typically the probability of a variation outperforming all other variations to be the best. VWO declares only one Winner.
How VWO Calculates the Threshold of Caring
The threshold of caring is the magnitude that decides whether to consider a variation to declare a Smart Decision/Winner or not.
The threshold of caring = Current conversion rate * multiplier * 100
where the multiplier is 0.001 for revenue goal and 0.0075 for non-revenue goal
In the current reporting system, the Conversion Rate of a variation is taken from the test report. Say it is 2.5%, the threshold of caring will be: 0.025 * 0.001 * 100 = 0.0025. Hence for the variation to be a winner, its potential loss must be below 0.0025, and the Probability to be best is greater than or equal to 95%
When is a Winner Retracted in VWO?
Once a variation has been declared as a winner, the decision will be reversed if the Probability to be best is below 90% or if the absolute potential loss is greater than 105% of the threshold of caring. Whenever the variation's probability to beat baseline or the potential loss breach the threshold, the Smart Decision is retracted.