VWO calls a variation a Winner when its potential loss is below the threshold of caring and its Probability to be best is higher than 95%. The probability to be best is typically the probability of a variation outperforming all other variations to be the best. VWO declares only one Winner.
NOTE: To declare a variation as a winner, VWO requires a minimum of
25 conversions per variation, 1500 visitors for the test, and the test
must be running for at least a week.
How VWO Calculates the Threshold of Caring
The threshold of caring is the magnitude that decides whether to consider a variation to declare a Smart Decision/Winner or not.
The threshold of caring = Current conversion rate * multiplier * 100
where the multiplier is 0.001 for revenue goal and 0.0075 for non-revenue goal
NOTE: We do not ask for these details, we compute them based on the
first week's data.
In the current reporting system, the Conversion Rate of a variation is taken from the test report. Say it is 2.5%, the threshold of caring will be: 0.025 * 0.001 * 100 = 0.0025. Hence for the variation to be a winner, it’s potential loss must be below 0.0025, and the Probability to be best is above 95%
NOTE: If the threshold of caring for any test is > 1%, we will set
it to 1%, and if the threshold is < 0.001%, we will set it to 0.001%.
NOTE: The calculations of the threshold of caring was changed in
September 2019. The formula earlier was-
The threshold of caring = Current conversion
rate * Expected lift in conversion rate * multiplier * 100 where
multiplier = 0.01 for High Certainty mode; 0.075 for Balanced mode;
and 0.2 for Quick Learnings mode and current conversion rate and
expected lift in conversion rate were accepted from the user.
When is a Winner Retracted in VWO?
Once a variation has been declared as a winner, the decision will be reversed if the Probability to be best is below 90% or if the absolute potential loss is greater than 105% of the threshold of caring. Whenever the variation's probability to beat baseline or the potential loss breach the threshold, the Smart Decision is retracted.
NOTE: Previously, the Smart Decision was retracted when the absolute
potential loss of the variation declared as Smart Decision was > 1%.
This logic was ineffective in scenarios where there was a marginal
difference in the conversion rate of the variations.