What is a Winner in VWO?

Vaibhav Singh

Updated
May 24, 2021 09:18

close this to read article

VWO calls a variation a Winner when its potential loss is below the threshold of caring and its Probability to be best is greater than or equal to 95%. The probability to be best is typically the probability of a variation outperforming all other variations to be the best. VWO declares only one Winner.

NOTE: To declare a variation as a winner, VWO requires a minimum of 25 conversions per variation, 1500 visitors for the test, and the test must be running for at least a week.

How VWO Calculates the Threshold of Caring
The threshold of caring is the magnitude that decides whether to consider a variation to declare a Smart Decision/Winner or not.

The threshold of caring = Current conversion rate * multiplier * 100
where the multiplier is 0.001 for revenue goal and 0.0075 for non-revenue goal

NOTE: We do not ask for these details, we compute them based on the first week's data.

In the current reporting system, the Conversion Rate of a variation is taken from the test report. Say it is 2.5%, the threshold of caring will be: 0.025 * 0.001 * 100 = 0.0025. Hence for the variation to be a winner, its potential loss must be below 0.0025, and the Probability to be best is greater than or equal to 95%

NOTE: If the threshold of caring for any test is > 1%, we will set it to 1%, and if the threshold is <0.001%, we will set it to 0.001%.

Screenshot_2019-10-09_at_3.58.16_PM.png

NOTE: The calculations of the threshold of caring was changed in September 2019. The formula earlier was- The threshold of caring = Current conversionrate * Expected lift in conversion rate * multiplier * 100 where multiplier = 0.01 for High Certainty mode; 0.075 for Balanced mode; and 0.2 for Quick Learnings mode and current conversion rate and expected lift in conversion rate were accepted from the user.

When is a Winner Retracted in VWO?
Once a variation has been declared as a winner, the decision will be reversed if the Probability to be best is below 90% or if the absolute potential loss is greater than 105% of the threshold of caring. Whenever the variation's probability to beat baseline or the potential loss breach the threshold, the Smart Decision is retracted.

NOTE: Previously, the Smart Decision was retracted when the absolute potential loss of the variation declared as Smart Decision was > 1%. This logic was ineffective in scenarios where there was a marginal difference in the conversion rate of the variations.

Articles in this section

Related articles