VWO calls a variation a Smart Decision when its potential loss is below the threshold of caring and its Probability to beat baseline is higher than 95%. Deploying the Smart Decision is a smart business decision because, in a business, you always try to maximize your output quickly. With our threshold in place, you will most likely not lose anything. And, in the best case scenario, you might even get something out of it.
NOTE: To declare a variation as a Smart Decision, VWO requires a
minimum of 25 conversions per variation, 1500 visitors for the
test, and the test must be running for at least a week.
How VWO Calculates the Threshold of Caring
The threshold of caring is the magnitude that decides whether to consider a variation to declare a Smart Decision or not.
The threshold of caring = Current conversion rate * multiplier * 100
where the multiplier is 0.001 for revenue goal and 0.0075 for non-revenue goal
In the current reporting system, the Conversion Rate of a variation is taken from the test results. Say it is 2.5%, the threshold of caring will be: 0.025 * 0.001 * 100 = 0.0025. Hence for a variation to be a Smart Decision, it’s potential loss must be below 0.0025, and the Probability to beat baseline is above 95%
NOTE: If the threshold of caring for any test is > 1%, we will set
it to 1%, and if the threshold is < 0.001%, we will set it to 0.001%.
When the potential loss of deploying a variation is small enough to be within the comfort threshold and the chance to beat baseline is above 95%, SmartStats recommends that it’s a Smart Decision to deploy this variation. VWO tests can have multiple Smart Decisions whenever more than one variation outperforms the base variation. In such cases, you can choose to deploy the variation having the least potential loss value.
NOTE: The calculations of the threshold of caring was changed in
September 2019. The formula earlier was-
The threshold of caring = Current conversion rate * Expected lift
in conversion rate * multiplier * 100
where multiplier = 0.01 for High Certainty mode; 0.075 for Balanced
mode; and 0.2 for Quick Learnings mode and current conversion rate
and expected lift in conversion rate were accepted from the user.
We do not ask for these details, we compute them based on the first
When is a Smart Decision Retracted in VWO?
Once a variation has been declared as a Smart Decision, the decision will be reversed if the Probability to beat baseline falls below 90% or if the absolute potential loss is greater than 105% than the threshold of caring. Whenever the variation's probability to beat baseline or the potential loss breach the threshold, the Smart Decision is retracted.
NOTE: Previously, the Smart Decision was retracted when the absolute
potential loss of the variation declared as Smart Decision was > 1%.
This logic was ineffective in scenarios where there was a marginal
difference in the conversion rate of the variations.