The VWO A/B Testing report engine runs on the Bayesian statistical model that calculates the conversion rate within a range instead of an absolute number percentage. A range is a more accurate prediction of the conversion rate of an ongoing campaign where new data is collected and calculations are adjusted accordingly, while an absolute number helps in calculating the conversion rate of the data from the past.
You can use the VWO date-range graph to see how the conversion rate of each variation has evolved over the duration of the test, and then derive the following insights:
- You can analyze the conversion trends of all variations in the test, leading up to the current state of the campaign.
- Identify any sudden changes in the conversion trends during the campaign.
- Compare how overlapping in conversion rates of different variations has changed over time. You will notice that the test starts with a huge overlap in the conversion rate across the variations. The overlap may change as VWO collects more data for the test. The calculation of Winner/SmartDecision depends on the extent of overlap across variations, so looking at the overlap trends can make you dive deep into reasons why Smart Decisions/Winners are declared.
- A narrow range means that there is a lesser probability of the extremes to happen and a much higher probability for the median to be the actual conversion rate.
How the Calculation Works
Instead of expressing the number of conversions as a percentage, the VWO Report calculates a conversion range within which the true conversation rate lies with 99% probability.
So, if a reported conversion rate ranges from 3.98% to 4.34%, it means that there is a 99% probability of the actual conversion rate lying in the range 3.98–4.34%. The more data your test collects, the narrower the range becomes with the highest likely values within it.
For example, if the conversion range is in the range 6.23–10.23% and the median conversion rate is 8.23%, it means that there is a 99% chance that the conversion rate lies in this range. And in that range, 8.23% has the highest probability.
The more data your test collects, the smaller this range gets, so we can be sure of the median value.
Interpreting the Conversion Graph
Current Snapshot
Displays the conversion rate (range) for all variations and highlights the overlap across the conversion range.
Date-Range Graph
Displays the conversion rate (range) of all variations over time.
With VWO, we start with an assumption that the conversion rate can be anything from 0 to 100 and every number in that range has an equal chance of being an accurate conversion rate. Therefore, our range starts with a huge range and gradually narrows down as the data is collected over time.
As we collect more data, we are able to associate probabilities with each number from 0 to 100, giving us a probability distribution of where our actual conversion rate is. With more data, this distribution starts converging to a narrower range which in turn makes our conversion rate range smaller.
To understand your conversion range graph, you must be familiar with the following terminology:
- Best/Worst Case: The extreme range values within which we can be 99% sure that the actual conversion rate of the test lies.
- Most Likely: The median value of the conversion range which has the highest probability of being the actual conversion rate. The actual conversion rate of the test lies closer to the median, rather than exceeding the range.
Show Conversion Rate Ranges (99% Percentile Intervals)
You can toggle between ranges on the graph. When selected, you can see the conversion rate (most likely value) along with the best/worst possible values. When clear, you can see only the median values plotted over time.
By definition, percentile intervals mean that the conversion rate can lie between two numbers with the most probable scenario rates equal to the median. The further your conversion rate is away from the median, the lesser the probability of it being the correct conversion rate.