Deriving meaningful insights from your test campaigns and applying these to improve your website conversion rate is the ultimate objective of your test campaigns. The test campaign reports help you to validate the hypothesis you test and derive a better understanding of visitor behavior on your website.
To access your test report, select the campaign for which you want to view the report, and go to Report tab
To view the report corresponding to a goal, select the goal from the Goals panel. Additionally, you can also compare the data of different goals using the Compare Goals option.
NOTE: While using the Date Range Filter, Segments Filter, or Changing the
Base Variation, a loader appears on the page notifying you that the changes
are being applied based on your selection. You are not allowed to perform
any action until the results are fetched. A success message appears as soon
as the changes are applied, and the report data is updated accordingly.
View Report for a Date Range
Date filter enables you to filter data for the specified date range. To specify a date range, you can either use the existing options or you can specify a custom date range using the calendar control.
View Report by Visitor Segments
To filter the report by visitor segments, click the All Visitors drop-down menu, and then select the segment you want to apply. You can use the Segment Gallery (default VWO segments) or filter the report per the segment you want to apply.
To learn how to create a custom segment, click here.
Warnings and Notifications
The notification box displays the status of your campaign and any warnings you must be aware of.
NOTE: To declare a variation as a winner or a smart decision, VWO requires
a minimum of 25 conversions per variation, 1500 visitors for the test, and
the test must be running for at least a week.
- The notification section displays the status of the campaign and any other issues with data collection in a variation or with conversion tracking. You can also view the Winner and Smart Decision notifications here.
- Warnings are displayed if errors were made when setting up the campaign which can impact the data integrity of your report. For example, if you modify a test after it starts running.
- Insights you derive from your test report, or even fresh ideas you may want to test.
Customizing the Report View
You can customize the layout of your report using the View Settings option. To access this option, click on the gear icon present next to the stats table.
- Layout Options: From this section, you can customize the view of your table and graph reports.
- Table Options: From this section, you can add or remove columns to the report table, customize table rows to display, and update data you want to view in the report. You can also move the columns to change how these should appear in the table.
For more details, refer View Settings.
Understanding the Reports Table
The report table displays the data collected for each variation included in the campaign. By default, the control is set as the base for comparing that data. You can change the base to any other variation for calculations using the Use this as Baseline option and you can disable or preview the variations using Disable variation and Preview variation options from the vertical ellipses present at the right end of the table row.
- Expected conversion rate/Expected revenue per visitor: This is the median conversion rate you can expect from the variation. The "best case" and "worst case" conversion rates represent the 99% credible interval where the conversion rate is likely to be contained.
- Improvement/Revenue improvement: This is the median improvement you can expect over the baseline if you implement the variation. The "best case" and "worst case" values represent the 99% credible interval where improvement is likely to be contained.
- Probability to beat control: The probability of a variation to perform better than the baseline. By default, control is the baseline.
- Probability to be best: This depicts the number of times a variation performed better than all other variations including control.
- Expected revenue per conversion: This is the median revenue per visitor you can expect from the variation. The "best case" and "worst case" revenue per visitor values represent the 99% credible interval where they are most likely to be contained.
- Conversions/Visitors: The ratio of the number of actual goal conversions to the total number of page visitors.
- Absolute potential loss: This depicts the differences between all the sample sets where variation is not a winner. Absolute Potential Loss finds out how much you stand to lose in a scenario where the variation in question always loses to another variation.
NOTE: If your Absolute Potential loss is 2% and the expected conversion
rate is 10%, you still have a chance to improve this conversion rate and
increase it to 12%.
Understanding the Graph View
The graph provides information on the conversion rate, conversions, visitors, revenue per visitor, and revenue. You can view the graph in either of the following ways:
- Date Range
- Box Plot
- Probability Density
In the Date Range view, you can view the Conversion Rate, Conversions, Visitors, and Revenue per visitors' data over a period of time. You can choose a custom date range using the Date Filter. For more information on Date Filter, refer to the View Report for a Date Range section.
NOTE: To get the most actual results, it is recommended to have a diverse
set of data. For example, people tend to buy more mobile phones when the
manufacturer announces a price cut and conversion rate for that month is not the best indicator of the overall picture.
Interpreting the Change in Conversions
The Graph below shows the comparison between the Control(C) and Variation(V1). The orange line denotes the Control whereas the blue line denotes the Variation. Initially, both the Control and Variation are at mark zero. As soon the inflow of visitor traffic starts, data collection begins and the conversions are tracked. Based on the data collected over time, the change in graph pattern could be seen.
In the above graph, the x-axis represents the dates and the y-axis represents the conversions. To identify the change between the two, let’s look at the graph pattern of June 21. It clearly shows the number of conversions in the Variation is well beyond the Control. Likewise, you can interpret the graph pattern for all other date ranges.
You can also turn on ranges and see how the conversion rate range for variations have evolved over time. Ideally, the ranges become smaller over time and that should reflect in the graph. If the ranges go wide, it indicates fluctuations in the data. If the ranges overlap, it signifies uncertainty about which variation is performing better. Read more about uncertainty overlap in Box Plot section.
In the Box Plot view, you can view Revenue, Revenue Per Visitor, Conversions, Conversion Rate, and Visitors as box plots to see any kind of uncertainty overlap amongst Variations and Baseline.
Info: Uncertainty Overlap is the area where there is an overlap among variations and we are uncertain about which variation is performing better. If your best performing variation has a lot of uncertainty overlap, we recommend that you should run the test for a longer duration. To see how the uncertainty overlap appears, refer to the blue-lined between the box plots of Control and Variation in the graph below.
Interpreting the Change in Box Plot
In this graph, the box plots of Control and Variations are shown along with the uncertainty overlap. The one in orange is the box plot of Control and the other one in blue is the box plot of Variation. Based on the data collected, the Best Case, Most Likely, and Worst Case are derived. Here, the Most Likely value represents the median value of conversion rate. The Best Case and Worst Case values represent extreme cases. In the probability distribution curve below, you’ll see that Best Case and Worst Case scenarios are way less probable than the Most Likely scenario. The more the data, higher will be the probability of conversion rate to be near the Most Likely(median) value. The area above the uncertainty overlap indicates that the conversions are much higher in the Variation compared to the Control.
The probability distribution curve tells us the probability of getting a conversion rate. In VWO graphs, the x-axis represents the conversion rate and the y-axis represents the probability. The top of the curve is the conversion rate that has the highest probability of getting expected results and the range of the graph represents the 99-percentile conversion rate range.
Info: By definition, the 99 percentile conversion rate range represents a range that we are 99% confident that the true conversion rate of your variation lies in this range. For instance, VWO could have chosen to keep the conversion rate to 80 percentile or even lesser but, this would produce incorrect results. Thus, to get the most actual results, VWO has kept the conversion rate range to 99 percentile.
Interpreting the Change in Probability Density curves
Our initial guess about the conversion rate of your website is anything between 0% and 100%, this indicates the probability of any conversion number is the same. For example, the probability of the conversion rate being 4% is the same as that of 32% and so on. When you start collecting data, we use it to update our initial guess. The updated guess is what you see in the reports as Conversion Rate. With each data point, you come closer to the actual conversion rate. If you plot the Conversion Rate against it’s updated probability, you get a graph as above.
NOTE: The probability of the values that are not close to the expected
conversion rate starts to drop off and the probability of the values near
the expected conversion rate increases.
In the graph, the probability density of Control and Variations are shown along with the uncertainty overlap. The orange curve represents the Control and the other one in blue represents the curve for Variation. The median value of Variation is 2.16% whereas for the control it is 0.86%. This clearly indicates that the probability of conversion is higher in Variation. More the data, higher will be the probability of conversion rate to be near the median value.