APPLICABILITY: A/B Testing, Split URL Testing, Mobile App A/B Testing, FullStack Feature Testing, and FullStack A/B Testing.

Deriving meaningful insights from your test campaigns and applying them to improve your website conversion rate is your test campaigns' ultimate objective. The test campaign reports help you validate the hypotheses you test and better understand your website visitors’ behaviour. 

Accessing Your Test Reports

To access your test campaign report, go to the respective campaign and click the Report tab.

Components of a Report

• Data Filter
• Timeline
• Data Table
• Performance Graphs
• Report Configuration Setting
• Report Status Features
• Heatmaps

Filter Your Report Data

By default, VWO presents the data from the time the campaign started. However, you can keep it specific to a certain date and segment of visitors. Additionally, you can apply visitor dimension to tune your focus to those areas which demand your attention.

Set the Date Range

It’s pretty straightforward. Just click on the date filter, set the date range for which you want to view the report and click Filter. To see the historical data for recent times, you can also use the quick filters - Today, Yesterday, Last 7 days, and Last 30 days. 

Specify the Visitor Segment

Read the behaviour of different cohorts towards your variations by creating different combinations of segmentation criteria to gain their insight by clicking All Visitors. Doing so displays the following sections:

• Standard Segments - Features predefined segment templates that feature system attributes you can apply by selecting the appropriate one.

• My Segments - Features the custom segments you created in the Custom Segments section and saved. In the case of a Data360 account, you can also see the segments you have created in Data360 > Segments.
  
  
• Custom Segment - Allows you to create custom segments using system attributes along with custom attributes (for Data360 accounts). 

To know how to apply these post-segmentation options, refer to this article.

Specify Visitor Dimensions

Visitor Dimensions are those standard visitor attributes with which you can classify your campaign report. These become the first column of your report. These include the country they came from, the device and the browser they used, and similar data points. To filter the report by visitor dimensions, click Visitor Dimensions and select the visitor dimension you want to apply. To know more about creating and applying Visitor Dimensions, refer to Breakdown Test Reports by Visitor Dimensions in VWO.

NOTE: You can add up to two visitor dimensions to your report.

Choose the Metric

Reports are presented for individual metrics. By default, you get to see the report for the primary metric. It’s simply significant because it’s the one based on which the campaign decisions are made. To view the report for other metrics, click the corresponding from the left.

Read the Timeline

Timeline gives you a condensed overview of the entire campaign journey. You get to know the current status and the historical points of notable moments of the campaign. When you’re on a fast track, or you just need to get an overall view of things, the timeline comes to your aid with the following data:

• Day count
• Time of duration calculation
• Decision timestamps for individual variations
• Final campaign decision

Know Your Report 

The report features the individual performances of each variation (control and variations) in tabular and graphical formats. 

Report Table

The report table presents the numerical values of the performances of the variations, providing scores for the following parameters:

 

In-Depth Data Review

This view enables you to get a detailed look at the report values. Switching this ON will give you the following additional details:

• Expected improvement displayed as a graph with ranges: The Expected Improvement column of the report table will display values in a graph in the range of -98% to 98%.
  
  
• Statistical settings used for your campaign: The Probability of Improvement column will display the following statistical settings:
  
  • • MDE - Minimum Detectable Effect
    • ROPE - Region of Practical Equivalence
    • Power - Statistical Power
    • FPR - False Positives Rate

You can adjust these parameters to configure the duration of your campaign. Remember, higher statistical accuracy would demand more samples to be evaluated and thereby consume a longer time to establish the results.

INFO: To well understand these parameters, refer to this article.

 

Take Actions on Your Variations

For Control

 

For a Variation

 

General Recommendations

• Allow your test campaigns to run for at least 7 days to account for weekly variations.
• Disable a variation when its probability of improvement goes below FPR.

NOTE: VWO declares the first variation, whose probability of improvement surpasses (100 - FPR)%, as the campaign winner. Subsequent variations may exhibit smaller uplifts even if they eventually win in the test.

Arrange Your Report Data

Sort

To compare the data within the variations, you can sort them in ascending or descending order for any given parameter at a time. Just click on the required parameter on the header row to toggle between the orders.

Filter

For a focused study, you can filter the variations. Click Variations on the header row, select the variations you want to focus on and click Apply.

Graphs

The performances of your variations are represented in the following perspectives:

Date Range

It features each variation’s daily and cumulative performance on the Y-axis over a range of dates on the X-axis. 

The performance measures that you get to see in this graph format are:

• The Conversion Rate percentage (You can also see the ranges of the conversion rates by selecting the Show ranges checkbox)
• The number of Unique Conversions
• The number of Visitors

By default, you will see the graph for all the variations. To view specific variations, keep the corresponding checkboxes selected for those variations at the bottom of the graph.

Add Annotations

For a collaborated study, you can add annotations to the dates on the graph. Hover on that date on the X-axis and click on the ‘+’ icon that appears. In the callout popup that appears, enter your annotation for that date. You can also change the date from the date dropdown to apply the annotation to that date. 

To add more annotations, click Add Annotation. You can edit or delete the annotation by clicking on the corresponding icons in the popup.

You can always access and manage your annotations from the Annotations section that appears below your graphs.

Probability Density

This “scarily-unfamiliarly-named” graph is indeed your ally. It gives you a deeper insight into the certainty of a variation to become the best performer.

The X-axis shows the conversion rates. The Y-axis shows the probability density (PD). The higher the PD, the higher the chance of a variation becoming the best performer for the given conversion rate.

INFO: To better understand the functionality of the graphs in your report, refer to this article.

Tailor the Look of Your Report

The View Settings option allows you to configure your report according to your preferred appearance. 

Click on the gear icon present at the top right of the report table to access the View Settings popup. It features two sections - Layout options and Table options.

By default, VWO displays the report table first before the graph. If you depend more on the graphs, you can choose the Graphs first option. Also, you can choose which graph to be displayed by default by selecting the appropriate option under Default graph.

You generally get to see the median values of the conversion and the improvement rates. If you want to keep track of the ranges of these parameters, you can select the Expected conversion rate and Expected improvement under Bayesian ranges.

Besides, if you don’t need the Total row or wish to hide the row for a disabled variation, deselect the Show 'total' row and Show rows for disabled variations options.

Once you’re done configuring, click Save. To rest to default, click on the Restore Defaults button.

Report Status Features

The report table comes with the following features along with the campaign stats:

 

Heatmaps

The Heatmaps section allows you to view the intensity of the visitors’ interaction for the given area on your webpage that is part of your campaign. It displays the density of clicks, count of clicks, scrolls, comparison of engagement among selected areas on a webpage, and clicks on specific elements on a webpage. To know more, refer to this article.