Handling Outliers in VWO Reports using Outlier Detection

Overview

In data analysis, an outlier is a data point that differs significantly from other observations.

For example, in a set of e-commerce transactions that typically range from $50 to $200, a single large corporate order of $9,000 is a clear outlier. In testing, these extreme values, such as unusually large purchase orders or test conversions with zero revenue, can skew your campaign results, leading to inaccurate conclusions and potentially misguided business decisions.

VWO's outlier detection feature helps you maintain the accuracy of your reports by automatically adjusting these extreme values. By setting a valid range for your metric data, you ensure that outliers do not disproportionately influence the outcomes, allowing you to make decisions based on more reliable and representative data.

Using this feature, you can:

• Prevent extreme values from skewing your test results
• Base your business strategies on data that truly represents user behavior
• Avoid the need to manually filter data or re-run tests due to anomalous results

How does outlier detection work in VWO?

Outlier detection works by capping any metric value that falls outside a predefined range to the nearest boundary of that range. This means the value used in the report will be either the minimum or maximum value you have set, rather than the original extreme value.

You can define this range using two value ranges:

• Absolute Range

  You set a fixed minimum and maximum value. Any tracked value that is lower than the minimum will be capped to the minimum value, and any value higher than the maximum will be capped to the maximum value. For example, if you set a minimum of 100 and a maximum of 2000, a conversion with a value of 50 will be treated as 100, and a conversion with a value of 3500 will be treated as 2000.

• Percentile Range

  You set a lower and upper percentile bound (for example, 10th percentile and 90th percentile). VWO dynamically calculates the absolute values corresponding to these percentiles based on the data collected. This range automatically adjusts as more data flows into your report. For example, if you set the range from the 10th to the 90th percentile, VWO will calculate the actual values at these points. If the 10th percentile value is $30 and the 90th percentile value is $270, any conversion below $30 is treated as $30, and any conversion above $270 is treated as $270.

Configure Outlier Detection

Prerequisite: You can apply outlier detection only to continuous metrics, which are metrics calculated using the Value of an event property option, for example, revenue conversion. This is available only for custom events.

Configure Outlier Detection for a Data360 Metric

1. While creating metrics in VWO, go to Advanced Settings > Outliers detection, and click Modify.
   
2. From the Method dropdown, select your desired method.
   1. For Absolute range, enter a Minimum value and a Maximum value.

   2. For Percentile range, enter a Lower bound and an Upper Bound percentile.
      
      It is not mandatory to specify both minimum and maximum values. You can define only one of the two limits. For example, if you specify only the minimum value, there is no upper limit for that metric. Any incoming data greater than the minimum value is accepted as is, while values below the minimum are recalibrated to the defined minimum.

      Tip: Begin by analyzing your historical data to derive the limits you want to set. Understand your typical value distribution (for example, average, median, and standard ranges for revenue). Use this insight to define a range that accurately separates genuine outliers from standard, high-value conversions. If you are unsure what your data range will look like, begin by using the Percentile Range method (for example, 5th to 95th percentile). This allows VWO to dynamically set boundaries based on incoming data. You can always switch to a fixed Absolute Range later if the data becomes more predictable.
3. Click Create to save your metric.

This metric is now available for use in any campaign.

Configure Outlier Detection for a Campaign-Specific Metric

The process is the same as configuring outlier detection for Data360 metrics, but it is performed from within a campaign.

1. While creating a campaign in VWO Testing, Personalize, Web Rollouts, or Feature Management, proceed to the metric configuration step.
2. Click Add to create and define a new (primary or secondary) metric.
3. Once you have created the metric, follow the same steps (1 through 3) to configure the metric with your desired outlier rules.

The outlier settings will now apply only to this specific campaign.

Override Outlier Settings at the Campaign Level

For Data360 metrics, you can override the global outlier configuration at the campaign level, allowing for different outlier configurations for the same metric in different campaigns.

Let’s see an example scenario.

Global Metric Setup: Suppose you have a Data360 metric named Revenue that you use across most of your website's campaigns. You have configured its outlier detection with a global absolute range from $10 to $2,000, which is suitable for your typical B2C (business-to-consumer) audience.

Specific Campaign Need: You are now launching a new campaign specifically targeting B2B (business-to-business) clients, where you anticipate much larger order values. In this B2B campaign, an order of $5,000 is a realistic and positive outcome, not an outlier. Using the global rule would incorrectly cap this value at $2,000, making your campaign seem less successful than it is.

The Override Action: Within the B2B campaign's setup, when you add the Revenue metric, you can edit its settings specifically for this campaign. Here, you override the global rule by setting a new absolute range from $500 to $10,000.

Result: Your B2B campaign report will now correctly handle revenue values up to $10,000. Meanwhile, all your other regular campaigns that use the same Revenue metric (without an override) will continue to use the original $10 - $2,000 cap. This ensures that the outlier settings are always relevant to the specific context of each campaign.

To override a global Data360 metric’s outlier settings:

1. When configuring the metrics for your campaign, add the Data360 metric you want to use.

2. Click and select Override Outliers from the list.

   Note: The Override Outliers option appears in the list only for Data360 metrics added to the campaign.
    

   

3. In the Override Outliers pop-up, from the Outliers detection dropdown, select your desired method (either Absolute range or Percentile range) and configure the values for this specific campaign.
4. Click Save override. A tag will now appear next to the metric, indicating that its outlier settings have been overridden for this campaign.
   
5. To revert to the default Data360 outlier settings, follow the steps above to open the Override Outliers pop-up again.

6. Click Reset to Data360 defaults, and then click Save override to confirm the changes. The campaign will now use the original outlier rules from the Data360 metric.

   Attention: Changing the outlier settings for a metric that is part of a running campaign does not flush the existing data. However, the new outlier rules are applied to both past and future data, which may alter the reported statistics. A warning message will appear when you make this change, indicating that VWO will reprocess your historical data based on the updated outlier settings.

Understand Outlier Data in Your Reports

When you use a metric configured with outlier detection in a campaign, VWO ensures transparency in your reports. The report displays the data with the outlier rules applied. At the bottom of the report table, a message appears: Some conversions might have been treated as outliers based on your settings.

Click the View Data link to open a modal.

This modal provides a comparison:

• Collected Total: The raw, original data collected before any outlier rules were applied.
• Adjusted Total: The final data used in the report after capping the outlier values.

In the above screenshot,

• For the Control variation, the Collected Total revenue is $1,167,046.00, but the Adjusted Total is $1,084,559.00. This difference of over $82,000 indicates that one or more conversions with very high revenue values were capped or adjusted down to the maximum limit defined in the outlier settings.
• Similarly, for Variation 1, the revenue was adjusted from $1,379,848.00 down to $1,225,681.00.

Understand Outlier Data in Downloadable CSV Reports

The adjustments made by the outlier detection feature are also reflected when you download your campaign reports. To download your campaign report:

1. Click in the campaign overview panel.
2. Click Download CSV and select the type of report you want to download.
   

Summary

The summary CSV report provides a high-level overview of your campaign's performance. When outlier detection is active, the Metric Value column in the downloaded CSV reflects the Adjusted Total values shown in the VWO UI report.

Detailed

This format provides all visitor and conversion data in a single, comprehensive CSV file. When outlier detection is enabled, the Metric 1 Revenue and Metric 1 Total columns in this report contain the raw, unadjusted values (equivalent to the Collected Total in the VWO UI report).

Additionally, you will find another important column for each conversion:

Metric 1 Adjusted Revenue: Contains the adjusted value after the outlier rules have been applied. If a value was not an outlier, this column will show the same value as Metric 1 Total.

Example: Metric using Sum
A single visitor makes two purchases, tracked as two separate conversion events: one for $50 and another for $2500. The metric’s calculation method is Sum of an event property. The Metric 1 Total for this visitor is calculated as $50 + $2500 = $2550. The outlier rule is configured with a Maximum Value of $2000. The outlier detection logic is applied to the final sum. Therefore, the Metric 1 Adjusted Revenue in the CSV is capped at $2000.

Troubleshooting

FAQs

• Does outlier detection permanently change my raw data?
  No. Outlier detection is a reporting-layer adjustment. Your original, raw conversion data is never altered or deleted. You can always view the original values by clicking the View Data link in your report.
   
• Can I disable outlier detection for a metric after a campaign has started?
  Yes. You can edit the metric (either the Data360 metric or the campaign-specific metric) and set the Method under Outliers detection to None. The report will stop applying outlier rules from that point forward.
   
• How does outlier detection work with report segmentation?
  VWO applies outlier adjustments to the entire dataset before segmentation. When you apply a segment to your report, you are viewing a subset of the already-adjusted data. The outlier rules are not recalculated dynamically for each segment.
   
• Is it mandatory to set both minimum and maximum values?
  No, it is not mandatory to configure both ends of the spectrum. You can define only one.
   
• Can I view all updates made to a metric’s outlier configuration?
  Yes, you can view these details in the campaign’s Activity Timeline.

Need more help?

For further assistance or more information, contact VWO Support.