|
This article covers the following: |
Overview
Biases creep into experimentation for various reasons. While it is impossible to ensure that campaign results will always be trustworthy, it is far more effective to test for conditions that compromise their validity.
Here’s where the Experiment Vitals feature helps you monitor certain checks that ensure your campaign results are trustworthy. Whenever a check is breached, it indicates a potential error in the campaign setup or the collected data.
When you open a campaign report, you can access Experiment Vitals from the Report Header. Click the heart icon in the report header (as highlighted in the following image) to view key integrity checks such as Data Tracking, Conversion Tracking, Minimum Runtime Alert, Guardrails, Experimentation Conduct, and Sample Ratio Mismatch (SRM).
VWO has multiple campaign health checks enabled in the campaign reports. These checks continuously run in the background and inform you if there is an issue with the campaign. The health checks are campaign-specific and metric-specific. Campaign-specific checks are common to the entire campaign, whereas metric-specific checks are applied separately to each metric.
Campaign-specific Checks
These are the essential configurational checks a campaign is expected to have. A red cross over these checks indicates any fault that arises, and the corresponding resolutions will also be provided. The checks are as follows:
| Data Tracking |
It indicates whether data tracking is functioning correctly.
Without proper data tracking, your test results may be inaccurate or inconclusive. In such cases, running the campaign becomes ineffective, and we strongly recommend:
|
| Minimum Runtime | Visitor behavior on modern websites often changes during the course of the week, revealing cyclic patterns. Therefore, it is recommended to run all A/B tests for at least 7 days to ensure the reliability of the campaign results. VWO highlights if a crucial recommendation arrives in less than the minimum runtime. |
| Guardrail Checks | Guardrail metrics are the user-defined metrics that should not be negatively impacted in a campaign. If a guardrail metric is negatively impacted and is not disabled, VWO highlights it in Experiment Vitals, so that appropriate action can be taken. |
| Experimentation Conduct | If changes related to metric definitions, traffic allocation, targeting conditions, or variation content are made to a running campaign, there is a risk that the calculated expected improvement will be biased. VWO highlights whenever actions are taken that compromise the proper conduct of experimentation in a campaign |
| Sample Ratio Mismatch (SRM) |
It occurs when the actual distribution of users in an A/B test deviates from the intended allocation. This can happen due to technical bugs that prevent certain variations from loading, users deleting or blocking cookies (which affects tracking), browser or device compatibility issues, regional or time-based traffic differences, or internal employee traffic (dogfooding) interfering with the experiment. If not addressed, these factors can lead to inaccurate data and unreliable test outcomes. To learn more about the possible reasons that cause SRM in a campaign, refer to sample ratio mismatch. |
Metric-specific Check
The following metric-specific check is designed to check for anomalies in the data tracked for each metric individually:
Conversion Tracking: If no conversions are tracked in a campaign for the first three days, it indicates that there may be an issue with the campaign setup or that the corresponding goal is not receiving any conversions. In either case, it is futile to allow the campaign to continue running, and we strongly recommend checking the campaign setup.
ATTENTION: It is strongly advised to carefully analyze the campaign results and scrutinize the final conclusion if one or more vital aspects of the campaign have been breached.