Biases creep into experimentation for a variety of reasons. While it is impossible to ensure that campaign results will always be trustworthy, it is far more effective to test for conditions that compromise the validity of campaign results.
The Experiment Vitals feature present at the top-right corner of your campaign report displays a set of essential checks that ensure that your campaign results are trustworthy. Whenever a check is breached, it indicates that there is a potential error in the campaign setup or the collected data.
VWO has the following checks enabled in your campaign reports that continuously run in the background and inform you if there is an issue with the campaign. The health checks are campaign-specific and metric-specific. Campaign-specific checks are common to the entire campaign, whereas metric-specific checks are applied separately to each metric.
Campaign-specific Checks
These are the essential configurational checks a campaign is expected to have. Any fault that arises will be indicated by a red cross over these checks, and the corresponding resolutions will also be provided. The checks are as follows:
| Data Tracking | Indicates the status of data tracking. If the traffic is very low or there are no visitors tracked for the first three days, it means that either there is an issue with the campaign setup or the corresponding page does not have sufficient traffic. Thus, running the campaign becomes futile. Hence, when such scenarios occur, we strongly recommend checking the campaign setup |
| Minimum Runtime | Visitor behavior on modern websites often changes during the course of the week, revealing cyclic patterns. Therefore, it is recommended to run all A/B tests for at least 7 days to ensure the reliability of the campaign results. VWO highlights if a crucial recommendation arrives in less than the minimum runtime. |
| Guardrail Checks | Guardrail metrics are the user-defined metrics that should not be negatively impacted in a campaign. If a guardrail metric is negatively impacted and is not disabled, VWO highlights it in Experiment Vitals, so that appropriate action can be taken. |
| Experimentation Conduct | If changes related to metric definitions, traffic allocation, targeting conditions, or variation content are made to a running campaign, there is a risk that the calculated expected improvement will be biased. VWO highlights whenever actions are taken that compromise the proper conduct of experimentation in a campaign |
Metric-specific Check
The following metric-specific check is designed to check for anomalies in the data tracked for each metric individually:
Conversion Tracking: If no conversions are tracked in a campaign for the first three days, it indicates that there may be an issue with the campaign setup or that the corresponding goal is not receiving any conversions. In either case, it is futile to allow the campaign to continue running, and we strongly recommend checking the campaign setup.
ATTENTION: It is strongly advised to carefully analyze the campaign results and scrutinize the final conclusion if one or more vital aspects of the campaign have been breached.