Biases creep into experimentation for a variety of reasons. While it is impossible to ensure that campaign results will always be trustworthy, it is far more effective to test for conditions that compromise the validity of campaign results.
The Experiment Vitals feature present at the top-right corner of your campaign report displays a set of essential checks that ensure that your campaign results are trustworthy. Whenever a check is breached, it indicates that there is a potential error in the campaign setup or the collected data.
VWO has the following checks enabled in your campaign reports that continuously run in the background and inform you if there is an issue with the campaign. The health checks are campaign-specific and metric-specific. Campaign-specific checks are common to the entire campaign, whereas metric-specific checks are applied separately to each metric.
Campaign-specific Checks
These are the essential configurational checks a campaign is expected to have. Any fault that arises will be indicated by a red cross over these checks, and the corresponding resolutions will also be provided. The checks are as follows:
| Data Tracking | Indicates the status of data tracking. When only a few or no visitors are being tracked in a campaign, it means that either there is an issue with the campaign setup or the corresponding page does not have sufficient traffic. Thus, running the campaign becomes futile. Hence, when such scenarios occur, we strongly recommend checking the campaign setup |
| 7-Day Runtime | Visitor behavior on modern websites often changes during the course of the week, revealing cyclic patterns. Therefore, it is recommended to run all A/B tests for at least 7 days to ensure the reliability of the campaign. VWO highlights if the campaign duration is less than 7 days or if you have received a crucial recommendation in less than 7 days |
| Experimentation Conduct | If changes such as metric definitions or target definitions are made in a running campaign, there is a risk that the calculated expected improvement will be biased. VWO highlights whenever actions are taken that compromise the proper conduct of experimentation in a campaign |
| Simpson’s Paradox | If the traffic split between variations is changed in a running campaign, there is a risk that the calculated expected improvement will be biased. This phenomenon is known as Simpson’s Paradox because it may lead to a situation where a variation appears to be better when, in fact, it is actually worse. In such cases, we recommend stopping the campaign and running a fresh campaign with a predefined traffic split |
Metric-specific Check
The following metric-specific check is designed to check for anomalies in the data tracked for each metric individually:
Conversions: When no conversions are being tracked in a campaign, it indicates that there may be an issue with the campaign setup or that the corresponding page does not receive any traffic. In either case, it is futile to allow the campaign to continue running, and we strongly recommend checking the campaign setup.
ATTENTION: It is strongly advised to carefully analyze the campaign results and scrutinize the final conclusion if one or more vital aspects of the campaign have been breached.