Warnings can tell you about the errors you can make due to various changes while running an experiment. For example, modifying the traffic allocation while your test was running.
These warnings tell you about the impact of the errors and how you could correct them. In VWO, if there are any warnings in your A/B test campaign, a warning icon appears on top of your reports section. You can view all your campaign warnings by clicking on it.
The warning section that you see in the reports section of your A/B test is grouped into the following three categories:
-
Simpson's Paradox
Simpson’s paradox occurs when the number of samples is not consistent for all variations throughout the test. By changing traffic split between variations in a running test, the data in each variation is not equal and hence a variation can present skewed insights when it is observed over a period of time. Say you start a test with two variations with a 90%-10% traffic split because you want to ensure that the new version doesn’t break, and after a few days, you change the traffic split to 50%-50%. You might think you were cautious of a change you made, and then you set it on the right track. But actually, many visitors have already become a part of the original version and will continue to see that and have more chances to convert.
Even if the conversion rate of variation is higher than the original because the number of visitors is more in control, the effect might reverse when the data is combined. See this table:
Day 1 (Control: 90%, Variation:10%)
Day 2 (Control: 50%, Variation 50%)
Total
Control (conversions/visitors)
40/900 = 4.45%
5/500 = 1%
45/1400 = 3.21%
Variation (conversions/visitors)
5/100 = 5%
6/500 = 1.2%
11/600 = 1.83%
Read more in these articles- The top 3 mistakes that make your A/B test results invalid and Seven Pitfalls to Avoid when Running Controlled Experiments on the Web.
We recommend that variation traffic allocation should be decided before starting the test and kept consistent throughout the test. Also, if it is important to disable a variation mid-test, you should exclude it from your data analysis, and it should not be enabled in the same test again. -
Weekday/weekend effect
Not running the test for a full week (or multiple weeks) will not include visitors evenly from all days of the week. If the conversion rate of visitors on weekends is higher than weekdays, including more weekends than weekdays can introduce a bias in the test. Read more in these articles- Not running tests for full weeks and On Heavy-user Bias in A/B Testing.
We recommend that you run the tests for whole weeks and do not pause and restart the test to avoid the day of the week effect on the results. -
Faulty experimentation conduct
If any test configuration such as target segment, goal/metrics definition, etc., is changed after the test has started, the test includes data from before and after the change is made. Doing this can pollute the data and leads to the erroneous calculation of statistical significance. In other words, when you mix the data collected with different hypotheses and contexts into one report, it won't give you any useful insights.
It is not recommended to make any configuration changes after the test has started. If this is unavoidable, we recommend that you clone this test and run the cloned version after making the desired changes to it.
NOTE: The latest warning always appears on the top of the warnings section.
How to Hide/Unhide the Warnings
Along with the categorization, VWO also enables you to hide a warning from a test report. While warnings inform you of errors you might have made while running the test, there might be instances where such changes are unavoidable. Suppose you created the test on a staging environment, and after verifying everything is working fine, you change the URL and goal/metric settings to match the production environment. This gives you a warning that you've changed your test settings after starting it. Since you know the warning is not applicable in this scenario, you can hide the warning.
To hide/unhide the warning, all you need to do is- click on the eye icon corresponding to the warning category. If you wish to view the hidden warnings as well, click on the Show hidden warnings checkbox.
If you see a warning in your test report, it means you’ve made some methodical mistake in your test campaign. The VWO app informs you about how you can resolve the warnings. To learn how you can do this, refer to the information present in the Warnings section.