Avoiding Faulty Experiment Conduct

Fetching a reliable result from your test campaign highly depends on maintaining proper experimentation conduct throughout, otherwise, the likelihood of landing on a biased result is higher.

Why Does Experiment Conduct Matter?

Imagine experimenting to compare two website layouts. If you change the way visitors are assigned to each layout (traffic split) during the experiment, it can introduce complexities that skew the results. This is just one example of how faulty conduct can lead to misleading conclusions. This specific phenomenon is called Simpson’s paradox and you can learn more about it here.

How VWO Protects Experiment Integrity

VWO prioritizes the reliability of your experiments by monitoring visitor actions that could introduce biases. Here's how they achieve this:

• Identifying Risky Actions: VWO detects actions that might compromise data integrity, such as adjusting traffic splits or re-enabling a disabled variation.
• Friction and Context: VWO discourages you from taking risky actions by adding a layer of friction (e.g., prompting confirmation). If you persist, VWO provides contextual information alongside the results. This helps ensure decisions are made with a full understanding of any potential biases.

Checklist for Avoiding Faulty Conduct

Here are some common pitfalls to avoid during experiments:

• Changing how success is measured (metric definition) mid-test.
• Re-enabling a variation that was previously disabled.
• Altering the audience targeted by the experiment.
• Modifying the content of a variation while the experiment is running.
• Changing the traffic allocation between variations in a running experiment.

VWO helps prevent these actions by raising a warning before they occur. If you choose to proceed with a risky action, a Faulty Experimentation Conduct alert is triggered. The only way to revert such an action is by flushing all prior data, which might be unreliable due to the biased introduction.

By prioritizing proper experiment conduct and leveraging VWO's safeguarding measures, you can ensure your test results are accurate and actionable.