Things to Consider While Working with VWO Testing

This article outlines key considerations and best practices to keep in mind while setting up and running testing campaigns in VWO. Following these guidelines helps ensure reliable experiment results and prevents conflicts between campaigns.

• Define a Hypothesis - Optional, but Recommended
• Test One Hypothesis per Funnel
• Ensure Cookies are Enabled
• Avoid Changing Traffic Distribution During a Live Test
• Avoid Running Split URL Tests and A/B Tests on the Same URL
• Review Browser Privacy Settings - May Affect Experiment Execution
• Run Experiments for an Adequate Duration

Define a Hypothesis - Optional, but Recommended

Adding a hypothesis to your experiment is optional in VWO. However, defining a clear hypothesis helps structure your experimentation program.

A good hypothesis typically includes:

• the change being tested
• the expected outcome
• the reasoning behind the expected impact

Documenting hypotheses helps teams:

• maintain clarity on experiment goals
• analyze results more effectively
• build a knowledge base for future experimentation

For more information, see Hypotheses in VWO.

Test One Hypothesis per Funnel

When multiple experiments are run simultaneously within the same conversion funnel, changes in one experiment may affect the results of another. This is known as an interaction effect. Interaction effects can make it difficult to determine which experiment caused a particular outcome.

To reduce this risk:

• test one hypothesis at a time within the same funnel
• avoid overlapping experiments that target the same user journey

Ensure Cookies are Enabled

VWO relies on browser cookies to identify visitors and maintain consistent variation assignment during an experiment.

If cookies are disabled, VWO may not be able to:

• recognize returning visitors
• maintain variation consistency
• accurately track experiment participation and conversions

If a visitor has cookies disabled, VWO cannot reliably track their activity or maintain variation consistency.

Avoid Changing Traffic Distribution During a Live Test

Changing traffic allocation between variations after a campaign has started can introduce bias into your experiment data.

When traffic distribution is modified during an active test:

• visitors are unevenly exposed to variations over time
• statistical calculations may become unreliable
• the validity of experiment results may be affected

For this reason, it is recommended to finalize traffic distribution before launching the campaign and avoid modifying it while the test is running.

Avoid Running Split URL Tests and A/B Tests on the Same URL

Running a split URL test and an A/B test, or multiple split URL tests, or a combination of Split URL tests and A/B tests on the same control URL, can lead to experiment conflicts.

This happens because:

• Split URL tests redirect visitors to a different URL
• A/B tests modify elements on the original (control) page

If both tests target the same URL:

• visitors may be redirected before the A/B test is executed
• experiment data may become inconsistent
• visitor traffic may be wasted across overlapping tests

To avoid these issues, run only one type of test on a given URL at a time.

Review Browser Privacy Settings - May Affect Experiment Execution

Certain browser privacy settings can prevent VWO SmartCode from executing, particularly in private or incognito browsing modes. If testing campaigns are not appearing during internal validation, check the browser’s tracking protection settings.

Safari

1. Open Safari.
2. Go to Safari > Settings.
3. Select Privacy.
4. Disable Prevent cross-site tracking.

Chrome

1. Open Chrome.
2. Click the three-dot menu in the upper-right corner and select Settings.
3. Navigate to Privacy and security.
4. Click Third-party cookies.
5. Ensure that Block third-party cookies is disabled, or add your website to the Sites allowed to use third-party cookies list.

Firefox

1. Open Firefox.
2. Go to Settings.
3. Select Privacy & Security.
4. Under Enhanced Tracking Protection, choose Standard.

Alternatively, if using Custom protection:

• disable Cookies
• disable Tracking Content

These settings ensure VWO SmartCode runs properly during testing.

Run Experiments for an Adequate Duration

The duration of an experiment depends on several factors, including:

• the number of visitors to the tested pages
• the baseline conversion rate
• the expected improvement in performance

Running experiments for too short a period may lead to unreliable conclusions. To determine how long your experiment should run, use the campaign duration estimation guidelines. For more information, see Estimate Your Campaign Duration.

Need more help?

For more information or further assistance, contact VWO Support.