Configure Observatory Mode in Your Campaigns to Avoid Making Premature Decisions

Overview

Observatory Mode is a feature within VWO's SmartStats engine designed to safeguard your test results from premature conclusions. It ensures that your campaign collects a sufficient amount of data before VWO begins calculating statistics and declaring a winner. When applied, a campaign remains in observatory mode until certain data thresholds are met. This feature allows you to customize thresholds such as the minimum number of visitors per variation, the minimum number of conversions per variation, and the minimum duration (in days) for which the test should run. Access to this data enables you to better align with your specific testing needs and business cycles, thereby increasing the reliability and confidence in your test results.

For example, an online travel agency is testing a new Weekend Getaway promotion. Their visitor behavior is drastically different on weekdays compared to weekends. During the week, visitors browse, and on weekends, they book. The agency can configure the observatory mode settings such that the test remains in observatory mode for 7 days. If they start the test on a Friday, they will likely see a significant surge in bookings over the weekend, making the new promotion look like a phenomenal success. If they declare a winner on Monday, the decision would be based on skewed, peak-period data. By setting a minimum duration of 7 days, they guarantee the test runs through an entire weekly cycle. This captures both high-intent weekend bookers and lower-intent weekday browsers, averaging out their behavior. The final result is a far more accurate and trustworthy measure of the promotion's actual impact across all days of the week.

Using observatory mode, you can:

• Prevent premature decisions: Avoid the common pitfall of peeking at results early and making decisions based on insufficient data.
• Increase statistical confidence: Ensure all visitor behavior is accounted for in the experiment to achieve more reliable statistical results.
• Practice customizable control: Get the flexibility to set data collection goals at a global (account) level or a granular (campaign) level.

This article further explains how you can configure observatory mode for your test campaigns and achieve the desired results.

Configure Observatory Mode Settings

You can configure observatory mode settings at two levels:

• Account Level: These are your global default settings. Any new campaign you create (after configuring the account-level observatory mode settings) will automatically inherit these observatory mode thresholds. This configuration is useful for maintaining consistency across all your organization's tests.
• Campaign Level: These are campaign-specific settings. When creating a campaign, you can override the account-level defaults to set unique thresholds for that individual test. This provides flexibility for campaigns with special requirements.

Configure Account-Level Observatory Mode Settings

Setting defaults at the account level ensures all future campaigns start with your preferred baseline.

To configure these settings:

1. Log in to your VWO account.
2. Navigate to settings by clicking in the top right corner and selecting Settings from the menu.
3. On the left panel, click Campaigns and scroll down to the Observatory Mode section.
4. Enter your desired values for:
   1. Minimum Visitors per variation
      This is the minimum number of unique visitors that must be tracked for each variation (including the Control/Baseline) in your campaign. It ensures that a sufficient sample size has been exposed to each test version before statistical analysis begins.
   2. Minimum Conversions per variation
      This value defines the minimum number of conversions that must be recorded for each variation. It prevents VWO from analyzing results until a meaningful number of successful outcomes have occurred, which is critical for stabilizing the conversion rate and ensuring the data is reliable.
   3. Compute statistics after X days
      Specify the minimum number of days VWO must wait before starting statistical calculation. If set to 0, the calculation begins immediately after the visitor and conversion requirement is met.
      The campaign remains in observatory mode until all defined conditions (visitors, conversions, and days, based on your settings) are met.
5. Click Save to confirm your settings.

These settings will now be the default for any new campaign created in the account.

Configure Campaign-Specific Observatory Mode Settings

You can customize the observatory mode settings for each campaign before you start it.

Prerequisite

The campaign must be in the Draft state. You cannot change the campaign-level observatory mode settings once a campaign is in the Running state. This is to ensure statistical validity. The data collection thresholds must be locked before the test begins. Changing these thresholds mid-campaign would break the statistical assumptions and produce unreliable results.

To configure campaign-specific observatory mode settings:

1. When creating a new test campaign or opening a draft test, navigate to Configuration > More configurations > SmartStats Configuration.
2. Scroll down to the Observatory Mode section. The fields will be pre-filled with the current account-level defaults.
   
3. Modify the values as needed for this specific campaign.
4. To revert to the account defaults, click Reset. This allows you to discard your custom campaign settings and revert to the current account-level defaults at any time before the campaign is launched.
5. Click Save Now to confirm your updates. Proceed with setting up your campaign and launching the test.

Configuring observatory mode for a specific campaign allows you to set precise data collection goals tailored to that test's unique context. Finalize these values before starting the campaign to ensure your results are built on a solid and reliable statistical foundation.

Understand How Observatory Mode Settings Impact Your Reports

While your campaign is in observatory mode and has not yet met the defined thresholds, you will notice the following in your VWO reports:

• A banner at the top of the report will state: Collecting minimum data for statistical calculations. Hovering over the info icon will show you the specific thresholds that need to be met.
  
• The main data table will display Collecting Data for metrics such as Significance Level or Probability of Better. Hovering over the info icon reveals that the campaign is in the observatory period. VWO will conduct the statistical analysis once the observatory period ends.
  
• Features such as In-depth data review, Statistical Parameters, and graph visualizations will be disabled, as there isn't enough reliable data to populate them yet. Hovering over the toggle will display the specific thresholds that must be met.
  

Once all the conditions set in the observatory mode are met, the report will automatically update, and VWO will begin displaying the complete statistical analysis.

FAQs

• Does the observatory mode affect how traffic is allocated to my variations?
  No. Traffic allocation begins the moment your campaign is started. The observatory mode is purely a reporting and analysis function. It does not prevent users from being categorized into different variations, it only delays the calculation and display of statistical results in your reports.
   
• What is more important to set: a high Minimum Visitors per variation or a high Minimum Conversions per variation threshold?
  Both are important, but they serve different purposes.
  • Setting a high Minimum Visitors per variation ensures you have a large and representative sample of your total audience.
  • Setting a high Minimum Conversions per variation ensures the stability and reliability of your conversion rate.
     
• What happens if I set one of the values, such as 'Compute statistics after X days', to 0?
  If you set a threshold to 0, VWO's SmartStats engine will effectively ignore that specific condition. For example, if you set Compute statistics after X days to 0, the campaign will start calculating results as soon as the Minimum Visitors per variation and Minimum Conversions per variation thresholds are met, regardless of how much time has passed.

Need more help?

For further assistance or more information, contact VWO Support.