Set Up Multivariate Rules in Feature Experimentation

Overview

Multivariate rules are experimentation rules that enable you to run sophisticated multivariate tests directly on flag-controlled features. Multivariate Testing (MVT) is a technique in which multiple elements are changed to determine the optimal combination of those changes. It helps you identify not just which individual changes work best, but how they work together. The objective of MVT is to determine which combination of elements works best for your audience.

For example, an e-commerce brand wants to test two pricing models and a feature flag for a promotional banner.

Using MVT, they can define two variables:

Variable 1 (Pricing): $10, $20

Variable 2 (Promo Banner Flag): False, True

VWO automatically generates four combinations (2 variables x 2 values) as follows:

VWO shows each combination to a different set of visitors to identify which pair performs best. The multivariate test helps you understand how changes to one element affect the performance of another. For example, the $20 price might lead to more conversions, but only when the Promo Banner Flag is enabled. On the other hand, the $10 price might work better when shown without the banner. 

This article further explains how you can set up a multivariate test in VWO Feature Experimentation.

Prerequisites

The following prerequisites must be in place before creating a rule:

• Ensure you have:
  • An active VWO account.
  • The appropriate user permissions within VWO to create and manage feature flags and rules. You need Design-level access or higher to create a feature flag and Publish-level access or higher to publish it. For more information, see VWO Account Hierarchy.
• Install and initialize the VWO Feature Experimentation SDK in your application.
• Create a Feature Flag in VWO Feature Experimentation that represents the feature you want to release, test, or personalize. Make sure you define variables and assign more than one value to each. Using the variables and their different values, VWO automatically creates Combinations, which are served to the visitors in the experiment.
• Define the metrics relevant to your experiment's hypothesis.

Set Up a Multivariate Rule

1. Log in to your VWO account.
2. From the main panel on the left, go to Feature Experimentation > Feature Flags. Create a new feature flag or select the one you want to work with.
3. Create variables for the components of your feature that you want to control directly from the VWO dashboard without deploying new code. Go to Configuration > Variables > Add Variables. For each variable:
   1. Assign a clear Name.
   2. Select the appropriate Type: String, Boolean (true/false), Integer, Float, or JSON (for more complex, dynamic configurations).
   3. Assign the different values you want to test.
      
4. Configure the metrics you want to track to measure the impact of the feature flag. Go to Metrics and add the primary and secondary metrics. For more information on how to do that, see Working with Metrics in VWO.
5. To configure the rule, navigate to the Rules tab.
6. Choose the Environment in which you want these rules to work. You'll see options like Production (for live users), Staging, or Development.
7. On the Testing and Personalize tab, click Create new Multivariate rule.
   
8. Configure the following details for the rule:
   • Name: Enter a clear and descriptive name for your experiment.
   • (Optional) Hypothesis: Select an existing hypothesis from the dropdown to link it to your rule. This helps keep your experimentation efforts organized and tied to strategic goals.
   • Audience: Specify which users should be included in this experiment. By default, all visitors become part of the experiment. This option allows you to use VWO's default segments or create a custom visitor target segment that meets your requirements. For more information, see Using Segmentation in VWO.
   • Traffic Allocation: Specify the percentage of visitor traffic you want to include in the experiment. Move the slider from left to right to increase the traffic percentage, or manually enter the percentage in the text box.
     
   • Experimentation Mode: Choose the experimentation mode. The default selection is set to Full Factorial.
     1. Full Factorial: This method tests all possible combinations of variables. It provides the most accurate results but requires more samples and time.

     2. Partial Factorial: Tests a subset of variable combinations using optimal arrays.
        
        In this method, VWO strategically selects only a subset of combinations to test. By analyzing how these selected combinations perform, VWO can estimate the impact of all possible combinations (including the ones that are not tested), while using fewer visitors and resources.
        

        Note: To use Partial Factorial, ensure you have added at least five variables with at least two values for each variable when setting up the feature flag.
   • Combinations to serve: When you define variables and assign values while setting up the feature flag, VWO automatically generates all possible combinations of those values. Each row under Combinations to serve represents a unique combination of variable values. Hover over a combination to see the actual values used in that combination.
     
   • Distribute Traffic: Choose how to split visitor traffic among the combinations. Assign the percentage of the targeted audience traffic to each combination. You can choose from:
     • Equal Distribution: VWO automatically splits traffic evenly among all combinations.
     • Custom Distribution: Manually enter the traffic percentage for each combination. The total must sum to 100%.
     • Auto-distribute (Multi-arm bandit): VWO dynamically allocates more traffic to combinations that perform better over time, maximizing the performance of the selected goal during the experiment. This is often used when you want to optimize conversions during the test itself.
   • Force users to combinations: This option allows you to force specific users to see specific combinations instead of random combinations. Check the box, mention the user IDs, and select the combination you want them to see.
9. (Optional) Add a description: Enter a clear description for your experiment.
10. (Optional) Configure the Advanced Settings to set up your reports using advanced statistical adjustments.
    • Set the Testing Approach: When running a test, you might need to check the results frequently to make timely decisions. This can lead to confusing results and false winners. This section lets you choose from two VWO tools to help you make accurate decisions, even if you check results early or have multiple variations in your test. Choose between:
      • Fixed Horizon: The traditional approach where the experiment runs until a pre-calculated sample size or duration is met, after which statistical significance is determined.
      • Sequential Testing: Allows you to analyze results and potentially reach a conclusion faster, as significance is evaluated continuously as data comes in. It's often suitable when speed is crucial, but requires careful interpretation.
    • Apply Bonferroni Correction: This is recommended when testing multiple combinations. When making multiple comparisons, it adjusts the significance threshold to reduce the chance of incorrectly declaring a winner due to random fluctuations (false positives).
    • Pause rule once duration is completed: Check this box if you want VWO to automatically pause the experiment rule once a statistically significant winner is declared according to your settings.
    • Enter Salt Value: This ensures sticky bucketing, meaning a specific user is consistently assigned to the same combination across multiple rules and flags, providing a better user experience during the test. For more information, see Salting.
11. Create and enable the rule: Once you have configured all the settings, click Create. You are auto-navigated to the Testing and Personalize tab. Switch the toggle on to enable the rule.
    
    Ensure both the main Feature Flag and the Experimentation Rule are activated (toggle switched on) for the test to start collecting data.
    

Monitor the results of your campaign. Allow the experiment to run for a sufficient duration to collect meaningful data. To monitor the performance of your combinations and check for statistical significance, go to Feature Management and Experimentation > Flags Multivariate > Your Campaign > Reports.

Troubleshooting

FAQs

• Can I add or remove variables after launching a Multivariate test?
   No. Modifying the variables or their values after the test starts will invalidate the results. You should pause the current test and create a new one with the updated variable configuration.

Need more help?

For further assistance or more information, contact VWO Support.