Auto-Deploy Winning Variations from A/B Tests in Feature Experimentation

Overview

When configuring A/B tests in Feature Experimentation, you can choose to automate the winning variation's rollout using the Auto-Deploy Winning Variation option. After you enable this option, when the test concludes and VWO identifies a statistically significant winner, a new Personalization rule will automatically be created using the same configuration as the test. The new rule is placed above the test campaign in the rule evaluation order to ensure the winning variation is served. The automated workflow eliminates the manual effort and delay between discovering a winning variation and deploying it, allowing you to maximize the impact of your optimizations immediately.

Let’s say a SaaS company is A/B testing two variations of a dashboard widget, one with a simplified layout and another with advanced analytics filters.

During the experiment, Variation B (with advanced filters) shows a significant uplift in user engagement and retention. With Auto-Deploy Winning Variation enabled:

• VWO automatically creates a Personalization rule using the original test’s targeting conditions, without requiring manual intervention from product or engineering teams.
• The new campaign is automatically placed above the test campaign in the rule evaluation order to ensure the winning variation (Variation B) is served to all users who match the original test’s targeting conditions.

This article further explains how you can set up the automated workflow.

Prerequisites

1. Create a Feature Flag in VWO Feature Experimentation that represents the feature you want to release, test, or personalize.
2. Configure the variables, variations, and metrics for the flag.
3. Set up an A/B testing rule for the flag.

Enable Auto-Deploy for Winning Variations

1. Log in to your VWO account.
2. From the main panel on the left, go to Feature Experimentation > Feature Flags. Select the feature flag you want to work with.
3. Navigate to the Testing and Personalize tab, and select the A/B testing rule for which you want to configure the automated workflow.
4. Click the hamburger icon on the right, and select Modify rule.
5. On the Edit Testing Rule panel, go to Advanced Settings and check the Auto Deploy Winning Variation checkbox.
6. Click Save.
7. To start the test, ensure both the feature flag and the testing rule are active. Allow the experiment to run and collect enough data for VWO’s stats engine to declare a statistically significant winner.
   
   For more information about how VWO declares a winning variation, see Performance-based Recommendations: Winning Recommendation.
   
   As soon as a winning variation is declared, one of two triggers (whichever occurs first) activates the deployment of the automated workflow:
   • Manual Trigger: A user views the report of the completed test.
   • Automatic Trigger: A periodic background job (cron job) checks for completed tests with winners.
8. Once a trigger is launched and the auto-deploy process is triggered, you will see two changes in your rules list:
   • A new Personalization rule, typically named Auto Deploy - [Original Test Name], is created and activated. This rule is configured to:
     • Target the same audience as the original A/B test.
     • Allocate the qualifying traffic to that winning variation based on the original test’s targeting conditions.
   • The new rule is automatically placed above the test campaign in the rule evaluation order to ensure the winning variation is served. This is crucial because it prevents visitors from being bucketed into the losing variation(s), ensuring that no new users are exposed to a suboptimal experience from this point forward.
     

Henceforth, users who meet the targeting conditions will start seeing the winning variation. This automation helps you quickly apply successful changes and see faster results.

Troubleshooting

FAQs

• Does Auto Deploy work for Multivariate (MVT) tests?
  No. Currently, the Auto Deploy Winning Variation feature is designed to work only with A/B Testing rules.
   
• What happens if my A/B test ends without a clear winner?
  The Auto Deploy Winning Variation function will only trigger if VWO's stats engine declares a statistically significant winner. If the test is inconclusive, no new rule will be created.
   
• Can I edit the new Auto Deploy rule that VWO creates?
  Yes. Once created, it behaves like any other Personalization rule. You can modify its name, audience, traffic, or other settings.
   
• What happens if I manually pause the A/B test before a winner is declared?
  The Auto Deploy feature will not trigger. It only activates when VWO's stats engine declares a winner based on collected data.
   
• If my A/B test used the Auto-distribute traffic (Multi-Armed Bandit) option, will Auto Deploy still work?
  Yes. Once the Multi-Armed Bandit algorithm declares a definitive winner with sufficient confidence, the Auto Deploy process will trigger just as it would for a test with a fixed traffic split.

Need more help?

For further assistance or more information, contact VWO Support.