How to Create an A/B Test for a Mobile Application in VWO? – VWO

close this to read article

NOTE: We have upgraded our VWO FullStack and Mobile Testing products into a more powerful and unified solution: Feature Experimentation. The legacy products are now deprecated for new users. To learn more about our new offering, refer to Getting Started with Feature Experimentation.

To A/B test your mobile apps, first install the mobile app SDK in VWO. Adding the mobile app SDK is a one-time integration process, after which VWO generates API keys that you can use for app initialization for both iOS and Android apps.

NOTE: We are introducing a module called, Websites and Apps, which is available under the early access program and will be rolled out in phases. If your account is enabled with the Websites and Apps module, you can find your mobile app SDK under Configurations > Websites and Apps > Websites and Apps. Click here to know more.

After installing the SDK, you can create and run any number of A/B tests on the mobile app. To create and run A/B tests, sign in to the VWO dashboard and select Mobile App A/B on the menu. If you are using the VWO A/B testing feature for the first time, click Start Mobile App A/B Testing to begin.

Mobile App Testing developer documents are at another location. Please use the following links to access the detailed documentation and FAQs: iOS App, Android App, Cordova, React Native App, and Flutter.

To create A/B tests for mobile apps by using VWO:

Add the mobile app to test
Define the variables you want to test
Create A/B tests

Adding the Mobile App to Test

To add a new app for A/B testing by using VWO, perform these:

Procedure

Go to the Apps section on the page.
On the right side of the screen, click Create App.
Type the name of the app you want to add, and then click Create.
As you add an app, VWO generates API Keys for both the iOS and Android platforms. You can note the API Key under the Settings section to use it during app initialization.

Installing the SDK

For IOS APP

You can use Cocoapods to install the VWO iOS SDK.
For Cocoapod installation, add the vwo pod to your Pod file.
Run the pod install command. Click here for detailed installation instructions.

For ANDROID APP

You can use Gradle to install the VWO Android SDK by creating a dependency for VWO functions in the app build.gradle file.
Click here for detailed installation instructions.

Initializing the SDK

For IOS APP

After installing the SDK, you can initialize the app in the AppDelegate file using the didFinishLaunchingWithOptions method.
Import VWO, and then call the launchForAPIKey method by using your App Key.
Click here for detailed instructions.

For ANDROID APP

After installing the SDK, you can initialize the app in the onCreate(Bundle) function of the activity or the application.
Click here for detailed instructions.

Defining the Variables You Want To Test

Test variables are elements or parameters of your mobile app. After you define a variable, you can run an unlimited number of A/B tests on the variable without any code changes or redeployment. For example, you can create a string-type variable for testing different text versions on the app screen.

Under the Apps tab, select the mobile app for which you want to create test variables.
To add an element for testing, under the Variables section, click Create Variable.
Assign a name to the variable, and then select its data type.
Type Default Value (current value or value if there is no A/B test).
To add the variable, click Create. You can add multiple variables to an app.

You can view the relevant Java, Objective, or Swift code snippet in the right panel as you create the variable. You can use this code snippet to update the changes in your mobile app code.

For Example, Let us consider you want to create an A/B test for your mobile app for the following use cases:

Control: Requires users to sign in (after sign up) to the app to make a booking.
Variation: Users can directly make the payment without signing in.

To create a variable for the use case, go to the Create Variable section in the VWO Mobile App dashboard.

Enter a variable name, for example, askForLogin.
Select Data Type as Boolean.
Enter a Default Value, and then click Create. For example, if your app requires login to book, select the value as true or false if login is not required.
To test the askForLogin variable, click Add Variation to add the variation and define the variation's default value. For example, to test askForLogin without signup set the variation default value as false.

Complete the test creation process by beginning data tracking for the scenario.

Creating A/B Tests for Mobile Apps

On the Mobile App A/B testing screen, go to the Campaigns tab and click Create.
Choose the App you want to test. All mobile apps you have added to VWO are listed here.
Select a platform where the app is running.
Enter a unique identifier in the Define a campaign key field to filter your tests easily. The campaign key helps you execute custom logic, as explained in this Code Blocks section.
Click NEXT.
Click Add Variable. All the variables you have created for the test are displayed here. You can choose to Create a Variable by adding a new test variable.
Select the variable you want to test, and then enter the variation values. You can test multiple variables in one test. In the example above, we have added the speed variable, defined value as 20 for the variation. For control, the value is 10, which is the default value for the variable. VWO generates the code snippet that you can use in the mobile app based on the campaign key and variation names.
To continue, click NEXT.

Add Conversion Goals

In the Goals section, add the goals you want to measure for the test variables you added to the tests. Conversion goals help you measure the success of your test. Define goals to track the relevant conversion events for your requirement.

You can add the following goal types:

Generates Revenue: Tracks revenue from purchases or sales made through your application by defining a value for every successful transaction.
Trigger Custom Conversion: Tracks goal conversion on any custom event such as taps, page views, or any other user action. For example, you can define a custom conversions goal to track taps on your application's pop-up event or track how many users viewed the pop-up. To learn more about adding Mobile App goals, click here.

ESTIMATED DURATION

Use VWO SmartStats to calculate the estimated duration required for the A/B test to present you with the best results. Enter details about your existing app traffic to get duration estimates per your current traffic and targeted improvement.

To learn more and obtain data regarding your control page, refer to any existing analytics tool you are using (Google Analytics (GA), Mixpanel, Kissmetrics, and so on) or run a Conversion Tracking in VWO.

Average number of monthly visitors: Type the average of monthly visitors to your mobile app.
Conversion rate for Control: Type the current conversion rate of the app. Conversion rate is the number of people who complete a goal conversion against the total number of visitors.
Minimum lift in conversion rate you care about: Type the improvement in the conversion rate you want to achieve through this test. For the increase in the conversion rate, you should choose a realistic value. Usually, a 15–20% change is worth caring about. For example, if your app's current conversion rate is 6%, you may be interested only if the variation's conversion rate goes up by 20%. (A 20% increase will boost your conversion rate from 6% to 7%.) To learn more about the calculations, click here.
Certainty vs. Speed: Select how you want to configure in terms of Certainty versus Speed.

Quick learning: For finding quick trends where tests don’t affect your revenue directly. You can choose this mode for testing non-revenue goals, such as the bounce rate and time spent on a page, or for quick headline tests. Through this mode, you can reduce your testing time for non-critical tests where there isn’t a risk of hurting your revenue directly by deploying a false winner.
Balanced: Ideal for most tests. This mode is used during most of the tests. As the name suggests, it is the best balance between the testing time and minimizing the potential loss.
High certainty: Best for revenue-critical tests when you want to minimize the potential loss absolutely. In general, it takes the longest to conclude a test. Suppose you have an e-commerce website and you want to test changes to the checkout flow. You want to be sure enough to minimize the potential loss from deploying a false winner, even if it takes a lot of time. This is the best mode for such critical tests, which affects your revenue directly.

To finalize your test, click NEXT.

Configure Advanced Options

You’re almost done! You can add a name, add notes and labels, and allocate traffic percentage for the test. You can take customization further by accessing Advanced Options that provide you more control over the A/B Test.

1. ALLOCATE TRAFFIC PERCENTAGE

Specify the percentage of app visitors you want to include in your test. Click the percentage figure to change the traffic volume or move the slider to increase or decrease traffic allocation.

If you get a lot of traffic to the page you are testing, you can lower the percentage, but it might take longer to get a concrete result.

2. MORE OPTIONS

Enable campaign for a specific user group: Enable the option to target the test for a specific user group. Using the option, you can group your app users into different segments. To learn how to create visitor segments, click here.
Enable scheduling: You can schedule your test to run, pause, or stop at specific times. To learn how to schedule your test, click here.
Customize traffic percentage for each variation: You can split your app traffic among the different variations you created for the test. Splitting visitor traffic among variations ensures that all the variations you created are proportionately tested.

For example, you may want to split your traffic among three different test page variations to find out which variation gets you the maximum conversion.

Start the Mobile App A/B Test

To start the test, click START NOW. To schedule it for a later time, click START LATER.