A/B Testing Push Notifications in VWO Engage

You can now A/B Test your VWO Engage (formerly PushCrew) notifications to see which ones perform the best! It will give you more insight on which titles, messages, images, features like CTA buttons, etc., work for your audience. A/B Testing is available on a full-fledged VWO Engage trial and Enterprise plans only.

How to A/B Testing Your Push Notification in VWO Engage?

VWO Engage lets you create 2 variations of the same notification. These variations will be sent to a sample size of your subscribers. Once sent, VWO Engage will collect click results of both variations for 1 hour. After an hour, the variant with the higher CTR will be declared the winner and will be automatically sent to the remaining subscribers.

Procedure

1. Login to your VWO Engage dashboard. Click on One-Time under Campaigns

2. Click on the Create button

3. You will see an option to Try out A/B testing. Click on it.

4. Here you will see two tabs named ‘Variant A’ and ‘Variant B’. This is where you can create two variations of the same notification. 

5. Fill in the Title, Message, Landing Page URL of both Variants A and B. You can also choose to change the image, add a Big Image and CTA Buttons. If you have set up Custom Attributes, you can use these to personalize your notifications.

6. When you switch over to Variant B, you have an option to ‘Copy from Variant A’ making it easy to fill in the details. You can then make the changes in any of the fields in Variant B. The vice versa is possible too.

7. Click on Next once you are done with creating the Variants.
   Note: If all fields in both variants are exactly the same, you will see an error message saying “The content for the A/B test variants cannot be exactly the same”. Please make the changes then and click on Next.

8. You can now select the Audience to which you want to send the notifications. You can either send it to All subscribers or any segment(s).
   Note: You need at least 100 subscribers to create an A/B test. VWO Engage will automatically determine the sample size of both Variants A and B.
   How is the sample size calculated?
   The size of the sample for A/B testing is calculated to maximize the probability of finding the winner while using the least possible number of subscribers. This way, you ensure that the better-performing message is sent to most of the audience for the campaign.

   

9. You can also decide the other Campaign Settings such as auto-hide, notification expiry, and scheduling.

10. Once done, click on Send Now.

11. You will be asked to Confirm Notification Details. Click on Send Notification once done.

12. For the one hour when the A/B testing is taking place, you will see the notification denoted with an hourglass symbol on your One-Time notifications page.
    

13. Once the one hour is completed, you will see the notification denoted by a green tick mark sign signifying that the winner variant has been sent to the rest of your subscriber base.

Reporting 

1. During the 1 hour when the A/B test is running, you will be able to see the Delivered, Total Clicks, and Click Rate data for both Variants by clicking on the notification from the One-Time page.

2. Once the 1 hour is over and notifications have been sent to the rest of your subscriber base, you can see Delivered, Total Clicks, and Click Rate data for the Overall Campaign
   

3. You will also be able to see Campaign and A/B Test information 
   

4. You can click on the individual tabs for Variant A and Variant B to see the overall performance on both.