AB Testing on Campaigns

AB Testing is a powerful tool that allows you to evaluate the most effective mix of factors for engaging with your audience. By alternating delivery times, copy, images and more, you can zero in on the best way to drive towards improved KPI's.


AB Testing Overview

There are several different campaign types you can utilize AB Testing within. In this article we are focusing on Campaigns. Voyage SMS supports AB testing for the following elements of Campaigns:

  • Message copy
  • Day of the week and time of send
  • Discount offer %
  • Image in message

You will need to use one of the AB testing segment groups (A/B/C/D) that are all 25% randomly generated sample for every AB test. In this article, we'll show you how to create these segments during new Campaign creation.

Back to Top

 


Statistical Significance when AB Testing

Please make sure you're running your experiments on enough subscribers to make good decisions. A common mistake is to stop running AB tests once a trend begins to emerge. We have a full article on Best Practices with AB Testing. In short, determine your necessary sample size in advance using tools such as this chi-square calculator. Run your test, and don't stop until you've reached or exceeded your predetermined sample size.

Back to Top

 


Creating Your First Campaign AB Test

To begin, go to Campaigns [1] on the main menu and select Create One-Time Campaign [2]:Navigate-to-Create-Campaign



Give your campaign a name [1]. We recommend including a segment designation in the name for clarity's sake. If you already have AB Testing segments created, you can chose them here [2], but we'll assume you need to create a new segment now [3]:Name-Campaign-and-Create-Segment



Name your segment appropriately [1] and ensure the match parameter is set to All [2]. Then click Add Filter [3]:Selecting-AB-Testing-Segments


You'll see a new row appear with your selection. Confirm that the connector is set to 'is' [1] and click on 'select values' [2]. From there, chose the appropriate segment for this test [3]. You'll need to create two segments and two campaigns to run an AB test. Click Create Segment [4] to continue:
Selecting-Group-A-Segment



The segment will be created, auto-selected [1], and you can click on Next [2] to continue:

Click-Next



This will put you on the message creation screen where you can define your messaging for the AB test. This is where you vary your message copy, image selection, or discount offer %:

Create-Message-AB-Test



Remember, to narrow down performance one factor, we recommend against combining variations. For example, you would not want to vary the image choice, the copy, and the send date & time all in the same AB test because you might not learn why one send is performing better than the other.

On the Preview screen, you will select the scheduled date & time [1], and click on Schedule [2] to finish:

Change-Scheduled-Time-Page

 

Again, you need to repeat these steps one more time with Segment B to have a comparison.

Back to Top

 



Campaign AB Testing Example

Let's imagine I'm lead of SMS marketing at an e-commerce company and want to maximize the click through rate on my campaigns. However, I’m not sure which message copy will perform best. Here are two copy options I want to test:

  1. Mercer and Main: Checkout our new arrivals and take advantage with 10% off your next order! https://mercerandmain.com/
  2. Mercer and Main: Checkout our new shirts that just got back in stock! https://mercerandmain.com/

These will be filled into my AB test as described above.

Back to Top

 



Reviewing AB Testing Results and Taking Action

After the two parts of your AB testing campaign have launched, you can simply navigate to the Campaigns main menu [1] and find your two test campaigns [2]. Look at the Clicks, Purchases, Revenue [3] or other metrics to determine a winner. You can see in our example, Group B performed much better than Group A:Review-Results



When examining results, remember to achieve a statistically significant number of responses.

So for our next steps, we would create a final campaign and send to Groups C and D using the same copy that we used in Group B.

Back to Top

 

 

If you have any additional questions please contact us at service@voyagesms.com.