A/B test

With an A/B test, you can test a mailing in different variants on a small portion of the recipient list before you start the actual dispatch. The variant that performs best in the criteria you defined is then optionally sent to the remaining recipients.

The variants (split mailings) usually differ only in few parameters from another. The most common variations include:

• Subject line. Test how different subject lines affect the opening rate.
• Header image, title, intro paragraph. Test various headers, titles and introductory paragraphs.
• Target group-specific testing. Test a mailing or a mailing variant to different target groups Subset of recipients defined by rules and conditions and a logic relationship between them. For example, all recipients in the United Kingdom. (for example, male and female recipients).
• Time of dispatch. Test the optimal time to send the master mailing.

A/B test in Smart Campaigns

To perform a A/B test in Smart Campaigns, do the following:

1. Open the Optimizely Campaign menu and select Campaigns > Smart Campaigns.
2. Click Create…
3. From the left action pane, drag the Recipients node Campaign: Item that determines when a recipient passes through the campaign, and the actions that are executed for the recipient. With the help of "connections", you can link nodes to specify the process of your campaign. The arrow at the end of each connection always points to the next node and the associated action. into the workspace and select one or more recipient lists.
4. Drag an A/B Test node from the left action area and drop it onto the desktop.
5. Connect the Recipients node to the A/B Test node.
6. Drag a message node (Email or Mobile Push) onto the desktop. Create a new message or edit an existing message.
7. Drag another message node of the same type onto the desktop. Select the message that you used for the first message node and change the content according to the criteria that you want to test.
8. Add further message nodes if necessary and follow the description in point 7.
9. Connect each message node to the A/B Test node.

   

10. In the A/B Test node, optionally activate Activate automatic optimization and/or Send time optimization. For example, you can configure that the mailing with the best unique opening rate is automatically sent to the rest of the recipient list two days after the dispatch of the split mailings. For more information, see A/B Test node and Configuring send time optimization in the A/B Test node.

    

11. In the Edit menu  of the respective connection, you can set, among other things, how many recipients should receive the split mailing. For example, if you select 10% for two split mailings, the mailing will be sent to the remaining 80% of the recipient list after the A/B test has been completed. For more information, see A/B Test node.

    

12. Click Save and Close. After the successful validation, you can activate the A/B test campaign.

A/B test for transactional mails

With an A/B test, you can send different variants of a transactional mail An email triggered by a recipient action (such as an order or purchase) or event (such as an anniversary). It is sent out subsequent to the event. and use the key figures to determine which variant performs better. To evaluate the mailing key figures, for example, you can use the Deep Analytics analysis tool.

To create an A/B test with transactional mails, do the following:

1. Open the Optimizely Campaign menu and select Campaigns > Transactional Mails.
2. Click Create….

   

3. From the left action pane, drag the Recipients node Campaign: Item that determines when a recipient passes through the campaign, and the actions that are executed for the recipient. With the help of "connections", you can link nodes to specify the process of your campaign. The arrow at the end of each connection always points to the next node and the associated action. into the workspace.
4. In the Recipient Lists drop-down list in the context menu, select a transaction recipient list.

   

   Customer support has set up a separate transaction recipient list for you. Make sure you use this transaction recipient list, instead of a regular recipient list. To see if a recipient list is a transactional recipient list, check the column Transaction API.

5. From the left sidebar, drag the A/B Test node into the workspace.
6. Connect the Recipients node Campaign: Item that determines when a recipient passes through the campaign, and the actions that are executed for the recipient. With the help of "connections", you can link nodes to specify the process of your campaign. The arrow at the end of each connection always points to the next node and the associated action. with the A/B Test node.
7. From the left sidebar, drag at least two message nodes for the desired media type into the workspace. The marketing channel Channel through which advertising messages or information are transmitted to customers and prospects. For example, email, SMS, print or push. must be compatible with recipient list type.

   You can select the following marketing channels for A/B tests with transactional mails:

   Marketing channel	Delivers	Required recipient list type
   Mail	email	Mail or multiple list
   SMS	SMS	SMS or multiple list
   Mobile Push	push message to a mobile device	Push or multiple list
   Web Push	push message that appears in the recipient's web browser	Push or multiple list

8. In the Name box in the context menu, enter a name for the transactional mail (for example Split 1).
9. In the Description box, optionally enter a description for the transactional mail (for example Optimized subject line).
10. Click Edit Content… and configure the message as described in Selecting the content type and Editing mailing content.

    

11. Connect the A/B Test node with the message nodes.
12. Click Properties  on the outgoing connections.
13. Specify to how many recipients the split mailing is to be sent.
    • Percent. Enter the percentage of recipients who should receive the split mailing.

      The sum of the percentages in all outgoing connections must be 100 percent.

    • Recipients. Enter the number of recipients who should receive the split mailing.
    • No limit. Select this option if you do not want to limit the number of recipients.

      You can select this option only for one outgoing connection.

      For example, if you enter 20% in the first split mailing and select No limit in the second split mailing, the second split mailing will be sent to the remaining 80% of recipients.

    

14. Click Save and Close.
15. To activate your transactional mails and start the sending process, select the mailing in the transactional mail overview and click Start.

To configure reports on each split mailing in Deep Analytics or to create target groups for segmentation, select the Transactional message mailing type.

A/B test in Mailings Classic

A split mailing Variation of a regular mailing where a subset of recipients is "split off". Also called split-run testing, A/B testing or bucket testing. Its purpose is to test various emails against each other to determine which version is best to use as the master mailing. in Mailings Classic is a variation of a regular mailing where a subset of recipients is "split off". You can edit every split mailing like a regular mailing and can also have a completely different look to the master mailing.

To create split mailings in Mailings Classic, do the following:

1. In the Mailings window, select the desired mailing – which is called the master mailing – and click Split. The Split Dispatch window opens.
2. You have two options.
   • Click Add split (master mailing) to create an exact copy of the master mailing with the same content. Edit this mailing by changing content elements One component of a campaign. As examples, recipients and email are elements. Nodes are used to transfer elements through the sequence of actions that occur when a campaign is activated., subject line, and so on.
   • Click Add split (new mailing) to create a new mailing with no content. Add the desired content as described in Mailings-Classic.

   In the Splits area, you can see the new split and existing splits of the selected master mailing. You can edit some parameters in the Split section of the Split Dispatch window, including:

   • Name. This name is used internally. By default, the split receives the same name as the master mailing.
   • Subject. Enter your desired variation to test different subject lines against one another.
   • Maximum number of recipients. Specify the number of recipients whom the split should be sent (relative to master mailing recipients, in absolute numbers or a percentage).
   • Target groups. Select up to three target groups for the split. See Target groups.
   • Sending date. Set the dispatch date for each split. See Dispatch order.
3. Click Save settings. The split is displayed in the mailing list.

   In the list, splits appear indented below the master mailing they belong to. In the Type column, they are designated as Split.

Recipients

To perform an A/B test, during which the best option between split A and B is selected and sent out as master mailing, splits are typically sent to a subset of total recipients. Each split can be sent to an absolute number of recipients, or a percentage of total recipients. If percentage is selected, it is based on the number of master mailing recipients. This principle also applies if a split uses a recipient list other than the master mailing.

For selecting a subset of recipients from a recipient list:

• Percentage. Enter a percentage in the Maximum recipients field. The number of recipients is based on the total number of master mailing recipients.

  Or, enter an absolute value. If the split contains the corresponding number of recipients, it is sent to exactly that number.

• Random. Recipients are randomly selected from the list. If you do not use this option, emails are sent to split recipients according to their order on the list.

In both instances, recipients who receive one split email will not be sent another. Similarly, recipients who are on several lists only receive one email. Optimizely Campaign monitors email addresses during dispatch to ensure that each recipient only receives one email (split or master mailing).

Recipient lists

In most cases, the same recipient lists are used for the master mailing and splits. However, for a split, you can use a recipient list other than master mailing. The master mailing recipient list is the basis for calculating the number of split recipients.

To use different recipient lists for the master and split mailing, in the Split Dispatch window, select off from the Automatically use the best split for the master mailing drop-down list.

If you are using different recipient lists in combination with recipient data-based target groups, be aware of potential conflicts between the different recipient lists. See Target groups.

Calculating recipients

The total number of recipients (master mailing and splits) is calculated every time before a mailing – including individual splits – is sent. The following illustration shows how subsets of recipients and the total number of recipients are calculated during dispatch.

Recipient calculation

• The symbol signifies that a subset of the total is being created.
• If a percentage is given for the number of split recipients, discrepancies between individual splits are possible, as cancellations and bounce overflows are taken into account for each case. This reduces the number of possible total recipients and the correspondingly calculated percentages.
• The number of possible recipients also considers the master mailing's target groups. If a target group with the recipient list of the master mailing contains fewer addresses than the same target group with the recipient list of the split, the number of recipients to be sent emails is the number calculated for the target group and recipient list of the master mailing.
• If the total recipient number is reached during the dispatch of a split, the subsequent splits and the master mailing are not be sent (see example 1 and example 2).

Target groups

Target groups used in the master mailing are inherited by the split mailings. You can add target groups to split mailings by linking with the target groups of the master mailing using AND logic.

The number of matches of a target group in the recipient list of the master mailing furthermore determines the maximum possible number of recipients for this target group. If the recipient list of a split contains more matching recipients for this target group, the total number of recipients sent an email will not exceed the number previously calculated on the basis of the target group and recipient list of the master mailing.

Note that the creation of target groups using recipient-based criteria can only apply when the corresponding criteria is also defined as a list box. Furthermore, if using a different recipient list for the split than for the master mailing, ensure that it incorporates the same data structure as the master mailing list – or at least that the list boxes for the master mailing target group can also be found in the split. The description of the corresponding list boxes must be an exact match, including capitalization.

Example

If a target group is used in the master mailing with the criteria Recipients with interest "Football", the recipient lists of both the master mailing and the split must contain a common corresponding field, such as Interests.

Dispatch

To send a split mailing in the Split window, click Start Split or select the check box below Mailing schedule and specify a dispatch date using the calendar.

Dispatch order

Set the order of splits in the lower section of the Split Mailing window via arrow keys to the right of each split. Splits associated with a master mailing are part of a hierarchy, which is determined by the order of the splits. This split hierarchy is referenced when calculating recipients of splits and the master mailing.

Optimizing and sending the master mailing

An A/B test lets you analyze each split's performance and send the best performing variation as a master mailing.

Optimizely Campaign also offers an automatic optimization. To activate it, select one of the following criteria from the Automatically use the best split for the master mailing drop-down list. This criterion determines which split is chosen for the master mailing.

• Best unique open rate. Split most opened by recipients (percentage).
• Best unique click rate. Split most clicked by recipients.
• Best effective click rate. Split with the best unique opens/unique clicks rate.
• Lowest unique unsubscribe rate. Split that created the fewest unsubscribes.
• Highest turnover. Split that created the highest revenue. This criterion only delivers valid results if post-click tracking Tracking method that measures and analyzes the click path that site users follow after clicking a link in an email. was configured for your client; otherwise splits have the result 0.
• Lowest bounce rate. Split that created the fewest bounces.

There are three options to start the master mailing.

• Save mailing and start it manually later. Start the master mailing manually by clearing the check boxes and selecting the default option when editing the mailing in the Options tab.
• Sending date for the master mailing. Set a sending date for the master mailing and select the check box to activate this option. Use this option if you usually send your newsletter at a certain day and time.
• Sending delay for the master mailing after the last split mailing has been sent. Set a delay between the sending of the last split mailing and the master mailing. Select the check box to activate this option.

Split mailing examples

The following examples to help you understand the split module by showing recipient calculations and their results. They illustrate the hierarchies and dependencies of the master mailing and splits.

In all examples, it is assumed that recipients are valid (that is, no one is blocked, has canceled, or exceeded the bounce limit).

Example 1

The master mailing and splits are sent to the same recipient list. A limit on the recipient number is set in the master mailing. The splits comprise A/B testing (split A against split B) with different target groups. The recipient list contains 100 recipients (40 male, 40 female, 20 unspecified).

Mailing	Recipient list	Max. number of recipients	Target group	Sent emails
Master mailing	A	28% (= 28 recipients)	—	0
Split 1	A	—	male	28
Split 2	A	—	female	0

Result: The total number of recipients, as defined in the master mailing, was reached during the dispatch of split 1. No emails are sent to Split 2 and the master mailing.

Example 2

Master mailing and splits are sent to different recipient lists. A limit for the total number of recipients is defined in the master mailing. Recipient list A (master mailing) contains 200 recipients (60 male, 60 female, 80 unspecified). Recipient list B (splits) contains 100 recipients (40 male, 40 female, 20 unspecified).

Mailing	Recipient list	Max. number of recipients	Target group	Sent emails
Master mailing	A	28% (= 56 recipients)	-	0
Split 1	B	—	male	40
Split 2	B	—	female	16

Result: The total number of recipients is determined by recipient list A and the limit of recipients defined in the master mailing. Split 1, only sent to male recipients with no sending limit, is sent to male recipients in list A. Split 2 is subsequently sent to 16 female recipients from the same list. At this point, the total number of recipients limit is reached.

Example 3

The master mailing and splits are sent to the same recipient list. Recipient list A contains 100 recipients (20 male, 20 female, 60 unspecified). Three splits are created.

• Split 1 has the target group "male"
• Split 2 "female"
• Split 3 has the target group "male OR female" (so gender must be specified).

The master mailing does not define a target group. Split 3 is to be sent first.

Mailing	Recipient list	Max. number of recipients	Target group	Sent emails
Master mailing	A	—	—	60
Split 1	A	10%	female	10
Split 2	A	10%	male	10
Split 3	A	10%	unspecified	0

Result: While the first two target groups are subsets of the master mailing target group, the target group for split 3 and the master mailing exclude one another. To select recipients who did not specify gender, the target group of the master mailing should be "males OR females OR blank field".