A/B test

With an A/B test, you can test a mailing in different variants on a small portion of the recipient list before you start the actual dispatch. The variant that performs best in the criteria you defined is then optionally sent to the remaining recipients.

The variants (split mailings) usually differ only in few parameters from another. The most common variations include:

A/B test in Smart Campaigns

To perform a A/B test in Smart Campaigns, do the following:

  1. Open the Optimizely Campaign start menuMain screen in Optimizely Campaign, where the available options are grouped together. and select Campaigns > Smart Campaigns.
  2. Click Create…
  3. From the left action pane, drag the Recipients nodeCampaign: Item that determines when a recipient passes through the campaign, and the actions that are executed for the recipient. With the help of "connections", you can link nodes to specify the process of your campaign. The arrow at the end of each connection always points to the next node and the associated action. into the workspace and select one or more recipient lists.

    Image: Adding a Recipients node

  4. Drag an A/B Test node from the left action area and drop it onto the desktop.

    Image: Draging an A/B test node onto the desktop

  5. Connect the Recipients node to the A/B Test node.

    Image: Connecting the Recipients node to the A/B test node

  6. Drag a message node (Email or Mobile Push) onto the desktop. Create a new message or edit an existing message.

    Image: Adding a message node

  7. Drag another message node of the same type onto the desktop. Select the message that you used for the first message node and change the content according to the criteria that you want to test.

    Image: Adding and editing another message node

  8. Add further message nodes if necessary and follow the description in point 7.
  9. Connect each message node to the A/B Test node.

    Image: Connecting the message nodes to the A/B test node

  10. In the A/B Test node, optionally activate Activate automatic optimization and/or Send time optimization. For example, you can configure that the mailing with the best unique opening rate is automatically sent to the rest of the recipient list two days after the dispatch of the split mailings. For more information, see A/B Test node and Configuring send time optimization in the A/B Test node.

    Image: Configuring the A/B test node

  11. In the Edit menu  of the respective connection, you can set, among other things, how many recipients should receive the split mailing. For example, if you select 10% for two split mailings, the mailing will be sent to the remaining 80% of the recipient list after the A/B test has been completed. For more information, see A/B Test node.

    Image: Configuring the connections to the message nodes

  12. Click Save and Close. After the successful validation, you can activate the A/B test campaign.

A/B test for transactional mails

To enable this feature, contact customer support.

With an A/B test, you can send different variants of a transactional mailAn email triggered by a recipient action (such as an order or purchase) or event (such as an anniversary). It is sent out subsequent to the event. and use the key figures to determine which variant performs better. To evaluate the mailing key figures, for example, you can use the Deep Analytics analysis tool.

You cannot use transactional mails from A/B tests in Marketing Automation.

To create an A/B test with transactional mails, do the following:

  1. Open the Optimizely Campaign start menu and select Campaigns > Transactional Mails.
  2. Click Create….

    Images: Create transactional mail

  3. From the left action pane, drag the Recipients nodeCampaign: Item that determines when a recipient passes through the campaign, and the actions that are executed for the recipient. With the help of "connections", you can link nodes to specify the process of your campaign. The arrow at the end of each connection always points to the next node and the associated action. into the workspace.

    Image: Recipients node

  4. In the Recipient Lists drop-down list in the context menu, select a transaction recipient list.

    Images: Recipient list options

    Customer support has set up a separate transaction recipient list for you. Make sure you use this transaction recipient list, instead of a regular recipient list. To see if a recipient list is a transactional recipient list, check the column Transaction API.

  5. From the left sidebar, drag the A/B Test node into the workspace.
  6. Connect the Recipients nodeCampaign: Item that determines when a recipient passes through the campaign, and the actions that are executed for the recipient. With the help of "connections", you can link nodes to specify the process of your campaign. The arrow at the end of each connection always points to the next node and the associated action. with the A/B Test node.

    Image: Connecting nodes

  7. From the left sidebar, drag at least two message nodes for the desired media type into the workspace. The marketing channelChannel through which advertising messages or information are transmitted to customers and prospects. For example, email, SMS, print or push. must be compatible with recipient list type.

    You can select the following marketing channels for A/B tests with transactional mails:

    Marketing channelDeliversRequired recipient list type
    MailemailMail or multiple list
    SMSSMSSMS or multiple list
    Mobile Pushpush message to a mobile devicePush or multiple list
    Web Pushpush message that appears in the recipient's web browserPush or multiple list
  8. In the Name box in the context menu, enter a name for the transactional mail (for example Split 1).
  9. In the Description box, optionally enter a description for the transactional mail (for example Optimized subject line).
  10. Click Edit Content… and configure the message as described in Selecting the content type and Editing mailing content.

    Images: Edit content

  11. Connect the A/B Test node with the message nodes.
  12. Click Properties  on the outgoing connections.
  13. Specify to how many recipients the split mailing is to be sent.
    • Percent. Enter the percentage of recipients who should receive the split mailing.

      The sum of the percentages in all outgoing connections must be 100 percent.

    • Recipients. Enter the number of recipients who should receive the split mailing.
    • No limit. Select this option if you do not want to limit the number of recipients.

      You can select this option only for one outgoing connection.

      For example, if you enter 20% in the first split mailing and select No limit in the second split mailing, the second split mailing will be sent to the remaining 80% of recipients.

    Image: Configurng the split

  14. Click Save and Close.
  15. To activate your transactional mails and start the sending process, select the mailing in the transactional mail overview and click Start.
  16. To configure reports on each split mailing in Deep Analytics or to create target groups for segmentation, select the Transactional message mailing type.

A/B test in Mailings Classic

A split mailingVariation of a regular mailing where a subset of recipients is "split off". Also called split-run testing, A/B testing or bucket testing. Its purpose is to test various emails against each other to determine which version is best to use as the master mailing. in Mailing Classics is a variation of a regular mailing where a subset of recipients is "split off". You can edit every split mailing like a regular mailing and can also have a completely different look to the master mailing.

A split mailing is associated with a master mailing, even if the master is moved to another folder.

To create split mailings in Mailings-Classic, do the following:

  1. In the Mailings window, select the desired mailing – which is called the master mailing – and click Split. The Split Dispatch window opens.
  2. You have two options.

    In the Splits area, you can see the new split and existing splits of the selected master mailing. You can edit some parameters in the Split section of the Split Dispatch window, including:

    • Name. This name is used internally. By default, the split receives the same name as the master mailing.
    • Subject. Enter your desired variation to test different subject lines against one another.
    • Maximum number of recipients. Specify the number of recipients whom the split should be sent (relative to master mailing recipients, in absolute numbers or a percentage).
    • Target groups. Select up to three target groups for the split. See Target groups.
    • Sending date. Set the dispatch date for each split. See Dispatch order.
  3. Click Save settings. The split is displayed in the mailing list.

    In the list, splits appear indented below the master mailing they belong to. In the Type column, they are designated as Split.

Recipients

To perform an A/B test, during which the best option between split A and B is selected and sent out as master mailing, splits are typically sent to a subset of total recipients. Each split can be sent to an absolute number of recipients, or a percentage of total recipients. If percentage is selected, it is based on the number of master mailing recipients. This principle also applies if a split uses a recipient list other than the master mailing.

For selecting a subset of recipients from a recipient list:

  • Percentage. Enter a percentage in the Maximum recipients field. The number of recipients is based on the total number of master mailing recipients.

    Or, enter an absolute value. If the split contains the corresponding number of recipients, it is sent to exactly that number.

  • Random. Recipients are randomly selected from the list. If you do not use this option, emails are sent to split recipients according to their order on the list.

In both instances, recipients who receive one split email will not be sent another. Similarly, recipients who are on several lists only receive one email. Optimizely Campaign monitors email addresses during dispatch to ensure that each recipient only receives one email (split or master mailing).

Recipient lists

In most cases, the same recipient lists are used for the master mailing and splits. However, for a split, you can use a recipient list other than master mailing. The master mailing recipient list is the basis for calculating the number of split recipients.

To use different recipient lists for the master and split mailing, in the Split Dispatch window, select off from the Automatically use the best split for the master mailing drop-down list.

If you are using different recipient lists in combination with recipient data-based target groups, be aware of potential conflicts between the different recipient lists. See Target groups.

Calculating recipients

The total number of recipients (master mailing and splits) is calculated every time before a mailing – including individual splits – is sent. The following illustration shows how subsets of recipients and the total number of recipients are calculated during dispatch.

Image: Calculating recipients

Recipient calculation

  • The Optimizely image symbol signifies that a subset of the total is being created.
  • If a percentage is given for the number of split recipients, discrepancies between individual splits are possible, as cancellations and bounce overflows are taken into account for each case. This reduces the number of possible total recipients and the correspondingly calculated percentages.
  • The number of possible recipients also considers the master mailing's target groups. If a target group with the recipient list of the master mailing contains fewer addresses than the same target group with the recipient list of the split, the number of recipients to be sent emails is the number calculated for the target group and recipient list of the master mailing.
  • If the total recipient number is reached during the dispatch of a split, the subsequent splits and the master mailing are not be sent (see example 1 and example 2).

Target groups

Target groups used in the master mailing are inherited by the split mailings. You can add target groups to split mailings by linking with the target groups of the master mailing using AND logic.

The number of matches of a target group in the recipient list of the master mailing furthermore determines the maximum possible number of recipients for this target group. If the recipient list of a split contains more matching recipients for this target group, the total number of recipients sent an email will not exceed the number previously calculated on the basis of the target group and recipient list of the master mailing.

Master mailing vs. split target groups

Avoid target groups in the main and split mailings that exclude each other (for example, Male and Female), because no recipients will be selected for the split.

Note that the creation of target groups using recipient-based criteria can only apply when the corresponding criteria is also defined as a list box. Furthermore, if using a different recipient list for the split than for the master mailing, ensure that it incorporates the same data structure as the master mailing list – or at least that the list boxes for the master mailing target group can also be found in the split. The description of the corresponding list boxes must be an exact match, including capitalization.

Example

If a target group is used in the master mailing with the criteria Recipients with interest "Football", the recipient lists of both the master mailing and the split must contain a common corresponding field, such as Interests.

Dispatch

To send a split mailing in the Split window, click Start Split or select the check box below Mailing schedule and specify a dispatch date using the calendar.

Dispatch order

Set the order of splits in the lower section of the Split Mailing window via arrow keys to the right of each split. Splits associated with a master mailing are part of a hierarchy, which is determined by the order of the splits. This split hierarchy is referenced when calculating recipients of splits and the master mailing.

Split hierarchy vs. sending order

When individual splits are manually started, the split hierarchy applies, without regard for dispatch order. For example, if the last split in a list is sent first, the recipients for the split and master mailings are calculated on the basis of the overall split hierarchy.

Under certain circumstances, this may lead to a split being sent to fewer or even no recipients despite having started first. This is because the splits higher up on the list are allocated the complete number of recipients, and no recipients remain for the subordinate split. For more information, see Split mailing examples.

If the split dispatch is started automatically, the split mailings are sent in the order of the split hierarchy.

Optimizing and sending the master mailing

An A/B test lets you analyze each split's performance and send the best performing variation as a master mailing.

Optimizely Campaign also offers an automatic optimization. To activate it, select one of the following criteria from the Automatically use the best split for the master mailing drop-down list. This criterion determines which split is chosen for the master mailing.

Which criterion you use depends on mailing type, the quality of recipient data, and your marketing goals.

There are three options to start the master mailing.

  • Save mailing and start it manually later. Start the master mailing manually by clearing the check boxes and selecting the default option when editing the mailing in the Options tab.
  • Sending date for the master mailing. Set a sending date for the master mailing and select the check box to activate this option. Use this option if you usually send your newsletter at a certain day and time.
  • Sending delay for the master mailing after the last split mailing has been sent. Set a delay between the sending of the last split mailing and the master mailing. Select the check box to activate this option.

Use this option to get reliable opens and click rates based on the split mailings, because not all recipients open and click the mailing immediately upon receiving it.

Split mailing examples

The following examples to help you understand the split module by showing recipient calculations and their results. They illustrate the hierarchies and dependencies of the master mailing and splits.

In all examples, it is assumed that recipients are valid (that is, no one is blocked, has canceled, or exceeded the bounce limit).

Example 1

The master mailing and splits are sent to the same recipient list. A limit on the recipient number is set in the master mailing. The splits comprise A/B testing (split A against split B) with different target groups. The recipient list contains 100 recipients (40 male, 40 female, 20 unspecified).

Mailing Recipient list Max. number of recipients Target group Sent emails
Master mailing A 28% (= 28 recipients) 0
Split 1 A male 28
Split 2 A female 0

Result: The total number of recipients, as defined in the master mailing, was reached during the dispatch of split 1. No emails are sent to Split 2 and the master mailing.

Example 2

Master mailing and splits are sent to different recipient lists. A limit for the total number of recipients is defined in the master mailing. Recipient list A (master mailing) contains 200 recipients (60 male, 60 female, 80 unspecified). Recipient list B (splits) contains 100 recipients (40 male, 40 female, 20 unspecified).

Mailing Recipient list Max. number of recipients Target group Sent emails
Master mailing A 28% (= 56 recipients) - 0
Split 1 B male 40
Split 2 B female 16

Result: The total number of recipients is determined by recipient list A and the limit of recipients defined in the master mailing. Split 1, only sent to male recipients with no sending limit, is sent to male recipients in list A. Split 2 is subsequently sent to 16 female recipients from the same list. At this point, the total number of recipients limit is reached.

Example 3

The master mailing and splits are sent to the same recipient list. Recipient list A contains 100 recipients (20 male, 20 female, 60 unspecified). Three splits are created.

  • Split 1 has the target group "male"
  • Split 2 "female"
  • Split 3 has the target group "male OR female" (so gender must be specified).

The master mailing does not define a target group. Split 3 is to be sent first.

Mailing Recipient list Max. number of recipients Target group Sent emails
Master mailing A 60
Split 1 A 10% female 10
Split 2 A 10% male 10
Split 3 A 10% unspecified 0

Result: While the first two target groups are subsets of the master mailing target group, the target group for split 3 and the master mailing exclude one another. To select recipients who did not specify gender, the target group of the master mailing should be "males OR females OR blank field".