Marketing > A/B testing

Episerver CMS iconA/B testing

A/B testing is an add-on and requires a separate installation, as it is not included by default in an Episerver installation. The A/B testing add-on requires no additional license.

A/B testing lets you create variations for a number of page elements (blocks, images, content, buttons, form fields, and so on), then measure the number of conversionsA conversion is when a website visitor takes a desired action. This desired action can be, for example, buying an item, filling in a form, or clicking a link. obtained from the original (control) versus the variation (challenger). The one that generates the most conversions during the testing period is typically promoted to the design for that page, which can help you maximize your website's ROIReturn on Investment.. You can use actual, timely data to drive decisions by finding the combination that most effectively compels site visitors to take the next step.

How it works

You want to know whether a different advertisement can generate more interest from your site visitors so you create two versions of an advertisement button image that links to a target pageCommerce: A confirmation when a website visitor completes a desired action on a target page, such as posting a form or completing a store checkout. (See also conversion page.).

  1. When a visitor views an A/B test page, the visitor sees either the original (A / Control) or the variation (B / Challenger) page. A/B testing logs which version the visitor sees.

    If a visitor returns to the test page, the visitor sees the same page (A or B) throughout the duration of the test. Each visitor gets a view of either A or B but not both. However, if they clear cookies, and revisit the test page, they are considered a new visitor in the test.

  2. If a visitor clicks on the image, the target page appears and A/B testing logs the action as a conversion. (The target page may contain a form that a visitor submits to get a promised award.)
  3. When the test duration completes, the version that achieves the best results is declared the winner of the test.
  4. Depending on your site configuration, you can manually pick a winner (usually the one with most conversions), which publishes that version, or the winner is automatically published when the test completes. Test winners are only automatically published if the test results are statistically significant.

Starting an A/B test

  1. Start with a published version of a page or block as the original (A / Control). For example, you have a site devoted to air travel tips and want to get visitors interested in exploring your site. Will a fancy graphic button (variation) entice more click-throughfollowing a hypertext link to a particular website; especially a commerce site.s than a plain button (original control) that says Click here for more information?
  2. Edit the content, such as changing an advertisement button, or some text. For example:
  3. Image on page A / Control: Image on page B / Challenger:
    Episerver image Episerver image
  4. Select Publish? > A/B Test changes. Do not publish the changed page. The A/B test view appears showing A / Control and B / Challenger thumbnail images.
  5. Configure your A/B test by setting the following options:
    Test GoalEnter your hypothesis for the test. This is for your information only.
    Conversion goalSelect the key performance indicator (KPIKey Performance Indicator (KPI); lets you define and measure actions on web pages.) that you want to measure.
    • Landing page. Select a target page to which visitor is taken when the visitor clicks through. Only a click-through is counted as a conversion.
    • Site stickiness. Select a target page and a timeout period (1-60 minutes). The A/B test counts a conversion if a visitor goes from the target page to any other page on your website during the set time period. If the visitor closes the browser and then opens your target page again within the specified time period, a new page view is not counted. However, a conversion is counted if the visitor goes from the target page to another page during the second visit.
    • Add to cart. Select a product a site visitor can add to a cart. If a visitor adds that product to a cart, it is counted as a conversion.
    • Purchase product. Select a product a site visitor can buy. If a visitor buys that product, it is counted as a conversion.
    • Average order. Select this KPI to track completed orders on each of the test pages. The KPI totals up the values of all Commerce carts created by visitors included in the A/B test. The test determines which page variant creates the highest average value for all those carts when picking a winner. If a visitor creates multiple carts, all the (purchased) carts are included in the total, which means that the visitor can “convert” many times in the test duration. On Commerce websites using different currencies, the test runs a calculation to convert all carts into the same currency.

    Note: You need Episerver Commerce to use Commerce-related KPIs such as Add to Cart, Purchase Product, and Average Order.

    Participation percentageEnter the percentage of total amount of traffic to your A/B test.

    If you set it at 100%, then half of all visitors get page A and the other half get page B. However, you may not want that many visitors to see page B because you may not want half of your visitors to see something on page B that might be unsuccessful. By lowering the participation percentage, you might limit exposure of a risky test. Therefore, lowering the percentage increases the number of people who see page A, and only the percentage of people included in the test count in the statistics.

    Test durationSpecify the number of days you want the test to run.
    Start testSelect one of the options. You can stop the test at any time before the number of specified days are done.
    • Start test immediately. Select this option and click Start Test after you specified the test parameters.
    • Schedule for later. Select this option; a date picker appears. Select a date and time to start the test. Click the Schedule test button.
    Advanced Options > Confidence levelSelect the confidence level of statistical significance you want from the results that you gather. For example, if you select a 98% confidence level, the test results report whether your results are significant at the 98% confidence level.
  1. When you are done, click Start Test.

Viewing an A/B test

  1. To view a running A/B test, open the page you are testing, and click View Test on the notification bar.

    Episerver image

    The test results are displayed and a flame graphic shows which page is leading the test: .

    Beneath the two page thumbnails you can view the currently collected test data, such as number of conversions and conversion rate. The conversion rate can be either a percentage rate or an amount, if you are using the Average Order KPI. (The pie chart is not displayed for Average Order KPI.)

    If a visitor returns to the test page, the visitor sees the same page (A or B) throughout the duration of the test, and no matter how many times the visitor views the page or clicks through to the conversion goal, it counts as one view and one conversion.

  2. You can select the following actions from the Options menu:
    • Pick The Winner. If you see enough data before the test completes, you can stop the test and pick a winner. For example, perhaps the changed page is a clear runaway winner such that another few days of testing may not significantly affect the result.

      If you select Pick The Winner, the Pick the Winner page appears. Click Pick The Winner and it is automatically published. (The leading page is highlighted in green)  When you have selected a winner), the loser is added to the Versions gadget as a historical artifact.

    • Abort A/B test. Stop the test and discard the results.

Managing A/B tests

You cannot edit the test settings or the content of a page when the test is running because you could possibly invalidate any of the results. If you need to change the test settings or something on the pages, you must cancel the test and make your changes, and start the test over. Click Abort A/B Test from the Options menu.

Use the Tasks tab in the navigation pane to manage A/B testing-specific actions.


Active Tests. Displays links to active test pages that are collecting data. Click an item to display the test page where you can click the View test link to display the snapshot of result data.

Archived Tests. Displays the content that has had an A/B test run against it.

Completed Tests. Displays links to completed tests. Data is no longer being collected. A winner is not yet chosen. Click a link to pick a winner.

Scheduled Tests. Displays links to tests that are ready to run on a schedule.

For other statuses in the Tasks bar, see Controlling the publishing process.

Setting default values for A/B testing

Administrators can set the default values and decide if a test winner should be automatically published or not. See A/B Testing configuration in Tool settings.

back to top Episerver image

Help topics   Need help?   FAQs   Glossary   Site map   Videos

Episerver User Guide 17-2 | Released: 2017-03-13 | © Episerver 2017 | Send feedback to us