Episerver CMS iconA/B testing

A/B testing lets you create variations for a number of page elements (blocks, images, content, buttons, form fields, and so on), then measure the number of conversionsThe number of visitors to a landing page in an A/B test that reach the conversion page by performing the desired action, such as posting a form. obtained from the original (control) versus the variation (challenger). The one that generates the most conversions during the testing period is typically promoted to the design for that page, which can help you maximize your website's ROIReturn on Investment.. You can use actual, timely data to drive decisions by finding the combination that most effectively compels site visitors to take the next step.

How it works

You want to know whether a different advertisement can generate more interest from your site visitors so you create two versions of an advertisement button image that links to a target pageCommerce: A confirmation when a website visitor completes a desired action on a target page, such as posting a form or completing a store checkout. (See also conversion page.).

  1. When a visitor views an A/B test page, the visitor sees either the original (A/Control) or the variation (B/Challenger) page. A/B testing logs which page view the visitor sees.

    If a visitor returns to the test page, the visitor sees the same page (A or B) throughout the duration of the test. Each visitor gets a view of either A or B but not both. However, if they clear cookies, and revisit the test page, they will be considered a new visitor in the test.

  2. If a visitor clicks on the image, the target page appears and A/B testing logs the action as a conversion. (The target page may contain a form that a visitor submits to get a promised award.)
  3. When the test duration completes, the version that achieves the best results is declared the winner of the test.
  4. When you select the winner (usually the one with most conversions), the winning version is published.

    The winner is not automatically published because your test may not achieve statistically significant results, in which case you may choose to stay with the control version. For example, if the original got 15 conversions and the variation got 17 conversions but you wanted hundreds of results, you can elect to keep the original and select it as the winner.

Creating an A/B test

  1. Start with a published version of a page or block as the original (A/Control). For example, you have a site devoted to air travel tips and information and want to get visitors interested in exploring your site. Will a fancy graphic button (variation) entice more click-throughfollowing a hypertext link to a particular website; especially a commerce site.s than a plain button (original control) that says Click here for more information?
  2. Edit the content, such as changing the an advertisement button, or some text.

    Image on Page A/Control: Episerver image

    Image on Page B/Challenger: Episerver image

  3. Select Publish > A/B Test changes. (Do not publish the changed page.)

    The A/B test view appears showing A/Control and B/Challenger thumbnail images.

  • Test Goal. Enter your hypothesis for the test. This is for your information only.
  • Conversion goal. Select the key performance indicator (KPIKey Performance Indicator (KPI); lets you define and measure actions on web pages.) that you want to measure.
    • Landing page. Select a target page to which visitor is taken when the visitor clicks through. Only a click-through is counted as a conversion.
    • Add to cart. Select a product a site visitor can add to a cart. If a visitor adds that product to a cart, it is counted as a conversion.

      You need Episerver Commerce to use Commerce-related KPIs such as Add to Cart.

    • Purchase product. Select a product a site visitor can buy. If a visitor buys that product, it is counted as a conversion.

      You need Episerver Commerce to use Commerce-related KPIs such as Purchase Product.

  • Participation percentage. Enter the percentage of total amount of traffic to your A/B test.

    If you set it at 100%, then half of all visitors get page A and the other half get page B. However, you may not want that many visitors to see page B because you may not want half of your visitors to see something on page B that might be unsuccessful. By lowering the participation percentage, you might limit exposure of a risky test. Therefore, lowering the percentage increases the number of people who see page A, and only the percentage of people included in the test count in the statistics.

  • Test duration. Specify the number of days you want the test to run.
  • Start test. Select one of the options. You can stop the test at any time before the number of specified days are done.
    • Start test immediately. Select this option and click Start Test after you specified the test parameters.
    • Schedule for later. Select this option; a date picker appears. Select a date and time to start the test. Click the Schedule test button.
  • Advanced Options.
    • Confidence level. Select the confidence level of statistical significance you want from the results that you gather. For example, if you select a 98% confidence level, the test results report whether your results are significant at the 98% confidence level.

Running the A/B test

  1. To start the test, click Start test after you specified the test parameters.
  2. When you view the page as a tester, you can view testing data by clicking View Test on the notification bar. A flame graphic shows which page is leading the test.

    Episerver image

    If a visitor views a version of either page A or page B, the page the visitor sees is always the page displayed upon return until the test is concluded. No matter how many times the visitor views the page or clicks through to the conversion goal, it counts as one view and one conversion.

  3. You can select the following actions from the Options menu:
    • Pick The Winner. If you see enough data before the test completes, you can stop the test and pick a winner. For example, perhaps the changed page is a clear runaway winner such that another few days of testing may not significantly affect the result.

      If you select Pick The Winner, the Pick the Winner page appears. Click Pick The Winner and it is automatically published. (The leading page is highlighted in green)  If you select B/Challenger (the winner), A/Control (the loser) is added to the Versions gadget as a historical artifact.

      The winner is not automatically published because your test may not achieve statistically significant results, in which case you may choose to stay with the control sample. For example, if A/Control got only 15 conversions and B/Challenger got only 17 conversions but you wanted hundreds of results, you can elect to keep A/Control and select it as the winner.

    • Abort A/B test. Stop the test and discard the results.

      You cannot edit a page that is being tested because you could possibly invalidate any of the results, so if you need to fix something on the pages you must cancel the test and edit the page, and start the test over.

Managing A/B tests

Use the Tasks tab in the navigation pane to manage A/B testing-specific actions.


Active A/B Tests. Displays links to active test pages that are collecting data. Click an item to display the test page where you can click the View test link to display the snapshot of result data.

Archived A/B Tests. Displays the content that has had an A/B test run against it.

Completed A/B Tests. Displays links to completed tests. Data is no longer being collected. A winner is not yet chosen. Click a link to pick a winner.

Scheduled A/B Tests. Displays links to tests that are ready to run on a schedule.

For other statuses in the Tasks bar, see Controlling the publishing process.

Click View test for details about the scheduled test. You cannot edit a page that is scheduled for a test (so you cannot invalid test result by making a change).

Setting default values for A/B testing

Administrators can set the default values and decide if a test winner should be automatically published or not. See A/B Testing configuration in Tool settings.

back to top Episerver image

Help topics   Need help?   FAQs   Glossary   Site map   Videos

Episerver User Guide 17-1 | Released: 2017-02-07 | © Episerver 2017 | Send feedback to us