Running A/B Tests

Digioh provides powerful functionality and analytics that allows you to optimize design, content, and targeting conditions that maximize your conversion metrics. You can run A/B tests with up to 10 different boxes, and your site visitors will be randomly bucketed so that they are only eligible to see one of the boxes. The test bucket is sticky across sessions, so your visitors will get a consistent experience.

Setting up an A/B Test

Hover on A/B Testing in the top navigation bar and you’ll see a dropdown list: Current Tests and Create New Test. Click on Create New Test.

Let’s walk through the available configuration options:

1. A/B Test Name: Choose a name that is brief and for purpose. You’ll be able to see all of the configuration options later, so don’t feel that you need to add a lot of detail.
2. Overall Traffic Percent: With this option, you can have a holdout group that will not see any boxes in the A/B test, or accept the default 100%. You might want to have a large holdout group if you are concerned that the boxes might harm your site conversion.
3. Automatically Start/Finish: This optional feature allows to set a future start date, and end date, or a maximum number of box impressions for the A/B test. When an A/B test is automatically finished, the winning box will be chosen based on the conversion rate of either submissions/impressions or redirects/impressions (whichever is larger). While this feature can be useful, it may not be suitable for all situations, e.g. if your boxes have multiple submissions. When in doubt, leave this option off and review detailed Digioh analytics for the A/B test to choose a winner manually.
4. Box N: Here you choose which boxes to include in the A/B test. Boxes can only be in one A/B test at a time, and A/B tests cannot start until all boxes are turned on and published.
5. Traffic %: This is the amount of enabled overall traffic that Digioh will randomly distribute as eligibility for each box. An equal split between boxes will generate statistically significant results the fastest, but if you are concerned about potential negative impact from one or more of the boxes you can reduce their percentage. For example, you may have an already live box that you want to run against a new “challenger”, with low traffic over a longer period.

Testing Tips

Typically, before configuring an A/B test, you will duplicate an existing box from an already live box on site (do this under the actions menu in the box list). Then you can make changes to the duplicate. When running A/B tests, it’s important to test one change at a time. If you change multiple aspects of the design and conditions between boxes, you won’t know what is causing a positive (or negative) effect. Or worse, you may have one change that is positive, but is negated by another change, and so appears like neither change had any effect.

Decide what metric(s) you want to optimize, and find them in Digioh analytics. Digioh does not dictate what a box “conversion” is, you’ll need to decide that based on what’s important to your business. It may be emails collected, redirects to another page on your site, quiz completions, or some other metric.

While you can test different conditions with A/B tests, there is some nuance to this. Generally, when testing conditions changes, you should care more about the total number of conversions vs. the conversion rate. As an extreme example, if you change conditions so that a particular box receives only 20% of the impressions vs. other boxes in the test, then that box will have to perform at 5X the conversion rate to have an equal effect on your business.

Managing A/B tests

Hover on A/B Testing in the top navigation bar and you’ll see a dropdown list: Current Tests and Create New Test. Click on Current Tests. You’ll see a list of all A/B tests ever created.

Use the down chevron control on the right to expand box details:

This shows configuration and basic metrics for each box in the A/B test, links to compare boxes in more detailed Digioh analytics, and a control to declare a box as the winner. Note that when you select a box as winner, then all other boxes will be turned off. Otherwise, potentially all of the boxes could be shown to your visitors at the same time!

When A/B tests finish automatically, Digioh will start showing the winning box to all visitors. You can override this decision simply by turning off boxes other than the one you choose.