Running A/B Tests

Digioh provides powerful functionality and analytics that allows you to optimize design, content, and targeting conditions that maximize your conversion metrics. You can run A/B tests with up to 10 different campaigns, and your site visitors will be randomly bucketed so that they are only eligible to see one of the campaigns. The test bucket is sticky across sessions, so your visitors will get a consistent experience.

Setting up an A/B Test

Hover on A/B Testing in the top navigation bar and you’ll see a dropdown list: Current Tests and Create New Test. Click on Create New Test.

Let’s walk through the available configuration options:

1. A/B Test Name: Choose a name that is brief and for purpose. You’ll be able to see all of the configuration options later, so don’t feel that you need to add a lot of detail.
2. Overall Traffic Percent: With this option, you can have a holdout group that will not see any campaigns in the A/B test, or accept the default 100%. You might want to have a large holdout group if you are concerned that the campaigns might harm your site conversion.
3. Automatically Start/Finish: This optional feature allows to set a future start date, and end date, or a maximum number of campaign impressions for the A/B test. When an A/B test is automatically finished, the winning campaign will be chosen based on the conversion rate of either submissions/impressions or redirects/impressions (whichever is larger). While this feature can be useful, it may not be suitable for all situations, e.g. if your campaigns have multiple submissions. When in doubt, leave this option off and review detailed Digioh analytics for the A/B test to choose a winner manually.
4. Campaign N: Here you choose which campaigns to include in the A/B test. Campaigns can only be in one A/B test at a time, and A/B tests cannot start until all campaigns are turned on and published.
5. Traffic %: This is the amount of enabled overall traffic that Digioh will randomly distribute as eligibility for each campaign. An equal split between campaigns will generate statistically significant results the fastest, but if you are concerned about potential negative impact from one or more of the campaigns you can reduce their percentage. For example, you may have an already live campaign that you want to run against a new “challenger”, with low traffic over a longer period.

Testing Tips

Typically, before configuring an A/B test, you will duplicate an existing campaign from an already live campaign on site (do this under the actions menu in the campaign list). Then you can make changes to the duplicate. When running A/B tests, it’s important to test one change at a time. If you change multiple aspects of the design and conditions between campaigns, you won’t know what is causing a positive (or negative) effect. Or worse, you may have one change that is positive, but is negated by another change, and so appears like neither change had any effect.

Decide what metric(s) you want to optimize, and find them in Digioh analytics. Digioh does not dictate what a campaign “conversion” is, you’ll need to decide that based on what’s important to your business. It may be emails collected, redirects to another page on your site, quiz completions, or some other metric.

While you can test different conditions with A/B tests, there is some nuance to this. Generally, when testing conditions changes, you should care more about the total number of conversions vs. the conversion rate. As an extreme example, if you change conditions so that a particular campaign receives only 20% of the impressions vs. other campaigns in the test, then that campaign will have to perform at 5X the conversion rate to have an equal effect on your business.

Managing A/B tests

Hover on A/B Testing in the top navigation bar and you’ll see a dropdown list: Current Tests and Create New Test. Click on Current Tests. You’ll see a list of all A/B tests ever created.

Use the down chevron control on the right to expand campaign details:

This shows configuration and basic metrics for each campaign in the A/B test, links to compare campaigns in more detailed Digioh analytics, and a control to declare a campaign as the winner. Note that when you select a campaign as winner, then all other campaigns will be turned off. Otherwise, potentially all of the campaigns could be shown to your visitors at the same time!

When A/B tests finish automatically, Digioh will start showing the winning campaign to all visitors. You can override this decision simply by turning off campaigns other than the one you choose.