Creating an AB Test

Creating several variations of a lightbox with the same conversion goal/KPI is an excellent way to learn which type of message in a lightbox leads to the best overall performance. Digioh’s Lightbox Variations combined with our A/B Testing Wizard will allow you to easily set up a testing environment, quickly assess performance, and immediately exploit a winning formula. This video and article below serves as a tutorial to our A/B Testing and Lightbox variation creation features.

Lightbox Variations

Before you create an A/B test you will first have to create one or more variations of your lightbox. To do this, login to your account and click on “+Add” in the variations column for the lightbox you wish to add a variation to.

Next, you will see a modal pop over the screen that allows you to create and manage your lightbox variations. There are three options available when adding a new variation.

Variation Option 1: New Variation from original. This will create a carbon copy of your original lightbox and add it as a variation. Once you Click “New Variation From Original” you will be taken to the Lightbox Management screen, and you can click on Edit under the Variations column to view and edit the lightbox variations.

Variation Option 2: New Variation From Theme. Click this option if you want to create a new variation from one of the pre-existing themes we have in our theme gallery.

Variation Option 3: New Variation From Another Lightbox. Choose this if you’d like to create a variation based off another lightbox that you’ve already created. Note that once you’ve created the variation, you can delete the original lightbox.

Variations Editor – “Primary” functionality

If you’re using variations, outside of an A/B test only one variation can be running at a time. By default, the primary lightbox will be running and all variations will be paused. If you’d like to have one of the variations serve as the lightbox that remains on, click “OFF” in the Primary column that is in the row of the lightbox you’d like to turn on. You will see “OFF” turn to “ON”. Finally, PUBLISH YOUR CHANGES.

Variation Editor Tips — You can use the rename functionality to rename your variations. We recommend using this feature to name variations in a way that helps you remember what the difference between each variation is. Only one variation can be running at a time without A/B Testing. Make sure your lightbox is live in the lightbox section and select “Edit” to edit variations. Finally, select which variation (if any) you’d like as the primary variation. You must publish before a variation will show as live in the variations editor. Please note that lightboxes that have variations will show a up has highlighted in a faint yellow color in the variations column.

A/B Testing

Once you create variations of a lightbox, you can test any two variation. This can be a the original against a single variation or any variation against another variation. Note that only two revisions can be tested at one time.  To go in the A/B Test Editor, login to Digioh and click on “A/B Testing”. Next, select “A/B Tests.”

You will then be taken to a page that allows you to see the performance of your existing A/B Tests. To start a new test click on “+New A/B Test”

Once in the Wizard  you will now be able to start a new A/B Test. Just follow the Wizard which requires you to:

1. Name the A/B Test

2. Select which lightbox you would like to test – Note that only lightboxes with variations will show up in the selectable list

3. Pick the lightbox variations – Remember you can only test two at at time

4. Select an impressions limit – Tis defaults to 500, but we recommend setting this value to at least 1500. It’s a good idea to look at historic page view data to get a good idea of what number you’d like to set

5. Select a date range – We recommend setting a date rage of at least 7 days.

Note that if your impressions limit is complete before the date range is complete, the test will stop running. The same is also true the other way around. If your date range ends before you hit your impression limit, your A/B test will also be complete. Also, note that there is a progress bar at the top of the wizard that shows you how far you are in the setup process.

Once you’ve completed the wizard you must publish changes for the A/B Test to go live. A/B Tests that are live will show up with checkmark next to them in the “Live” column in the A/B test section.


Note that:

  • You will not be able to delete lightboxes that are live with active A/B tests
  • You cannot create more than one A/B test for the same lightbox
  • You will only be able to test Design formats. If you want to test Conditions (like 3 second timer vs. Exit Intent) you will need to use our multi variate testing features.

Ending Tests

You can stop a test at any time, and behavior will revert to showing the “primary” variation. Usually this is the original parent box. If a test ends naturally, by reaching the impression target or end date, then Digioh will stop randomizing the boxes and automatically display the winning box (with the best conversion rate) going forward. We recommend using an AB test significance calculator to ensure that the results as significant to p=0.95. In general, you should let tests run for as long as necessary to achieve significance, but understand that some tests may never achieve a significant result.

After an AB Test is completed it will automatically show the winner.

Tests are persistent at the visitor level. Meaning a visitor will be tied to a particular Box within the AB Test, for the entirety of the AB Test duration timeframe.

For example, let’s say you have an AB Test with “Original” versus “Variation 1”. A visitor comes to the site and sees “Original”. That visitor will only continue to see “Original” as long as the AB Test is running. They will not be shown “Variation 1” at all, once they are tied to “Original”.