Creating several variations of a lightbox is an excellent way to learn which type of message leads to the best overall performance. Digioh’s Lightbox Variations combined with our A/B Testing Wizard allow you to set up a testing environment, assess performance, and execute a winning formula. Watch the video or read the article below to learn about our A/B Testing and Lightbox variation creation features.

Please note: It is possible to run an A/B test for an inline box, but the steps are slightly different – please refer to this page for further information.

Lightbox Variations

Before you create an A/B test, you first have to create one or more variations of your lightbox. Click “+Add” in the variations column for the lightbox you wish to add a variation to.

Next, you will see a modal that allows you to create and manage your variations. There are three options available.

Option 1: New Variation from original. This will copy your original lightbox and add it as a variation. Once you click “New Variation From Original” you will be taken to the Lightbox Management screen. Here, you can click Edit under the Variations column to view and edit the lightbox variations.

Option 2: New Variation from theme. Click this to create a new variation from one of the pre-existing themes in our theme gallery.

Option 3: New Variation from another lightbox. Choose this to create a variation based on an existing lightbox. Note that once you create the variation, you can delete the original lightbox.

Tips: If you want to run an A/B Test where you evaluate the performance of a box being present or not, you can do this using the 1×1 Pixel Theme. To create this variation, select Option 2: New Variation from theme and select the 1×1 Pixel Theme (located under the Custom category). This will automatically create a variation that is essentially invisible which you can then use in your A/B test.

Variations Editor – Primary functionality

If you’re using variations, outside of an A/B test only one variation can run at a time. By default, the primary lightbox will run, and all variations will be paused. If you want one of the variations yo serve as the lightbox that remains on, click “OFF” in the Primary column that is in the row of the lightbox you’d like to turn on. You will see “OFF” turn to “ON”. Finally, publish your changes.

Tips: Rename your variations in a way that helps you remember what the difference between each variation is. Only one variation can be running at a time without A/B Testing. Make sure your lightbox is live in the lightbox section, and select “Edit” to edit variations. Finally, select which variation (if any) you’d like as the primary variation. You must publish before a variation will show as live in the variations editor. Please note that lightboxes with variations will be highlighted in a faint yellow color in the variations column.

A/B Testing

Once you create variations of a lightbox, you can A/B test any two variations. This can be the original against a single variation or any variation against another variation. Note that only two can be tested at one time. Click “A/B Testing,” then select “A/B Tests.”

The A/B Tests page allows you to see the performance of existing A/B Tests. To start a new test, click +New A/B Test.

Once in the Wizard, you can create a new A/B Test. The Wizard will guide you through these steps:

1. Name the A/B Test.
2. Select the lightbox to test (note: only lightboxes with variations will show up in the list).
3. Pick the lightbox variations (remember, you can only test two at at time).
4. Select an impressions limit. We recommend setting it to at least 1500. Look at historic page view data to get a sense of what number to set.
5. Select a date range. We recommend setting a date range of at least 7 days.

If you reach your impressions limit before the date range is complete, the test will stop running. The opposite is also true: if your date range ends before you hit your impression limit, your A/B test will end.

After completing the wizard, you must publish changes for the A/B Test to go live. A/B Tests that are live will show up with a checkmark in the “Live” column of the A/B test section.

 

Note:

• You cannot delete live lightboxes with active A/B tests.
• You cannot create more than one A/B test for the same lightbox.
• You will only be able to test Design formats. To test Conditions (like 3-second timer vs. Exit-Intent), you must use multivariate testing.

Ending A/B Tests

You can stop a test at any time, and behavior will revert to showing the “primary” variation. Usually this is the original parent box. If a test ends by reaching the impression target or end date, Digioh will automatically display the winning box (with the best conversion rate) going forward. We recommend using an A/B test significance calculator to ensure that results are statistically significant to p=0.95. In general, you should let tests run for as long as necessary to achieve significance, but understand that some tests may never achieve a significant result.

Tests are persistent at the visitor level, meaning a visitor will be tied to a particular variation for the entirety of the A/B Test’s duration. For example, say you have an A/B Test with “Original” versus “Variation 1”. A visitor sees “Original”. That visitor will continue to see “Original” as long as the A/B Test is running.