Business To Go - Learn, Implement, Analyze, Earn - Put Your Business on the Web

Split testing provides a means of comparing two or more "things" against each other in real-time to see which (if any) resonates better with your audience.

We've relied heavily on split testing in various projects over the years to optimize our price points, feature sets, creative, and messaging to appeal to the broadest segment within our user populations.

While split testing can be complicated, it doesn't have to be. In fact arguably the best split tests are the simple ones. Simple both in the number of moving parts, and in the nature of the variables being tested against each other.

As an example, we wanted to test to see if one specific price point worked better than another. To test it we created two price points, one low, and one higher. These were going to run on our website's home page with the hope that we would be able to determine whether we could charge the higher price without suffering a drop in the take rate large enough to offset the increased revenue the higher price point would represent on a per sub basis.

Our marketing web producer built a mechanism to examine the visitor's browser for a cookie containing a previously set value (this would apply to a returning user, or someone price shopping by going back and forth in our shopping cart application adding and removing items to gauge the impact on their total price). If the cookie was not present, then a server side script would pick a random number between 1 and 4. We would serve split A - with the lower price if that value was 1 or 2. And, we would serve split B - with the higher price is that value was 3 or 4. This technique of testing two variables in four cases permits you to sanity check your results and reduce the likelihood that an anomaly has crept in to your data.

Tracking codes were created for each of the four split cases. When a split was served, we would call a pixel and pass the tracking code for that split in order to record an "impression". By comparing the impressions of all four splits as the test proceeds and at the end when making a final call it serves two purposes.

First, it would demonstrate that we served the tests equally, producing a perfect four slice pie showing 25% of the impressions in each of the four tests being served. This validated the random mechanism^*

Secondly, if the event that there are slight variations in the impressions served, having an impression count for each split allows you to calculate a click to impression ratio (the truest measure of user interest relative to the volume of the splits served). In some tests where you want to serve a disproportionate ratio of A to B for example, using the impression count in conjunction with the clicks for that split is the only way to determine how that test stacked against it's larger or smaller sibling.

When a user would click on the offer, they would start into the shopping cart process which hopefully would culminate in a completed purchase. Clicks are the truest measure of user interest. I equate clicks with votes. Keep in mind it is possible for clicks to be gamed, just as it is for other variables, but if you put aside the trust issues, clicks are the best means to identify user interest. Particularly if subsequent steps in your shopping cart may sour the sale, you don't want to count those unrelated variables against your marketing efforts. They need to be addressed, but within another scope, otherwise your tests will always fall to unchecked feature creep. When a user would click, we would record that click.

At the end of the sales process, after we successfully charged their credit card, we passed a final pixel to record a conversion. Along with the pixel we passed a dollar amount for the sale, and number of units sold. Our analytics solution paired the click that occurred steps earlier, passing the tracking code associated with the price point split so that we could correctly attribute the sale back to that originating creative element.

Within a few days of turning the test on, the data clearly indicated that more users successfully completed the sale at the lower price point. And, that the fewer conversions at the higher price point could not make up for the lower number of completed sales. So, the conclusion, while painful for the business owners, was clearly supported by data. Later different versions of the test were created, and an optimal price point was discovered slightly higher than the lower price in this first test.

Split testing offers a powerful, objective method for testing variables and their impact on adoption by users. Reports based on that data are difficult to refute, and provide compelling arguments to overcome the natural tendency of strong opinions that often share the table, but which often don't have anything but anecdotal evidence, or intuition to fall back on.

If you would like more information about split testing, here are some useful resources. If you would like us to help you create or manage a split test, contact the author with your request.

Happy hunting

* Note: usually there is slight variance, particularly if your cookied visitors return often, they can then skew your results. You need to take this into account as you construct your tests.

Put Your Business on the Web

A/B Split Testing

Additional Resources