A/B Split Testing Crash Course

  |  April 27, 2010   |  Comments

Learn the basics of A/B split testing and the 10 advantages and disadvantages of this type of testing.

A/B split testing is the most basic landing page optimization method available. The name comes from the fact that two versions of your landing page ("A" and "B") are tested. "Split testing" refers to the random assignment of new visitors to the version of the page that they see. In other words, the traffic is split and all versions are shown in parallel throughout the data collection period (usually in equal proportions). This is an important requirement. Parallel tests should always be conducted (as opposed to one-after-the-other "sequential" ones). This allows you to control as many outside factors as possible. The random assignment of new visitors to particular landing page designs is also critical, because randomness is the basis for the probability theory that underlies the statistical analysis of the results.

Usually, version "A" is defined as your original control page, or baseline (commonly called the champion version). The other version is the alternative (commonly called the challenger). If the challenger proves to be better than the champion, the challenger replaces the champion after the test and becomes the new champion to beat in any subsequent tests.

You can have more than two versions in a split test. For example, if you had one original and two alternative versions, you would have an A/B/C split test, and so on. In practice, split tests rarely have more than 10 versions of the page. The variable in your split test can be very granular (e.g., a single change such as headline text), or it can be a whole-page redesign of your landing page that is radically different than the current page.

A/B Split Testing Advantages

Split tests have several advantages:

Ease of test design. Unlike more complicated multivariate tests, split tests do not have to be carefully designed or balanced. You simply decide how many versions you want to test, and then split the available traffic evenly among them. No follow-up tests are required to verify the results - the best performer in the test is declared the winner once enough data is collected.

Ease of implementation. Many software packages are available to support simple split tests. If you are testing granular test elements, you can design, set up your test, and collect data within minutes. This can be done in most cases without support from your IT department or others. You may even be able to collect the data you need with your existing Web analytics tools, without the use of additional landing page testing tools.

Ease of analysis. Only very simple statistical tests are needed to determine the winner. Basically, all you have to do is compare the baseline version to each challenger to see if you have reached your desired statistical confidence level.

Ease of explanation. No complicated analyses or charts are needed to present your results to others. You can simply declare that you are very confident that a particular version is better than another. You can also give a likely range of percentage improvement (based on the amount of data you have collected and the width of the error bars).

Flexibility in defining the variable values. In whole-page split tests, you have complete flexibility in how different the proposed alternatives are. For example, in one alternative, you may simply choose to test a different headline. In another you may completely restructure everything about the page (layout, color scheme, sales copy, offer, and call-to-action). This ability to mix and match allows you to test a range of evolutionary and revolutionary alternatives in one test, without being constrained by the more granular definition of variables in a multivariate test.

Useful in low data rate tests. If your landing page only has a few conversions per day, you simply cannot use more advanced tuning methods. But with the proper selection of the test variable and alternative values, you can still achieve significant results in a split test. Improvements in the double or even triple digits are not uncommon.

A/B Split Testing Disadvantages

Split tests also have several drawbacks:

Limited number of versions. The number of versions in a typical split test is usually very small. If you did your homework properly, you probably came up with dozens of potential issues with your landing page, and also constructed many alternative variations to test. However, because of the limited scope of split testing, you will be reduced to testing your ideas one at a time. You will also be forced to guess which ideas to test first (based on your intuition about which ones might make the most difference). In other tuning methods, you may be able to test many of your key ideas at once and find all of the changes that improve your conversion rate in one test.

Does not consider context or variable interactions. By definition, split tests consider only one variable at a time, so you cannot detect variable interactions (how combinations of variables influence each other). A series of split tests covering several variables is not the same as a multivariate test with the same variables. Depending on the variable interactions, you may not be able to find the best-performing combination of variables on the page at all. Whether you do depends on the order in which you conduct your split tests, and the exact nature of the interactions.

No way to discover the importance of page elements. Often, you may choose very coarse variables for your split test. Because of the limited data rate, you are forced to make your best guess at page elements that might improve performance. These elements may actually involve many simultaneous changes to your landing page. In the extreme case of a whole-page redesign, you may have changed dozens of details on the page in question and defined them as a single alternative version.

However, the same flexibility that allows you to do this also limits your ability to interpret results and attribute credit for the conversion improvement to any particular change that you made. Was it the button color? Or was it the headline change? Or was it the different offer? You will never know. By squashing multiple changes into one page, you have confounded their effects and lost the ability to look at them separately.

In practice, this may not be such a huge issue, since many of the so-called learnings about the relative importance of variables are based on the spurious assumption that they are all independent of each other. Furthermore, the biggest conversion improvements may be due to the specific variable values you have chosen, and not the variable itself. For example, a particular headline that you chose to test was very powerful. But this does not allow you to generalize about headlines being more important than the other variables tested. In any case, you should avoid trying to interpret split test results if the variable values involve changing multiple elements on the page.

Inefficient data collection. Multivariate tests are often carefully constructed in order to get the most information from a smaller data sample. In effect, they allow you to more efficiently conduct multiple split tests simultaneously, and even to detect certain kinds of variable interactions. Conducting multiple split tests back-to-back is the most wasteful kind of data collection - none of the information from a previous test can be reused to draw conclusions about the other variables that you may want to test in the future.

ClickZ Live New York Want to learn more?
Attend ClickZ Live New York March 30 - April 1. With over 15 years' experience delivering industry-leading events, ClickZ Live brings together over 60 expert speakers to offer an action-packed, educationally-focused agenda covering all aspects of digital marketing. Register today!


Tim Ash

Tim Ash is CEO of SiteTuners.com, a landing page optimization firm that offers conversion consulting, full-service guaranteed-improvement tests, and software tools to improve conversion rates. SiteTuners' AttentionWizard.com visual attention prediction tool can be used on a landing page screenshot or mock-up to quickly identify major conversion issues. He has worked with Google, Facebook, American Express, CBS, Sony Music, Universal Studios, Verizon Wireless, Texas Instruments, and Coach.

Tim is a highly-regarded presenter at SES, eMetrics, PPC Summit, Affiliate Summit, PubCon, Affiliate Conference, and LeadsCon. He is the chairperson of ConversionConference.com, the first conference focused on improving online conversions. A columnist for several publications including ClickZ, he's host of the weekly Landing Page Optimization show and podcast on WebmasterRadio.fm. His columns can be found in the Search Engine Watch archive.

He received his B.S. and M.S. during his Ph.D. studies at UC San Diego. Tim is the author of the bestselling book, "Landing Page Optimization."

Connect with Tim on Google+.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!




Featured White Papers

A Buyer's Guide to Affiliate Management Software

A Buyer's Guide to Affiliate Management Software
Manage your performance marketing with the right solution. Choose a platform that will mutually empower advertisers and media partners!

Google My Business Listings Demystified

Google My Business Listings Demystified
To help brands control how they appear online, Google has developed a new offering: Google My Business Locations. This whitepaper helps marketers understand how to use this powerful new tool.


    • Website Optimizer - SEO, CRO, Analytics
      Website Optimizer - SEO, CRO, Analytics (Marcel Digital) - ChicagoMarcel Digital, an award winning interactive marketing agency established in 2003...
    • Director of Marketing
      Director of Marketing (Patron Technology) - New YorkDirector of Marketing We are seeking a Director of Marketing to manage and build our marketing...
    • Senior Interactive Producer
      Senior Interactive Producer (Ready Set Rocket) - New YorkWhat You'll Do As a member of our team, the Senior Producer reports directly to our...