A/B Split Testing Crash Course

  |  April 27, 2010   |  Comments

Learn the basics of A/B split testing and the 10 advantages and disadvantages of this type of testing.

A/B split testing is the most basic landing page optimization method available. The name comes from the fact that two versions of your landing page ("A" and "B") are tested. "Split testing" refers to the random assignment of new visitors to the version of the page that they see. In other words, the traffic is split and all versions are shown in parallel throughout the data collection period (usually in equal proportions). This is an important requirement. Parallel tests should always be conducted (as opposed to one-after-the-other "sequential" ones). This allows you to control as many outside factors as possible. The random assignment of new visitors to particular landing page designs is also critical, because randomness is the basis for the probability theory that underlies the statistical analysis of the results.

Usually, version "A" is defined as your original control page, or baseline (commonly called the champion version). The other version is the alternative (commonly called the challenger). If the challenger proves to be better than the champion, the challenger replaces the champion after the test and becomes the new champion to beat in any subsequent tests.

You can have more than two versions in a split test. For example, if you had one original and two alternative versions, you would have an A/B/C split test, and so on. In practice, split tests rarely have more than 10 versions of the page. The variable in your split test can be very granular (e.g., a single change such as headline text), or it can be a whole-page redesign of your landing page that is radically different than the current page.

A/B Split Testing Advantages

Split tests have several advantages:

Ease of test design. Unlike more complicated multivariate tests, split tests do not have to be carefully designed or balanced. You simply decide how many versions you want to test, and then split the available traffic evenly among them. No follow-up tests are required to verify the results - the best performer in the test is declared the winner once enough data is collected.

Ease of implementation. Many software packages are available to support simple split tests. If you are testing granular test elements, you can design, set up your test, and collect data within minutes. This can be done in most cases without support from your IT department or others. You may even be able to collect the data you need with your existing Web analytics tools, without the use of additional landing page testing tools.

Ease of analysis. Only very simple statistical tests are needed to determine the winner. Basically, all you have to do is compare the baseline version to each challenger to see if you have reached your desired statistical confidence level.

Ease of explanation. No complicated analyses or charts are needed to present your results to others. You can simply declare that you are very confident that a particular version is better than another. You can also give a likely range of percentage improvement (based on the amount of data you have collected and the width of the error bars).

Flexibility in defining the variable values. In whole-page split tests, you have complete flexibility in how different the proposed alternatives are. For example, in one alternative, you may simply choose to test a different headline. In another you may completely restructure everything about the page (layout, color scheme, sales copy, offer, and call-to-action). This ability to mix and match allows you to test a range of evolutionary and revolutionary alternatives in one test, without being constrained by the more granular definition of variables in a multivariate test.

Useful in low data rate tests. If your landing page only has a few conversions per day, you simply cannot use more advanced tuning methods. But with the proper selection of the test variable and alternative values, you can still achieve significant results in a split test. Improvements in the double or even triple digits are not uncommon.

A/B Split Testing Disadvantages

Split tests also have several drawbacks:

Limited number of versions. The number of versions in a typical split test is usually very small. If you did your homework properly, you probably came up with dozens of potential issues with your landing page, and also constructed many alternative variations to test. However, because of the limited scope of split testing, you will be reduced to testing your ideas one at a time. You will also be forced to guess which ideas to test first (based on your intuition about which ones might make the most difference). In other tuning methods, you may be able to test many of your key ideas at once and find all of the changes that improve your conversion rate in one test.

Does not consider context or variable interactions. By definition, split tests consider only one variable at a time, so you cannot detect variable interactions (how combinations of variables influence each other). A series of split tests covering several variables is not the same as a multivariate test with the same variables. Depending on the variable interactions, you may not be able to find the best-performing combination of variables on the page at all. Whether you do depends on the order in which you conduct your split tests, and the exact nature of the interactions.

No way to discover the importance of page elements. Often, you may choose very coarse variables for your split test. Because of the limited data rate, you are forced to make your best guess at page elements that might improve performance. These elements may actually involve many simultaneous changes to your landing page. In the extreme case of a whole-page redesign, you may have changed dozens of details on the page in question and defined them as a single alternative version.

However, the same flexibility that allows you to do this also limits your ability to interpret results and attribute credit for the conversion improvement to any particular change that you made. Was it the button color? Or was it the headline change? Or was it the different offer? You will never know. By squashing multiple changes into one page, you have confounded their effects and lost the ability to look at them separately.

In practice, this may not be such a huge issue, since many of the so-called learnings about the relative importance of variables are based on the spurious assumption that they are all independent of each other. Furthermore, the biggest conversion improvements may be due to the specific variable values you have chosen, and not the variable itself. For example, a particular headline that you chose to test was very powerful. But this does not allow you to generalize about headlines being more important than the other variables tested. In any case, you should avoid trying to interpret split test results if the variable values involve changing multiple elements on the page.

Inefficient data collection. Multivariate tests are often carefully constructed in order to get the most information from a smaller data sample. In effect, they allow you to more efficiently conduct multiple split tests simultaneously, and even to detect certain kinds of variable interactions. Conducting multiple split tests back-to-back is the most wasteful kind of data collection - none of the information from a previous test can be reused to draw conclusions about the other variables that you may want to test in the future.

ClickZ Live Chicago Join the Industry's Leading eCommerce & Direct Marketing Experts in Chicago
ClickZ Live Chicago (Nov 3-6) will deliver over 50 sessions across 4 days and 10 individual tracks, including Data-Driven Marketing, Social, Mobile, Display, Search and Email. Check out the full agenda and register by Friday, Oct 3 to take advantage of Early Bird Rates!

ABOUT THE AUTHOR

Tim Ash

Tim Ash is CEO of SiteTuners.com, a landing page optimization firm that offers conversion consulting, full-service guaranteed-improvement tests, and software tools to improve conversion rates. SiteTuners' AttentionWizard.com visual attention prediction tool can be used on a landing page screenshot or mock-up to quickly identify major conversion issues. He has worked with Google, Facebook, American Express, CBS, Sony Music, Universal Studios, Verizon Wireless, Texas Instruments, and Coach.

Tim is a highly-regarded presenter at SES, eMetrics, PPC Summit, Affiliate Summit, PubCon, Affiliate Conference, and LeadsCon. He is the chairperson of ConversionConference.com, the first conference focused on improving online conversions. A columnist for several publications including ClickZ, he's host of the weekly Landing Page Optimization show and podcast on WebmasterRadio.fm. His columns can be found in the Search Engine Watch archive.

He received his B.S. and M.S. during his Ph.D. studies at UC San Diego. Tim is the author of the bestselling book, "Landing Page Optimization."

Connect with Tim on Google+.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!

COMMENTS

UPCOMING EVENTS

Featured White Papers

IBM: Social Analytics - The Science Behind Social Media Marketing

IBM Social Analytics: The Science Behind Social Media Marketing
80% of internet users say they prefer to connect with brands via Facebook. 65% of social media users say they use it to learn more about brands, products and services. Learn about how to find more about customers' attitudes, preferences and buying habits from what they say on social media channels.

An Introduction to Marketing Attribution: Selecting the Right Model for Search, Display & Social Advertising

An Introduction to Marketing Attribution: Selecting the Right Model for Search, Display & Social Advertising
If you're considering implementing a marketing attribution model to measure and optimize your programs, this paper is a great introduction. It also includes real-life tips from marketers who have successfully implemented attribution in their organizations.

Jobs

    • Recent Grads: Customer Service Representative
      Recent Grads: Customer Service Representative (Agora Financial) - BaltimoreAgora Financial, one of the nation's largest independent publishers...
    • Managing Editor
      Managing Editor (Common Sense Publishing) - BaltimoreWE’RE HIRING: WE NEED AN AMAZING EDITOR TO POLISH WORLD-CLASS CONTENT   The Palm...
    • Senior Paid Search & Advertising Manager
      Senior Paid Search & Advertising Manager (Smarty Had A Party) - St. LouisCompany Description: A warm, loving, [slightly wacky] startup, we view...