There are two main design approaches used in multivariate testing: full factorial and fractional factorial. Before you get too far in planning your multivariate testing, you'll need to understand the advantages and disadvantages of each.
Full Factorial Test Designs
Full factorial experimental designs sample data across your whole search space. If this is done properly, the subsequent analysis allows you to consider not only the main effects, but all variable interactions as well (including higher-order ones).
Because of the sheer volume of interactions tested with a full-factorial design, this type of test generally requires a lot of data, meaning that the site being tested should have a pretty high rate of traffic. But because of the exponential growth of the number of model co-efficiencies, full factorial design quickly hits its limit if you are planning to conduct an analysis of all the possible interactions. For this reason, landing page tests that use full factorial designs often have a relatively small search space.
Full factorial data collection offers a number of advantages:
Full factorial data is powerful, yet the analysis can be complicated and overwhelming. While many tools exist that can collect and interpret data on main effects, it's advisable to have a strong background in statistics to determine the most meaningful variable interactions.
Fractional Factorial Test Designs
Fractional factorial multivariate testing allows you to simultaneously test several key elements of your website or landing pages. In effect, it's like running several A/B split tests at once, but with total traffic requirements that are significantly less than would be necessary for an equivalent number of separate A/B split tests. Fractional factorial test designs have the advantage of compressing the amount of data (website traffic) required, while still giving you the benefits of multiple simultaneous A/B split tests.
Fractional factorial designs fall under the design of experiments (DOE) umbrella. DOE is a systematic approach to getting the maximum amount of useful information about the process that you are studying, while minimizing the amount of effort and data collection required. Commonly known by the name Taguchi Method, fractional factorial testing requires you to determine upfront what elements and variations you will test. "Fractional" means just that: you will be collecting data on only some, or a fraction, of the possible recipes, or variables.
One feature of fractional factorial testing is that it allows you to explicitly define the interactions among the variables that you want to study and examine. But this can be a disadvantage as well, because you'll need to make guesses about which interactions will be important and then build those assumptions into your model upfront.
There are other key disadvantages to fractional factorial test designs:
However, by limiting data collection to fewer elements, a fractional factorial test can be completed more quickly and with lower traffic rates than a full factorial test. For those desiring a more iterative approach to their testing, fractional factorial testing may provide some quick answers on key page elements.
As with most things in life, there is an inherent trade-off among various multivariate test constructions. Full factorial parametric designs do not scale very well, but get more complete information about the exact relationship among all main and interaction effects tested. Fractional factorial designs can scale to larger search spaces, but make assumptions about the underlying process that may not be valid and may actually lead you astray.
Tim Ash is CEO of SiteTuners.com, a landing page optimization firm that offers conversion consulting, full-service guaranteed-improvement tests, and software tools to improve conversion rates. SiteTuners' AttentionWizard.com visual attention prediction tool can be used on a landing page screenshot or mock-up to quickly identify major conversion issues. He has worked with Google, Facebook, American Express, CBS, Sony Music, Universal Studios, Verizon Wireless, Texas Instruments, and Coach.
Tim is a highly-regarded presenter at SES, eMetrics, PPC Summit, Affiliate Summit, PubCon, Affiliate Conference, and LeadsCon. He is the chairperson of ConversionConference.com, the first conference focused on improving online conversions. A columnist for several publications including ClickZ, he's host of the weekly Landing Page Optimization show and podcast on WebmasterRadio.fm. His columns can be found in the Search Engine Watch archive.
He received his B.S. and M.S. during his Ph.D. studies at UC San Diego. Tim is the author of the bestselling book, "Landing Page Optimization."
May 22, 2013
1:00pm ET / 10:00am PT
June 5, 2013
1:00pm ET / 10:00am PT