Data Collection in Multivariate Testing

  |  June 6, 2011   |  Comments

Full factorial vs. fractional factorial designs; a look at the advantages and disadvantages to each.

There are two main design approaches used in multivariate testing: full factorial and fractional factorial. Before you get too far in planning your multivariate testing, you'll need to understand the advantages and disadvantages of each.

Full Factorial Test Designs

Full factorial experimental designs sample data across your whole search space. If this is done properly, the subsequent analysis allows you to consider not only the main effects, but all variable interactions as well (including higher-order ones).

Because of the sheer volume of interactions tested with a full-factorial design, this type of test generally requires a lot of data, meaning that the site being tested should have a pretty high rate of traffic. But because of the exponential growth of the number of model co-efficiencies, full factorial design quickly hits its limit if you are planning to conduct an analysis of all the possible interactions. For this reason, landing page tests that use full factorial designs often have a relatively small search space.

Full factorial data collection offers a number of advantages:

  • Considers interactions. If you use full factorial data collection coupled with a complete model, you can detect all important variable interactions.
  • Unrestricted test design. You can choose any number of test variables, and arbitrary branching factors for each one.
  • Better estimation of main effects. Even if you discard the interaction data and only build a model of the main effects, your estimate of the main effects will be more accurate than with fractional factorial designs. This is because data is collected across all recipes rather than a subset of your search space.

Full factorial data is powerful, yet the analysis can be complicated and overwhelming. While many tools exist that can collect and interpret data on main effects, it's advisable to have a strong background in statistics to determine the most meaningful variable interactions.

Fractional Factorial Test Designs

Fractional factorial multivariate testing allows you to simultaneously test several key elements of your website or landing pages. In effect, it's like running several A/B split tests at once, but with total traffic requirements that are significantly less than would be necessary for an equivalent number of separate A/B split tests. Fractional factorial test designs have the advantage of compressing the amount of data (website traffic) required, while still giving you the benefits of multiple simultaneous A/B split tests.

Fractional factorial designs fall under the design of experiments (DOE) umbrella. DOE is a systematic approach to getting the maximum amount of useful information about the process that you are studying, while minimizing the amount of effort and data collection required. Commonly known by the name Taguchi Method, fractional factorial testing requires you to determine upfront what elements and variations you will test. "Fractional" means just that: you will be collecting data on only some, or a fraction, of the possible recipes, or variables.

One feature of fractional factorial testing is that it allows you to explicitly define the interactions among the variables that you want to study and examine. But this can be a disadvantage as well, because you'll need to make guesses about which interactions will be important and then build those assumptions into your model upfront.

There are other key disadvantages to fractional factorial test designs:

  • The Frankenstein effect. Because you are collecting data about several variables simultaneously yet independently, you run the risk of creating a final page recipe comprised of all your winning variables, yet find that combined these variables don't perform well. It is difficult to determine the best results without considering how the variables interact with one another.
  • Restrictive test design. Your test design must follow a certain pattern, in terms of the number of variables and their branching factors. As a result, you are forced to either stick with well-known "standard" designs from statistical textbooks, or construct your own (with the help of statisticians).
  • Throttling is difficult. If you throttle your data collection rates and don't devote equal bandwidth to each recipe in your test, your analysis will be invalid.

However, by limiting data collection to fewer elements, a fractional factorial test can be completed more quickly and with lower traffic rates than a full factorial test. For those desiring a more iterative approach to their testing, fractional factorial testing may provide some quick answers on key page elements.


As with most things in life, there is an inherent trade-off among various multivariate test constructions. Full factorial parametric designs do not scale very well, but get more complete information about the exact relationship among all main and interaction effects tested. Fractional factorial designs can scale to larger search spaces, but make assumptions about the underlying process that may not be valid and may actually lead you astray.


Tim Ash

Tim Ash is CEO of, a landing page optimization firm that offers conversion consulting, full-service guaranteed-improvement tests, and software tools to improve conversion rates. SiteTuners' visual attention prediction tool can be used on a landing page screenshot or mock-up to quickly identify major conversion issues. He has worked with Google, Facebook, American Express, CBS, Sony Music, Universal Studios, Verizon Wireless, Texas Instruments, and Coach.

Tim is a highly-regarded presenter at SES, eMetrics, PPC Summit, Affiliate Summit, PubCon, Affiliate Conference, and LeadsCon. He is the chairperson of, the first conference focused on improving online conversions. A columnist for several publications including ClickZ, he's host of the weekly Landing Page Optimization show and podcast on His columns can be found in the Search Engine Watch archive.

He received his B.S. and M.S. during his Ph.D. studies at UC San Diego. Tim is the author of the bestselling book, "Landing Page Optimization."

Connect with Tim on Google+.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!



Featured White Papers

2015 Holiday Email Guide

2015 Holiday Email Guide
The holidays are just around the corner. Download this whitepaper to find out how to create successful holiday email campaigns that drive engagement and revenue.

Three Ways to Make Your Big Data More Valuable

Three Ways to Make Your Big Data More Valuable
Big data holds a lot of promise for marketers, but are marketers ready to make the most of it to drive better business decisions and improve ROI? This study looks at the hidden challenges modern marketers face when trying to put big data to use.