Data Collection in Multivariate Testing

  |  June 6, 2011   |  Comments

Full factorial vs. fractional factorial designs; a look at the advantages and disadvantages to each.

There are two main design approaches used in multivariate testing: full factorial and fractional factorial. Before you get too far in planning your multivariate testing, you'll need to understand the advantages and disadvantages of each.

Full Factorial Test Designs

Full factorial experimental designs sample data across your whole search space. If this is done properly, the subsequent analysis allows you to consider not only the main effects, but all variable interactions as well (including higher-order ones).

Because of the sheer volume of interactions tested with a full-factorial design, this type of test generally requires a lot of data, meaning that the site being tested should have a pretty high rate of traffic. But because of the exponential growth of the number of model co-efficiencies, full factorial design quickly hits its limit if you are planning to conduct an analysis of all the possible interactions. For this reason, landing page tests that use full factorial designs often have a relatively small search space.

Full factorial data collection offers a number of advantages:

  • Considers interactions. If you use full factorial data collection coupled with a complete model, you can detect all important variable interactions.
  • Unrestricted test design. You can choose any number of test variables, and arbitrary branching factors for each one.
  • Better estimation of main effects. Even if you discard the interaction data and only build a model of the main effects, your estimate of the main effects will be more accurate than with fractional factorial designs. This is because data is collected across all recipes rather than a subset of your search space.

Full factorial data is powerful, yet the analysis can be complicated and overwhelming. While many tools exist that can collect and interpret data on main effects, it's advisable to have a strong background in statistics to determine the most meaningful variable interactions.

Fractional Factorial Test Designs

Fractional factorial multivariate testing allows you to simultaneously test several key elements of your website or landing pages. In effect, it's like running several A/B split tests at once, but with total traffic requirements that are significantly less than would be necessary for an equivalent number of separate A/B split tests. Fractional factorial test designs have the advantage of compressing the amount of data (website traffic) required, while still giving you the benefits of multiple simultaneous A/B split tests.

Fractional factorial designs fall under the design of experiments (DOE) umbrella. DOE is a systematic approach to getting the maximum amount of useful information about the process that you are studying, while minimizing the amount of effort and data collection required. Commonly known by the name Taguchi Method, fractional factorial testing requires you to determine upfront what elements and variations you will test. "Fractional" means just that: you will be collecting data on only some, or a fraction, of the possible recipes, or variables.

One feature of fractional factorial testing is that it allows you to explicitly define the interactions among the variables that you want to study and examine. But this can be a disadvantage as well, because you'll need to make guesses about which interactions will be important and then build those assumptions into your model upfront.

There are other key disadvantages to fractional factorial test designs:

  • The Frankenstein effect. Because you are collecting data about several variables simultaneously yet independently, you run the risk of creating a final page recipe comprised of all your winning variables, yet find that combined these variables don't perform well. It is difficult to determine the best results without considering how the variables interact with one another.
  • Restrictive test design. Your test design must follow a certain pattern, in terms of the number of variables and their branching factors. As a result, you are forced to either stick with well-known "standard" designs from statistical textbooks, or construct your own (with the help of statisticians).
  • Throttling is difficult. If you throttle your data collection rates and don't devote equal bandwidth to each recipe in your test, your analysis will be invalid.

However, by limiting data collection to fewer elements, a fractional factorial test can be completed more quickly and with lower traffic rates than a full factorial test. For those desiring a more iterative approach to their testing, fractional factorial testing may provide some quick answers on key page elements.

Summary

As with most things in life, there is an inherent trade-off among various multivariate test constructions. Full factorial parametric designs do not scale very well, but get more complete information about the exact relationship among all main and interaction effects tested. Fractional factorial designs can scale to larger search spaces, but make assumptions about the underlying process that may not be valid and may actually lead you astray.

ClickZ Live San Francisco This Year's Premier Digital Marketing Event is #CZLSF
ClickZ Live San Francisco (Aug 11-14) brings together the industry's leading practitioners and marketing strategists to deliver 4 days of educational sessions and training workshops. From Data-Driven Marketing to Social, Mobile, Display, Search and Email, this year's comprehensive agenda will help you maximize your marketing efforts and ROI. Register today!

ABOUT THE AUTHOR

Tim Ash

Tim Ash is CEO of SiteTuners.com, a landing page optimization firm that offers conversion consulting, full-service guaranteed-improvement tests, and software tools to improve conversion rates. SiteTuners' AttentionWizard.com visual attention prediction tool can be used on a landing page screenshot or mock-up to quickly identify major conversion issues. He has worked with Google, Facebook, American Express, CBS, Sony Music, Universal Studios, Verizon Wireless, Texas Instruments, and Coach.

Tim is a highly-regarded presenter at SES, eMetrics, PPC Summit, Affiliate Summit, PubCon, Affiliate Conference, and LeadsCon. He is the chairperson of ConversionConference.com, the first conference focused on improving online conversions. A columnist for several publications including ClickZ, he's host of the weekly Landing Page Optimization show and podcast on WebmasterRadio.fm. His columns can be found in the Search Engine Watch archive.

He received his B.S. and M.S. during his Ph.D. studies at UC San Diego. Tim is the author of the bestselling book, "Landing Page Optimization."

Connect with Tim on Google+.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!

COMMENTS

UPCOMING EVENTS

Featured White Papers

BigDoor: The Marketers Guide to Customer Loyalty

The Marketer's Guide to Customer Loyalty
Customer loyalty is imperative to success, but fostering and maintaining loyalty takes a lot of work. This guide is here to help marketers build, execute, and maintain a successful loyalty initiative.

Marin Software: The Multiplier Effect of Integrating Search & Social Advertising

The Multiplier Effect of Integrating Search & Social Advertising
Latest research reveals 68% higher revenue per conversion for marketers who integrate their search & social advertising. In addition to the research results, this whitepaper also outlines 5 strategies and 15 tactics you can use to better integrate your search and social campaigns.

WEBINARS

    Information currently unavailable

Jobs

    • Internet Marketing Campaign Manager
      Internet Marketing Campaign Manager (Straight North, LLC) - Fort MillWe are looking for a talented Internet Marketing Campaign Manager to join the...
    • Online Marketing Coordinator
      Online Marketing Coordinator (NewMarket Health) - BaltimoreWant to learn marketing from the best minds in the business? NewMarket Health, a subsidiary...
    • Call Center Manager
      Call Center Manager (Common Sense Publishing) - Delray BeachWanted: Dynamic Call Center Manager with a Proven Track Record of Improving Response...