Data Collection in Multivariate Testing
Full factorial vs. fractional factorial designs; a look at the advantages and disadvantages to each.
Full factorial vs. fractional factorial designs; a look at the advantages and disadvantages to each.
There are two main design approaches used in multivariate testing: full factorial and fractional factorial. Before you get too far in planning your multivariate testing, you’ll need to understand the advantages and disadvantages of each.
Full Factorial Test Designs
Full factorial experimental designs sample data across your whole search space. If this is done properly, the subsequent analysis allows you to consider not only the main effects, but all variable interactions as well (including higher-order ones).
Because of the sheer volume of interactions tested with a full-factorial design, this type of test generally requires a lot of data, meaning that the site being tested should have a pretty high rate of traffic. But because of the exponential growth of the number of model co-efficiencies, full factorial design quickly hits its limit if you are planning to conduct an analysis of all the possible interactions. For this reason, landing page tests that use full factorial designs often have a relatively small search space.
Full factorial data collection offers a number of advantages:
Full factorial data is powerful, yet the analysis can be complicated and overwhelming. While many tools exist that can collect and interpret data on main effects, it’s advisable to have a strong background in statistics to determine the most meaningful variable interactions.
Fractional Factorial Test Designs
Fractional factorial multivariate testing allows you to simultaneously test several key elements of your website or landing pages. In effect, it’s like running several A/B split tests at once, but with total traffic requirements that are significantly less than would be necessary for an equivalent number of separate A/B split tests. Fractional factorial test designs have the advantage of compressing the amount of data (website traffic) required, while still giving you the benefits of multiple simultaneous A/B split tests.
Fractional factorial designs fall under the design of experiments (DOE) umbrella. DOE is a systematic approach to getting the maximum amount of useful information about the process that you are studying, while minimizing the amount of effort and data collection required. Commonly known by the name Taguchi Method, fractional factorial testing requires you to determine upfront what elements and variations you will test. “Fractional” means just that: you will be collecting data on only some, or a fraction, of the possible recipes, or variables.
One feature of fractional factorial testing is that it allows you to explicitly define the interactions among the variables that you want to study and examine. But this can be a disadvantage as well, because you’ll need to make guesses about which interactions will be important and then build those assumptions into your model upfront.
There are other key disadvantages to fractional factorial test designs:
However, by limiting data collection to fewer elements, a fractional factorial test can be completed more quickly and with lower traffic rates than a full factorial test. For those desiring a more iterative approach to their testing, fractional factorial testing may provide some quick answers on key page elements.
Summary
As with most things in life, there is an inherent trade-off among various multivariate test constructions. Full factorial parametric designs do not scale very well, but get more complete information about the exact relationship among all main and interaction effects tested. Fractional factorial designs can scale to larger search spaces, but make assumptions about the underlying process that may not be valid and may actually lead you astray.