Multivariate Test Construction

  |  June 20, 2011   |  Comments

Choose wisely between unconstrained and constrained designs as your test design will limit your options for data collection and data analysis.

There are two primary types of multivariate test designs: unconstrained designs and constrained designs. Let's discuss these further.

Unconstrained Designs

Unconstrained design test variables can be created and displayed independently of each other on your landing page. For instance, if all variables can be displayed in separate locations on your landing page, they are usually unconstrained. Most basic landing page tests involve unconstrained designs.

This does not mean that there are no variable interactions among variables in unconstrained designs. Some variables may interact with each other. For example, the call-to-action text on the button may interact with the button color. Imagine that the button text says "go." On a green button this makes sense.

However, the same text displayed on the red button would be confusing since people commonly associate red with the word "stop." So you should look for possible interactions among all the possible variable value combinations of V3 and V4, but your choices of the exact wording of the alternative calls-to-action text can probably be considered independently of your choices about alternative button colors.

Constrained Designs

Constrained designs involve conditional rules for constructing certain recipes. In other words, some of the allowed values for a particular variable are contingent on the setting of others, or can exist only under certain conditions. Under such circumstances, take special care to properly define and structure your variables. You may also have to make sure that you sample appropriately during your test and do not accidentally create improper recipes for presentation to your visitors.

Let's consider a simple example. Assume that you have a two-page registration form. You want to test an alternative design for the second page, but you also want to consider a design where all of the form fields are moved onto the first page (i.e., there's no need for a second page at all).

One possible test construction approach is as follows:

V1 - first page contents

a - original first page

b - extended first page (containing all original first and second page fields)

V2 - second page contents

a - original second page

b - alternative second page design

c - no second page (all content moved to V1b first page)

Under this unspecified constraint approach, you will need to keep track of the fact that V1b and V1c can only appear together, and enforce it through rules that are external to the variable definition itself.

Another common solution is to flatten the constrained variables and create a single variable containing values for each allowable design.

V1 - registration process

a - original first and second pages

b - original first page and alternative second page

c - extended one page (containing all original first and second page fields)

As a real-life example of a constrained design, ScanAlert, the creators of the Hacker Safe website trustmark, worked with my company to conduct multivariate testing of its trustmark to qualifying prospects and customers of Hacker Safe. The goal was to find the biggest conversion rate improvement possible by adding the trustmark. The company tested six different logos in multiple positions on the client's landing page.

We tested against the original client landing page, which did not contain a trustmark. Obviously, if the trustmark is not present on the landing page, the specific logo chosen and its position becomes meaningless. To run these tests, we chose the specified constraint test construction approach. In other words, we reserved one variable specifically for the constrained condition, and a placeholder variable value for it in all other appropriate variables:

V1 - trustmark presence (branching factor = 2)

a - no trustmark

b - trustmark

V2 - trustmark logo (branching factor = 7)

a - no logo (placeholder for constraint in V1)

b - red-horizontal logo

c - red-vertical logo

d - black-horizontal logo

e - black-vertical logo

f - white-horizontal logo

g - white-vertical logo

V3 - trustmark location (branching factor = 5)

a - no location (placeholder for constraint in V1)

b - location no. 1

c - location no. 2

d - location no. 3

e - location no. 4

Based on the branching factors here, you might assume that this test has 70 distinct recipes (2 - 7 - 5). However, it only has 25 (one original plus 24 trustmark logo and position combinations). Besides the baseline recipe (specified by aaa), no other recipes are allowed to contain any a's.

The Bottom Line

Unconstrained test designs are far easier to implement. You can set a different number of values for each variable so the branching factor is determined simply by the number of good options you want to test for each variable. Unconstrained designs go hand-in-hand with full factorial data collection.

Constrained test designs require you to predefine the number of variables and their branching factors. These designs are used with fractional factorial data collection and require you to choose from a standardized test construction.

Either way, it's important to choose wisely, as your test design will limit your options for data collection (full factorial vs. fractional factorial) as well as data analysis (parametric vs. non-parametric).

ClickZ Live New York What's New for 2015?
You spoke, we listened! ClickZ Live New York (Mar 30-Apr 1) is back with a brand new streamlined agenda. Don't miss the latest digital marketing tips, tricks and tools that will make you re-think your strategy and revolutionize your marketing campaigns. Super Saver Rates are available now. Register today!

ABOUT THE AUTHOR

Tim Ash

Tim Ash is CEO of SiteTuners.com, a landing page optimization firm that offers conversion consulting, full-service guaranteed-improvement tests, and software tools to improve conversion rates. SiteTuners' AttentionWizard.com visual attention prediction tool can be used on a landing page screenshot or mock-up to quickly identify major conversion issues. He has worked with Google, Facebook, American Express, CBS, Sony Music, Universal Studios, Verizon Wireless, Texas Instruments, and Coach.

Tim is a highly-regarded presenter at SES, eMetrics, PPC Summit, Affiliate Summit, PubCon, Affiliate Conference, and LeadsCon. He is the chairperson of ConversionConference.com, the first conference focused on improving online conversions. A columnist for several publications including ClickZ, he's host of the weekly Landing Page Optimization show and podcast on WebmasterRadio.fm. His columns can be found in the Search Engine Watch archive.

He received his B.S. and M.S. during his Ph.D. studies at UC San Diego. Tim is the author of the bestselling book, "Landing Page Optimization."

Connect with Tim on Google+.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!

COMMENTS

UPCOMING EVENTS

UPCOMING TRAINING

Featured White Papers

Google My Business Listings Demystified

Google My Business Listings Demystified
To help brands control how they appear online, Google has developed a new offering: Google My Business Locations. This whitepaper helps marketers understand how to use this powerful new tool.

5 Ways to Personalize Beyond the Subject Line

5 Ways to Personalize Beyond the Subject Line
82 percent of shoppers say they would buy more items from a brand if the emails they sent were more personalized. This white paper offer five tactics that will personalize your email beyond the subject line and drive real business growth.

WEBINARS

    Information currently unavailable

Jobs

    • Lead Generation Specialist
      Lead Generation Specialist (The Oxford Club) - BaltimoreThe Oxford Club is seeking a talented writer/marketer to join our growing email lead-generation...
    • Health Marketing Editor
      Health Marketing Editor (Agora Inc.) - BaltimoreCome flex your intellectual muscle as part of Agora, Inc’s (http://agora-inc.com/) legal team...
    • Marketing Systems Analyst
      Marketing Systems Analyst (OmniVista Health) - BaltimoreOmniVista Health is looking to add a Marketing Systems Analyst to our expanding team. We...