Multivariate Test Construction

  |  June 20, 2011   |  Comments

Choose wisely between unconstrained and constrained designs as your test design will limit your options for data collection and data analysis.

There are two primary types of multivariate test designs: unconstrained designs and constrained designs. Let's discuss these further.

Unconstrained Designs

Unconstrained design test variables can be created and displayed independently of each other on your landing page. For instance, if all variables can be displayed in separate locations on your landing page, they are usually unconstrained. Most basic landing page tests involve unconstrained designs.

This does not mean that there are no variable interactions among variables in unconstrained designs. Some variables may interact with each other. For example, the call-to-action text on the button may interact with the button color. Imagine that the button text says "go." On a green button this makes sense.

However, the same text displayed on the red button would be confusing since people commonly associate red with the word "stop." So you should look for possible interactions among all the possible variable value combinations of V3 and V4, but your choices of the exact wording of the alternative calls-to-action text can probably be considered independently of your choices about alternative button colors.

Constrained Designs

Constrained designs involve conditional rules for constructing certain recipes. In other words, some of the allowed values for a particular variable are contingent on the setting of others, or can exist only under certain conditions. Under such circumstances, take special care to properly define and structure your variables. You may also have to make sure that you sample appropriately during your test and do not accidentally create improper recipes for presentation to your visitors.

Let's consider a simple example. Assume that you have a two-page registration form. You want to test an alternative design for the second page, but you also want to consider a design where all of the form fields are moved onto the first page (i.e., there's no need for a second page at all).

One possible test construction approach is as follows:

V1 - first page contents

a - original first page

b - extended first page (containing all original first and second page fields)

V2 - second page contents

a - original second page

b - alternative second page design

c - no second page (all content moved to V1b first page)

Under this unspecified constraint approach, you will need to keep track of the fact that V1b and V1c can only appear together, and enforce it through rules that are external to the variable definition itself.

Another common solution is to flatten the constrained variables and create a single variable containing values for each allowable design.

V1 - registration process

a - original first and second pages

b - original first page and alternative second page

c - extended one page (containing all original first and second page fields)

As a real-life example of a constrained design, ScanAlert, the creators of the Hacker Safe website trustmark, worked with my company to conduct multivariate testing of its trustmark to qualifying prospects and customers of Hacker Safe. The goal was to find the biggest conversion rate improvement possible by adding the trustmark. The company tested six different logos in multiple positions on the client's landing page.

We tested against the original client landing page, which did not contain a trustmark. Obviously, if the trustmark is not present on the landing page, the specific logo chosen and its position becomes meaningless. To run these tests, we chose the specified constraint test construction approach. In other words, we reserved one variable specifically for the constrained condition, and a placeholder variable value for it in all other appropriate variables:

V1 - trustmark presence (branching factor = 2)

a - no trustmark

b - trustmark

V2 - trustmark logo (branching factor = 7)

a - no logo (placeholder for constraint in V1)

b - red-horizontal logo

c - red-vertical logo

d - black-horizontal logo

e - black-vertical logo

f - white-horizontal logo

g - white-vertical logo

V3 - trustmark location (branching factor = 5)

a - no location (placeholder for constraint in V1)

b - location no. 1

c - location no. 2

d - location no. 3

e - location no. 4

Based on the branching factors here, you might assume that this test has 70 distinct recipes (2 - 7 - 5). However, it only has 25 (one original plus 24 trustmark logo and position combinations). Besides the baseline recipe (specified by aaa), no other recipes are allowed to contain any a's.

The Bottom Line

Unconstrained test designs are far easier to implement. You can set a different number of values for each variable so the branching factor is determined simply by the number of good options you want to test for each variable. Unconstrained designs go hand-in-hand with full factorial data collection.

Constrained test designs require you to predefine the number of variables and their branching factors. These designs are used with fractional factorial data collection and require you to choose from a standardized test construction.

Either way, it's important to choose wisely, as your test design will limit your options for data collection (full factorial vs. fractional factorial) as well as data analysis (parametric vs. non-parametric).

ClickZ Live Toronto Twitter Canada MD Kirstine Stewart to Keynote Toronto
ClickZ Live Toronto (May 14-16) is a new event addressing the rapidly changing landscape that digital marketers face. The agenda focuses on customer engagement and attaining maximum ROI through online marketing efforts across paid, owned & earned media. Register now and save!*
*Early Bird Rates expire April 17.

ABOUT THE AUTHOR

Tim Ash

Tim Ash is CEO of SiteTuners.com, a landing page optimization firm that offers conversion consulting, full-service guaranteed-improvement tests, and software tools to improve conversion rates. SiteTuners' AttentionWizard.com visual attention prediction tool can be used on a landing page screenshot or mock-up to quickly identify major conversion issues. He has worked with Google, Facebook, American Express, CBS, Sony Music, Universal Studios, Verizon Wireless, Texas Instruments, and Coach.

Tim is a highly-regarded presenter at SES, eMetrics, PPC Summit, Affiliate Summit, PubCon, Affiliate Conference, and LeadsCon. He is the chairperson of ConversionConference.com, the first conference focused on improving online conversions. A columnist for several publications including ClickZ, he's host of the weekly Landing Page Optimization show and podcast on WebmasterRadio.fm. His columns can be found in the Search Engine Watch archive.

He received his B.S. and M.S. during his Ph.D. studies at UC San Diego. Tim is the author of the bestselling book, "Landing Page Optimization."

Connect with Tim on Google+.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!

COMMENTS

UPCOMING EVENTS

Featured White Papers

ion Interactive Marketing Apps for Landing Pages White Paper

Marketing Apps for Landing Pages White Paper
Marketing apps can elevate a formulaic landing page into a highly interactive user experience. Learn how to turn your static content into exciting marketing apps.

eMarketer: Redefining Mobile-Only Users: Millions Selectively Avoid the Desktop

Redefining 'Mobile-Only' Users: Millions Selectively Avoid the Desktop
A new breed of selective mobile-only consumers has emerged. What are the demos of these users and how and where can marketers reach them?

Jobs

    • Contact Center Professional
      Contact Center Professional (TCC: The Contact Center) - Hunt ValleyLooking to join a workforce that prides themselves on being routine and keeping...
    • Recruitment and Team Building Ambassador
      Recruitment and Team Building Ambassador (Agora Inc.) - BaltimoreAgora, www.agora-inc.com, continues to expand! In order to meet the needs of our...
    • Design and Publishing Specialist
      Design and Publishing Specialist (Bonner and Partners) - BaltimoreIf you’re a hungry self-starter, creative, organized and have an extreme...