Multivariate Test Construction

Choose wisely between unconstrained and constrained designs as your test design will limit your options for data collection and data analysis.

There are two primary types of multivariate test designs: unconstrained designs and constrained designs. Let’s discuss these further.

Unconstrained Designs

Unconstrained design test variables can be created and displayed independently of each other on your landing page. For instance, if all variables can be displayed in separate locations on your landing page, they are usually unconstrained. Most basic landing page tests involve unconstrained designs.

This does not mean that there are no variable interactions among variables in unconstrained designs. Some variables may interact with each other. For example, the call-to-action text on the button may interact with the button color. Imagine that the button text says “go.” On a green button this makes sense.

However, the same text displayed on the red button would be confusing since people commonly associate red with the word “stop.” So you should look for possible interactions among all the possible variable value combinations of V3 and V4, but your choices of the exact wording of the alternative calls-to-action text can probably be considered independently of your choices about alternative button colors.

Constrained Designs

Constrained designs involve conditional rules for constructing certain recipes. In other words, some of the allowed values for a particular variable are contingent on the setting of others, or can exist only under certain conditions. Under such circumstances, take special care to properly define and structure your variables. You may also have to make sure that you sample appropriately during your test and do not accidentally create improper recipes for presentation to your visitors.

Let’s consider a simple example. Assume that you have a two-page registration form. You want to test an alternative design for the second page, but you also want to consider a design where all of the form fields are moved onto the first page (i.e., there’s no need for a second page at all).

One possible test construction approach is as follows:

V1 – first page contents

a – original first page

b – extended first page (containing all original first and second page fields)

V2 – second page contents

a – original second page

b – alternative second page design

c – no second page (all content moved to V1b first page)

Under this unspecified constraint approach, you will need to keep track of the fact that V1b and V1c can only appear together, and enforce it through rules that are external to the variable definition itself.

Another common solution is to flatten the constrained variables and create a single variable containing values for each allowable design.

V1 – registration process

a – original first and second pages

b – original first page and alternative second page

c – extended one page (containing all original first and second page fields)

As a real-life example of a constrained design, ScanAlert, the creators of the Hacker Safe website trustmark, worked with my company to conduct multivariate testing of its trustmark to qualifying prospects and customers of Hacker Safe. The goal was to find the biggest conversion rate improvement possible by adding the trustmark. The company tested six different logos in multiple positions on the client’s landing page.

We tested against the original client landing page, which did not contain a trustmark. Obviously, if the trustmark is not present on the landing page, the specific logo chosen and its position becomes meaningless. To run these tests, we chose the specified constraint test construction approach. In other words, we reserved one variable specifically for the constrained condition, and a placeholder variable value for it in all other appropriate variables:

V1 – trustmark presence (branching factor = 2)

a – no trustmark

b – trustmark

V2 – trustmark logo (branching factor = 7)

a – no logo (placeholder for constraint in V1)

b – red-horizontal logo

c – red-vertical logo

d – black-horizontal logo

e – black-vertical logo

f – white-horizontal logo

g – white-vertical logo

V3 – trustmark location (branching factor = 5)

a – no location (placeholder for constraint in V1)

b – location no. 1

c – location no. 2

d – location no. 3

e – location no. 4

Based on the branching factors here, you might assume that this test has 70 distinct recipes (2 – 7 – 5). However, it only has 25 (one original plus 24 trustmark logo and position combinations). Besides the baseline recipe (specified by aaa), no other recipes are allowed to contain any a’s.

The Bottom Line

Unconstrained test designs are far easier to implement. You can set a different number of values for each variable so the branching factor is determined simply by the number of good options you want to test for each variable. Unconstrained designs go hand-in-hand with full factorial data collection.

Constrained test designs require you to predefine the number of variables and their branching factors. These designs are used with fractional factorial data collection and require you to choose from a standardized test construction.

Either way, it’s important to choose wisely, as your test design will limit your options for data collection (full factorial vs. fractional factorial) as well as data analysis (parametric vs. non-parametric).

Subscribe to get your daily business insights

Whitepapers

US Mobile Streaming Behavior
Whitepaper | Mobile

US Mobile Streaming Behavior

5y

US Mobile Streaming Behavior

Streaming has become a staple of US media-viewing habits. Streaming video, however, still comes with a variety of pesky frustrations that viewers are ...

View resource
Winning the Data Game: Digital Analytics Tactics for Media Groups
Whitepaper | Analyzing Customer Data

Winning the Data Game: Digital Analytics Tactics for Media Groups

5y

Winning the Data Game: Digital Analytics Tactics f...

Data is the lifeblood of so many companies today. You need more of it, all of which at higher quality, and all the meanwhile being compliant with data...

View resource
Learning to win the talent war: how digital marketing can develop its people
Whitepaper | Digital Marketing

Learning to win the talent war: how digital marketing can develop its peopl...

2y

Learning to win the talent war: how digital market...

This report documents the findings of a Fireside chat held by ClickZ in the first quarter of 2022. It provides expert insight on how companies can ret...

View resource
Engagement To Empowerment - Winning in Today's Experience Economy
Report | Digital Transformation

Engagement To Empowerment - Winning in Today's Experience Economy

3m

Engagement To Empowerment - Winning in Today's Exp...

Customers decide fast, influenced by only 2.5 touchpoints – globally! Make sure your brand shines in those critical moments. Read More...

View resource