Mix and Match Your Way to Failure

When professional basketball players were first allowed to play in the Olympic Games, the United States assembled a “dream team” from the ranks of top NBA superstars. The expectation was that this high-powered assembly of top talent would walk all over their competition. However, the United States lost in the gold-medal match to Yugoslavia.

How could this have happened? Clearly the individual U.S. players were superior to their Yugoslav counterparts. But the Yugoslav squad had trained together and was used to playing by the slightly different rules of Olympic basketball. By contrast, the U.S. team was assembled shortly before the games and had not practiced very much. They had not jelled as a team.

Similarly, some landing page elements that you are testing may be superstars individually. But you should be looking for the combination of variables that performs best when presented together.

What is a variable interaction? Simply put, it is when the setting for one variable in your test positively or negatively influences the setting of another variable. If they have no effect on each other, they are said to be independent. In a positive interaction, two (or more) variables create a synergistic effect (yielding results that are greater than the sum of the parts). In a negative interaction, two (or more) variables undercut each other and cancel out some of the individual effects.

Let’s look at a simple example. Let’s assume that you are an auto dealer who sells both Ferraris and Volvos. Your goal is to sell cars and you want to test two different headlines and two different accompanying pictures. So there are a total of four possible versions based on your two variables.

If you believe that there are no interactions, then you must also believe that there is a “best” headline regardless of the accompanying picture, and that there is a “best” picture regardless of the headline used.

Clearly this is not the case. Each variable depends on the context in which it is seen. Figure 1 has a strong positive interaction (connecting the speed and power in the picture with the word “Fast” in the headline).

Figure 1 – Example ad: Picture A, Headline A

Figure 2 has a strongly negative interaction (making you think about the consequences of fast driving -“speed kills”).

Figure 2 – Example ad: Picture B, Headline A

Figure 3 has a positive interaction (playing on the fear of accidents and highlighting Volvo’s longstanding safety record).

Figure 3 – Example ad: Picture B, Headline B

So it’s not the picture, and it’s not the headline that determines the performance of the ad. It is their particular combination.

In online marketing, we want interactions. We want the picture to reinforce the headline, and the sales copy, and the offer, and the call-to-action. Similarly, we want to detect any parts of the landing page that are working at cross-purposes and undercutting the performance of other page elements. Our goal should be to find the best performing group of landing page elements.

Most landing page testing methods (such as A/B split testing and many forms of fractional factorial multivariate testing such as the Taguchi Method and Design of Experiments or DoE) assume that there are absolutely no interactions among your variables (that they are completely independent of each other). Obviously for online marketing this is an absurd assumption. Very strong interaction effects (often involving more than two variables) definitely exist, and in my experience are pretty common. So while you may be able to get some positive results by ignoring interactions, you will not be getting the best results. You will have stitched together a Frankenstein’s monster made of workable pieces that do not make an effective whole. More importantly, you will have left money on the table during your landing page test.

So where can you look for interactions? In general, there is no way to guarantee that any subset of your testing elements does not interact. However, you should consider elements that are in physical proximity, or that are otherwise confounded with each other. For example, let’s assume that you are testing a form and have chosen to test the call-to-action button color and text. Although these may seem independent, that is not the case. They both combine to create the specific presentation of the call-to-action; you should test for possible interactions.

Similarly, if you are testing different headlines followed by different sales copy, you should expect interactions. The headline is supposed to draw the visitor into reading further. If there is a disconnect between the headline and the following text, you can expect negative interactions. If they reinforce each other, you should expect positive synergies. So far I have primarily focused on interactions between two test elements. In fact, there are often strong interactions among several variables on a landing page. Think through the possible combinations of variables in your test for best results and use testing technology that allows you to find any interactions that exist.