Testing multiple variables within a single email campaign can get a bit tricky. One way to do it is with the all-important and extremely valuable “panel test.” It requires pretty aggressive numbers for deployment, yet can be an excellent way to get to know the strongest (and weakest) responders and/or variables within a mailing. And as far as applications for reading and applying results go, this method of testing can get you the most bang for the buck.
Essentially, a panel test comprises individual groups of email lists, each of which consists of like-minded people. Each group represents a different variable that you’d like to test and is flagged appropriately. When it comes time for deployment, all groups are sent at the same time. (This is critical.) And when the campaign is complete, final results are determined based on reading response at both the list and the group level.
Let’s take a closer look. Say you have an online party supply store and have a house list of 200,000 previous buyers and/or newsletter subscribers. You consistently send them, en masse, a promotional newsletter every two weeks in an attempt to get them back to your site to buy. Response is okay, but it could be better, in your opinion. Your goal? To find the cream of the crop within your audience and which demographics and segments will produce the best results when matched across your artillery of offers, products, subject lines and creative, in order to enhance your results.
First order of business: Split your list. To avoid too much confusion and for purposes of this article, let’s just say that you have the ability and want to split your list according to 1) gender, 2) whether the previous buyers made purchases within the last 30 days, 60 days or six months. Simple enough.
As far as the offer goes, you want to pit your tried-and-true newsletter (which contains content AND promotes products) against a straight product offer that has two different versions. One version promotes one product only while the other promotes multiple products. Added to that is a test of two different subject lines, along with two different pieces of copy. Whew!
So to clarify, and because I’m a visually oriented person, here’s a grid to demonstrate the various panels and their contents. (I’ll address the blank boxes in a moment.)
Panel 1 – Newsletter (Control)
|
Panel 2 – One Product
|
Panel 3 – Multi- Products
|
Panel 4 – Subject Line 1
|
Panel 5 – Subject Line 2
|
Panel 6 – Copy A
|
Panel 7 – Copy B
|
MALE
|
MALE
|
MALE
|
MALE
|
|
MALE
|
|
FEMALE
|
FEMALE
|
FEMALE
|
FEMALE
|
FEMALE
|
FEMALE
|
FEMALE
|
LAST 30 DAYS
|
LAST 30 DAYS
|
LAST 30 DAYS
|
LAST 30 DAYS
|
LAST 30 DAYS
|
LAST 30 DAYS
|
LAST 30 DAYS
|
LAST 60 DAYS
|
LAST 60 DAYS
|
LAST 60 DAYS
|
LAST 60 DAYS
|
LAST 60 DAYS
|
LAST 60 DAYS
|
LAST 60 DAYS
|
LAST 6 MO
|
LAST 6 MO
|
LAST 6 MO
|
LAST 6 MO
|
|
LAST 6 MO
|
|
The quantity of each subsection or segment (i.e., male, female, etc.) should be equal within each panel and across all panels. And you don’t necessarily have to email your entire list. Remember, this is a test. As long as you have taken an equal number of random names across each segment representing similarly minded people, presumably that is the most important thing. To be statistically significant (and to ensure the validity of the campaign), most email marketers will split the list into groups of at least 3,000 to 5,000 people per segment.
One rewarding thing to note about panel testing is that you can lower your risk a bit by reducing the number of people in your mailing that you might assume are lower-than-average responders, such as those who haven’t bought in a while. In this case, the assumption was made that both “males” and older (last six months) buyers were higher risk. So in two of the panels Panel 5 and Panel 7 those segments were not emailed the corresponding version.
However, results could still be determined based on the relative strength of the counterparts. In other words, Panel 4 (Subject Line 1), which is Panel 5’s counterpart (meaning everything else is the same except for the subject line), will get results for all segments. Panel 5 can also learn some things about its own components based on Panel 4’s results. So say the click-through rates for each segment of those two panels shake out as follows:
Panel 4 – Subject Line 1
|
CTR
|
Panel 5 – Subject Line 2
|
CTR
|
MALE
|
3.6%
|
Didn’t email
|
|
FEMALE
|
7.9%
|
FEMALE
|
8.3%
|
LAST 30 DAYS
|
11.2%
|
LAST 30 DAYS
|
11.1%
|
LAST 60 DAYS
|
3.6%
|
LAST 60 DAYS
|
4.0%
|
LAST 6 MO
|
1.2%
|
Didn’t email
|
|
As you can see, the results clearly indicate that, on an overall basis, Panel 5 Subject Line 2 is stronger than Panel 4. The slight difference in the “30 days” list’s results is not significant enough to cause concern. And the response rates of the segments that were a part of both panels indicate that the weaker segments will be relatively as strong as their counterparts. Meaning the click-through for “males” and older (last six months) buyers would have been, in all likelihood, stronger in Panel 5 had they been mailed.
When all is said and done, your final results will be based on both the overall results of each panel’s weighted average, along with the results of the individual lists within each panel. Sometimes a panel will be an overall winner, with one or two lists being clear-cut losers. When that happens, it’s simply a matter of doing a little tweaking here, a little fine-tuning there. Before you know it, you’ll have optimized your email campaigns. And isn’t optimization what testing is all about?