Five Mini Case Studies

I’m back from ClickZ E-Mail Strategies in New York — another successful event — and this week’s column comes to you courtesy of the case studies panel I hosted. One panelist, Clink Kaiser of Blue Ink Solutions, presented a series of mini case studies, all of which revolved around testing. If you weren’t able to attend the conference, here’s a peek at what you missed.

Demo-Versus-Download Showdown

Savvy marketers know giving away something for free is a great way to entice potential customers, but they often struggle with the question of exactly what to give away. Company A, which offers high-end document management software, wanted to run a campaign that would drive sales leads. So it came up with two offers. Offer 1 would allow recipients to download free software. Offer 2 would allow recipients to run a free demo. The subject lines were:

• Download $1000 Worth of Electronic Forms Design Software
• Demo The Only Complete eForms Toolkit

Both offers required registration and were sent to a large list of similar recipients. OK, take your best guess at which offer performed better. Ready?

As you may have guessed, the CTR for Offer 1 outperformed Offer 2 by a factor of two. In other words, twice as many recipients of Offer 1 registered at the site compared to the number of those who registered after receiving Offer 2.

But when the sales reps followed up with phone calls, it turned out the quality of those leads was, in Kaiser’s words, “significantly lower.” Kaiser noted the demo leads were at least twice as qualified as those who signed up to get free software, which could wash out the CTR advantage.

The main lesson here? When testing, test all the way through the cycle for the best results. CTR can be a good indication of message receptivity, but it can also be misleading.

The Copy Length and Subject Lines Test

N2H2 produces business-to-business (B2B) Web filtering software for corporations, schools, libraries, and other large organizations. To get the word out about its products, it sent one of four messages to approximately 8,000 people, or about 2,000 people per group. The messages had either short or long copy, and the two subject lines were:

• The Dangers of Inappropriate Web Usage
• N2H2’s Traffic Report Highlights Web Use Issues

Your guess?

Statistically speaking, the differences were insignificant (granted, this example doesn’t show the conversion rate, but it’s a great starting point for testing):

• Short copy and subject line 1: 50 percent open rate and 8 percent CTR
• Long copy and subject line 1: 53 percent open rate and 10 percent CTR
• Short copy and subject line 2: 49 percent open rate and 9 percent CTR
• Long copy and subject line 2: 49 percent open rate and 10 percent CTR

So why do I even bother telling you about this case study? The lesson here is even no data can be data. Occasionally, you’ll find no real difference in the items you test, but that certainly doesn’t mean testing is a waste of time. N2H2 has learned something more about its target audience, and now the company can focus on other items, such as…

Personalization in the Subject Line

N2H2 sent out tens of thousands of messages to its audience, all with the same subject line: “Is Your Network In Need Of Spring Cleaning?” But there was one variation. Half of the recipients had their names in the subject line, like “Heidi, Is Your Network In Need Of Spring Cleaning?”

Conventional wisdom says personalization helps lift. But in this case, when the company tested open rates, it found the subject lines without personalization had a 5 percent higher open rate. One theory is though consumers respond better to their names in subject lines, business users have a more critical eye.

There are several lessons here, but the one I focus on is personalization has a time and place and doesn’t improve results in every case.

The “How To” Hook

Anark designs high-end multimedia authoring software. For a sales-generation program, the company offered a white paper. To tens of thousands of recipients, it sent nearly identical messages with two different subject lines:

• How To Stay Ahead Of Your Multimedia Competition
• Multimedia Technologies That Will Get You Ahead

Nearly identical messages, to be sure.

Kaiser noted, however, the first message had a better open rate of 5 percentage points(40.5 percent vs. 35.4 percent). Though this may not seem overwhelmingly significant, he noted at 35,000 messages, with a CTR of about 10 percent, this could lead to an additional 1,750 or so opens and 150 to 170 additional click-throughs. He added his company regularly sees putting “how to” in the subject line measurably lifts response, so that is this mini case study’s lesson.

ADV: Should You Use It?

Lastly, Kaiser talked about a case study regarding “ADV” in the subject line. Seems a company’s conservative legal counsel advised that even though the company was emailing to a permission-based rented list, the company should place ADV in the subject line. (Not surprisingly, the list rental company and Blue Ink Solutions were not in favor.) The offer generated, in Kaiser’s experience, one of the worst responses he’d seen (a .86 percent open rate).

A few months later, the company sent the same offer to the same list (albeit to different individuals), with the same subject line — minus the ADV. You guessed it; the second time around, the offer performed better by multiple factors.

Although the lessons in today’s column’s mini case studies vary, they all can be summed up in one lesson: Test, test, and test. You’ll learn valuable information about your target audience.

Don’t miss ClickZ’s Weblog Business Strategies in Boston, June 9-10.