What to Test… and What Not to Test

I’m a huge fan of testing when you send email. It’s the best way to continually improve metrics and optimize your efforts. That said, not everything warrants an A/B split test. Here are a few scenarios I’ve come across with clients recently, along with my test/don’t test advice.

Click to View a Web-Based Version of This E-Mail

This was originally used in text email to send people to a Web-based HTML version, in the era before most email clients could read HTML. It had just about died out when some email clients began to block images and stop links from appearing live in HTML email. That gave it a new purpose in life.

If you’re concerned images and links aren’t working for your recipients, by all means include this in your HTML email. Yet this change doesn’t warrant an A/B split test.

In my experience (and this may change), not many people click these links. When I say not many, I mean way less than 100; in some cases, less than a dozen. Whether the link is used should be your gauge of whether it’s worth including. It’s unlikely adding this link will lift CTR (define); most likely you’ll see no change.

Add Our Address to Your Address Book

There was a big push on this a few years back by one of the big email service providers (ESPs). I haven’t heard much about it since. It’s true if you can get recipients to add your sender address to their address books, you may be whitelisted (which may have benefits such as guaranteed delivery to the inbox and non-blocked images and live links). But it’s easier said than done.

Since there’s no click involved, it’s hard to gauge how many recipients take this action. It would be impossible to test whether it’s effective, and it’s doubtful adding this will depress CTRs. Best approach? If you want to add it, do so. Don’t worry about testing it.

A word of advice: if you’re going to do this, the traditional placement is at the very top of the message, usually in text above the masthead. You may be tempted to bury it in the footer, but this only makes it less likely readers will see it, as the primary reason people view the footer is to unsubscribe. That doesn’t make them great candidates to add you to their address books.

Changes to the Sender Line

With one exception, you should always test any changes you plan to make here. The sender line is key to getting opened; if your messages aren’t opened, recipients can’t click and your metrics will take a hit.

The single exception is when the change is mandatory, not voluntary. Changing your sender line to read “ABC’s Widgets” instead of just “ABC Company” would be a voluntary change; you’d want to test it to see if it lifted your metrics.

But what if ABC Company were purchased by XYZ Company and it decided to discontinue the ABC brand? This isn’t a testing situation, since keeping the ABC brand in the sender line isn’t an option. Here I’d recommend a transition strategy for the sender line; perhaps make it “XYZ (formerly ABC)” for a while, then migrate to “XYZ” after you feel people are familiar with the new moniker.

Subject Line Changes

Always, always test. There’s so much advice out there on subject lines, and much of it is conflicting. The only way to find out what gives you the best chance at getting the open is to test. This is just as critical as the sender line; though you don’t want to use the same exact subject line each time (which you do want to do with the sender line), you do want to have a formula to follow. If you’re selling music, for example, your formula may be to say “New Music:” and then to list a few bands whose new releases you’re featuring.

Tag Line Changes

Many email newsletters have tag lines after their titles (in the body of the email, not the sender or subject line) that tell the reader a bit about the e-newsletter. Though important, these rarely lift or depress response; most people who read your email newsletter every month probably don’t read the tag line anymore. In most cases, the tag line isn’t critical to the reader experience, so changing it doesn’t warrant an A/B split test. Make the change and test something more likely to have an impact, like the sender or subject lines, format, or email content.

Try it for yourself and let me know how it goes!


Please take the time to participate in ClickZ’s short reader survey, which will be live until March 3. To thank you for your participation, ClickZ will make a donation to charity on your behalf for every survey completed. Details of the charities you can nominate are provided at the end of the questionnaire.

Want more email marketing information? ClickZ E-Mail Reference is an archive of all our email columns, organized by topic.

Related reading