Home  › Email › Email Marketing

E-mail Testing Basics

  |  November 1, 2010   |  Comments

An in-depth look at an e-mail testing case study with a 30 percent lift in conversions.

One of the benefits of e-mail is the ease with which you can test - and the quickness with which you can get your results. But it's important to follow some best practices in order to get reliable results. Often it's desirable to test more than one item at once; the best way to do this is with a test grid.

test-grid

In this case study, we were testing a new layout and a different way to express the offer (the value of the offer remained the same). The key to the grid is Cell D; with it we can see whether the test offer (Cell B), the test layout (Cell C), or a combination of the two (Cell D) will produce the biggest lift. This is a simple concept, but one that many e-mail marketers without prior offline marketing experience miss.

Wireframes for the control layout and offer (Cell A) and the test layout and offer (Cell D) are below.

control-layout

test-layout

The modifications made to the control layout to create the test are minor, but important. We used learnings from a previous test to guide our redesign. This previous test failed to beat the control, but in analyzing the clicks we found that it drove a higher percentage of overall clicks to the primary call to action. We brainstormed a few reasons that this might be and addressed them in the revised layout:

  • No extraneous links in the header to distract readers from the primary call to action
    • So we removed the register call to action (which was not primary) as well as the website navigation
  • No white list request, which allowed more of the e-mail to be viewed in the preview pane
    • So we removed the white list request
  • A primary call to action button placed prominently between two paragraphs of copy
    • We were concerned about moving it down slightly in the layout, but felt that this placement made it stand out more
    • In order to accomplish this we split the original opening paragraph into two, which we felt also made it easier to skim
  • A brief sentence highlighting the exclusivity of the offer which appeared directly below the primary call to action button
    • We added this, verbatim, to the modified layout
  • A headline that appeared at the very top of the e-mail
    • In the original test the headline took the place of the logo, but we felt that keeping the logo would add credibility so we placed the headline next to it

For the offer test, we used exactly the same copy as the control in the sidebar and in the expanded list of bullets. This wasn't our first choice, but due to legal constraints we were not allowed to state the offer in a different way in these locations.

Instead we split the closing paragraph, which reiterated the offer (in the control language) and included the primary call to action as a text link. In between these paragraphs we added an additional paragraph which presented the offer in a different way and expanded on it.

If you look at the wireframes, they're really not all that different:

  • The items with the same copy and placement both appear in blue
  • The items with the same copy but different placement are yellow
  • The items which appear on one layout but not the other are in orange

None of these changes were expensive to make, so any lift would outweigh the cost incurred.

The results were impressive. Both Cells B and D, which featured the test offer, showed a lift over the control (Cell A) in click-through rate on the key call to action link. Cell B's performance was good, at +24 percent; Cell D did even better, beating the control by 36 percent on this metric.

click-through

Conversion metrics were a bit more interesting. Although Cell C had the same click-through rate as Cell A (the control), the conversion rate was actually 40 percent less. Cell B, which had nearly a quarter more clicks than Cell A, delivered 20 percent fewer conversions. But Cell D, which had beaten the control in clicks, also beat it in conversions - by 30 percent.

conversion

If we had tested just two cells against the control - one with a modified layout and the control offer and one with the control layout and the test offer, we would not have seen any increase in conversions and the control would have been declared the winner. But by combining the new layout with the new offer we were able to recognize a significant boost in the client's bottom line.

Words to the wise: always use a grid format when you're testing two variables. And always look at conversions, as well as clicks, to determine which cell won the test.

Until next time,

Jeanne

ClickZ Live San Francisco This Year's Premier Digital Marketing Event is #CZLSF
ClickZ Live San Francisco (Aug 11-14) brings together the industry's leading practitioners and marketing strategists to deliver 4 days of educational sessions and training workshops. From Data-Driven Marketing to Social, Mobile, Display, Search and Email, this year's comprehensive agenda will help you maximize your marketing efforts and ROI. Register today!

ABOUT THE AUTHOR

Jeanne Jennings

Jeanne Jennings is a 20 year veteran of the online/email marketing industry, having started her career with CompuServe in the late 1980s. As Vice President of Global Strategic Services for Alchemy Worx, Jennings helps organizations become more effective and more profitable online. Previously Jennings ran her own email marketing consultancy with a focus on strategy; clients included AARP, Hasbro, Scholastic, Verizon and Weight Watchers International. Want to learn more? Check out her blog.

COMMENTSCommenting policy

comments powered by Disqus

Get ClickZ Email newsletters delivered right to your inbox. Subscribe today!

COMMENTS

UPCOMING EVENTS

Featured White Papers

BigDoor: The Marketers Guide to Customer Loyalty

The Marketer's Guide to Customer Loyalty
Customer loyalty is imperative to success, but fostering and maintaining loyalty takes a lot of work. This guide is here to help marketers build, execute, and maintain a successful loyalty initiative.

Marin Software: The Multiplier Effect of Integrating Search & Social Advertising

The Multiplier Effect of Integrating Search & Social Advertising
Latest research reveals 68% higher revenue per conversion for marketers who integrate their search & social advertising. In addition to the research results, this whitepaper also outlines 5 strategies and 15 tactics you can use to better integrate your search and social campaigns.

WEBINARS

    Information currently unavailable

Jobs

    • Interactive Product Manager
      Interactive Product Manager (Western Governors University) - Salt Lake CityWestern Governors University, one of the 20 largest universities...
    • SEO Senior Analyst
      SEO Senior Analyst (University of Phoenix (Apollo Education Group)) - San FranciscoSEO Senior Analyst   Position Summary...
    • SEM & Biddable Media Manager
      SEM & Biddable Media Manager (Kepler Group LLC) - New YorkAs an Optimization & Innovation Manager at Kepler Group, you will be on the bleeding...