Home  › Email › Email Marketing

Testing: Why Failure Must Be an Option

  |  October 10, 2011   |  Comments

Better to test a small bit of your list and experience some small failure, then launch a mailing to your entire list and experience failure in a massive way.

Several years ago I was speaking at a marketing conference and had to follow NASA icons Gene Kranz and Jim Lovell, from the historic Apollo 13 mission that never landed on the moon. Mr. Kranz repeatedly used the phrase "failure is not an option" when describing his days running mission control and most certainly the story of Apollo 13. "Failure is not an Option" is also the title of his book that he was promoting at the time. As I listened and contemplated how I was going to follow these two American icons and pivot the audience from exciting space stories to discussing the need for relevant marketing, it occurred to me that; "failure must be an option." So I got up to the stage and reminded the audience that I was from Jupiter and knew nothing about bringing men back safely from the moon, however, that for us as marketers, we are one of the few professions where failure is not only tolerated, but actually is a necessary ingredient to optimizing programs. Failure is an element of testing; even simple A/B tests are predicated on the notion that one part will outperform (succeed) the other part, which is a degree of failure.

At my firm, we do a lot of survey work and market research, and recently we completed a survey that had some staggering findings. In a survey last month of 368 U.S. marketing executives, we asked them to identify marketing tactics that they regularly utilize (at least once a month). The data revealed that just 31 percent of the executive respondents stated that they conduct A/B testing for their email marketing campaigns. When we asked about more sophisticated types of testing methods, those results tracked even lower. Perhaps the real failure here is that so few email marketing executives are using testing as a scientific tactic to optimizing mailings; instead, most are simply guessing about which elements would optimize the performance of their mailings.

Granted, employing testing does increase production time and often requires more labor resources, but embarking on a new segmentation scheme without properly testing it is a futile exercise that wastes resources. Additionally, I did a research study way back in 2004 that proved that email marketers that were regularly testing had higher conversion rates than those that did not. Testing can make a real difference to your program and here are some suggestions on how to get started with the basics.

  • Test only one element or variable at a time. For example, test different offers and test different creative formats, but do not test both at the same time. This will help to determine which changes are driving the campaign's performance.
  • Expose customers to tests at the same time. In order to reduce the number of variables, execute tests on the same day, in the same mail stream. This will account for any day-of-week response fluctuations. Alternatively, you will likely want to test day-of-week performance, but as with the aforementioned note, you shouldn't also be testing subject line length. Again, testing is identifying which variable is making the most difference, and to start, keep it simple; test one element at a time.
  • Tests must be measurable and statistically relevant. While it's easier to create accurate samples in a 50/50 A/B split, marketers employing percent or nth testing must ensure that the test cells are evenly split and large enough to be more than directional anecdotal data. Many ESPs suggest that each test cell should result in at least 100 responses, requiring marketers to back into their desired results sample after taking into account the likely response rate. A general rule is to allocate at least 5 percent of the total list for testing.
  • Maintain a control group. For comparison purposes, create and maintain a control group that does not get exposed to any of the variables being tested. Such an unadulterated group will allow test results to be contrasted to this static group over time.
  • Get help. Working with a marketing service provider, agency, or consultant that has experience in different testing methodologies will reduce your knowledge gap with testing and speed your time to realize the optimized mailing. There are several great resources and books that provide much greater detail on testing best practices, so be sure to make use of them.

While email marketing is not rocket science, marketing does necessitate some science. Be thankful that you are a marketer and embrace that notion of optimization that a little bit of failure is a necessary part of the practice of scientific marketing. Better to test a small bit of your list and experience some small failure, then launch a mailing to your entire list and experience failure in a massive way. Test early and test often and make it an element of your production process.

Until next time, keep on mailing.
David

David is off today. This column was originally published on June 20, 2011 on ClickZ.

ClickZ Live Chicago Learn Digital Marketing Insights From Leading Brands!
ClickZ Live Chicago (Nov 3-6) will deliver over 50 sessions across 10 individual tracks, including Data-Driven Marketing, Social, Mobile, Display, Search and Email. Check out the full agenda, or register and attend one of the best ClickZ events yet!

ABOUT THE AUTHOR

David Daniels

For more than 20 years, David has been an industry proponent. Direct Magazine said David is "one of the most influential experts in email marketing, if not the most influential." Co-author of "Email Marketing An Hour A Day," David has held senior level positions at Forrester and JupiterResearch, Apple, Anthropologie, MacWarehouse, Proteam, and retailers that dotted the early days of CompuServe. David advises many industry organizations including the OTA, DMA, eec, and has been a contributor to the Weekend Today Show on NBC. Learn more about connected marketing and download free research with registration here. Follow David on Twitter @emaildaniels and learn more at www.relevancygroup.com.

COMMENTSCommenting policy

comments powered by Disqus

Get ClickZ Email newsletters delivered right to your inbox. Subscribe today!

COMMENTS

UPCOMING EVENTS

UPCOMING TRAINING

Featured White Papers

Google My Business Listings Demystified

Google My Business Listings Demystified
To help brands control how they appear online, Google has developed a new offering: Google My Business Locations. This whitepaper helps marketers understand how to use this powerful new tool.

5 Ways to Personalize Beyond the Subject Line

5 Ways to Personalize Beyond the Subject Line
82 percent of shoppers say they would buy more items from a brand if the emails they sent were more personalized. This white paper offer five tactics that will personalize your email beyond the subject line and drive real business growth.

WEBINARS

    Information currently unavailable

Resources

Jobs

    • Executive Assistant
      Executive Assistant (Agora Inc. ) - BaltimoreAgora Inc., an international publishing company, located in the Mt. Vernon district of Baltimore, MD...
    • Paid Search Specialist
      Paid Search Specialist (Boathouse, Inc.) - Waltham  Boathouse is looking for a Paid Search Specialist to work as a part of the Digital Acquisition...
    • Paid Search / Search Engine Marketing (SEM, PPC) Specialist
      Paid Search / Search Engine Marketing (SEM, PPC) Specialist (HeBS Digital) - New YorkJOB TITLE:     Paid Search / Search Engine Marketing...