Home  › Email › Email Marketing

Which Vendor Is Better?

  |  March 5, 2007   |  Comments

Why can't the e-mail industry agree on measurement metrics? Two professionals weigh in.

It happened again last week. A client sent an e-mail campaign using a new vendor. When the results came in, they were phenomenal! More than 3 times the open rate and 10 times the click-through rate, as compared with the prior campaign. We were thrilled... until we learned the results were actually the same as they were with the old vendor. They merely appeared higher because they were calculated differently.

This is the third time this has happened in one year. I'm tired of it. Why can't this industry agree on measurement metrics? Is this even an issue that anyone else has noticed?

To find out, I asked two experts in the e-mail space: Deirdre Baird, CEO of Pivotal Veracity, and Dave Lewis, VP of market development for StrongMail Systems. Here's what they told me.

Jeanniey Mullen: Do other people feel a lack of measurement standards is a real issue that affects a number of marketing efforts?

Deirdre Baird: I chair the Email Experience Council's deliverability roundtable. Our first initiative was to assess the state of industry metrics and bounce management. This week, we're releasing an extensive whitepaper and the survey results from 321 mailers and 29 ESPs [e-mail service providers], representing thousands of client companies. What we discovered is others feel the same way you do, and we've also uncovered three key reasons:

  • Conflicting metrics. There's a lack of consistency in calculating key performance metrics (delivery, open, click rates) that makes it impossible to establish industry benchmarks or compare results.

  • Inconsistent bounce data and definitions. Getting standardized, accurate bounce data from ISPs is a top concern, but there's no industry consensus on what the key terms mean, such as "hard bounce," or how they should be applied.

  • Inadequate bounce management. Everyone agrees that e-mail deliverability is very important, but many lack the reporting systems to really understand their results or act on them.

These results paint an alarming picture and should serve as an industry wakeup call to address our inability to define, calculate, view, and act on key metrics.

Dave Lewis: The consequences of what Deirdre has outlined are important to understand. Without reliable metrics and performance data, e-mail marketers are flying blind. They don't know their "real" results, as your client discovered. Nor can they adequately maintain their lists or proactively manage their practices, all of which increases their deliverability risk. This situation makes it extremely difficult to compare results across vendors/solutions or even to value e-mail relative to other channels. I see this latter point as a threat to the channel itself. Without standardized metrics, you're significantly disadvantaged when it comes to the budget battles over marketing dollars.

JM: Given that these issues are broad, how do you see marketers and mailers reacting to this information? Can they leverage it to improve what they are doing?

DL: As Deirdre said, our findings should serve as a wakeup call for the industry, including mailers. As every practitioner of DM knows, the devil is in the details. E-mail marketing is no different. Our report clearly indicates the areas that mailers should examine in assessing the quality of their own performance metrics, bounce definitions, bounce management practices, or those of the service provider that supports them. In a very real sense, we're empowering mailers with the information to ask the right questions, and that will contribute to the momentum for change.

JM: How do you think ESPs will react? Do you think it is possible for all ESPs to ever agree to some standardized processes?

DB: I believe ESPs will respond favorably to any endeavor that is in the best interest of their clients. We shouldn't forget that ESPs are forced, due to inadequate, incomplete, and sometimes inaccurate data supplied by the ISPs, to try to create meaningful metrics for their clients to use. Their efforts in this respect are almost universally more successful than that which in-house mailers have accomplished and should be applauded. However, even if ESPs were all to agree to utilize the same metrics, their ability to calculate metrics with absolute accuracy will be hampered by the data they receive from ISPs. So we need to be realistic about all metrics and, in addition to defining them, be clear on the limitations.

We aren't espousing that certain metrics be abandoned for others. Even though different ESPs calculate certain metrics differently, that doesn't mean the metrics they are using have no value. Click rates, when calculated using unique clickers versus total clicks, are both valuable metrics, and an ESP who uses one versus the other would certainly be able to present a sound argument as to their choice. We aren't suggesting one should be abandoned, but rather that we either use different names for these two different metrics or, minimally, clearly qualify on reports how they are calculated so that mailers are more readily cognizant of what they are measuring.

JM: Realistically, how long do you think it will take for the industry get to a point where we have standards that work?

DL: There's no question that achieving consensus on standardized metrics, definitions, and practices will be the hard part. But the basis for that consensus is in our findings, and I'm confident we'll get there, partly because we have no other choice. Our current metrics muddle serves no one, and e-mail marketing has matured to a point where standard metrics and measures are required, especially if we're to hold our own with other DM channels. The catalyst for change will come from two sectors: the mailers who will demand consistent metrics and more visibility into their results from the IT departments and vendors that support them, and the industry analysts and pundits who assess the various service and solution providers and examine e-mail marketing in a multichannel context.

Want more e-mail marketing information? ClickZ E-Mail Reference is an archive of all our e-mail columns, organized by topic.

ClickZ Live San Francisco This Year's Premier Digital Marketing Event is #CZLSF
ClickZ Live San Francisco (Aug 11-14) brings together the industry's leading practitioners and marketing strategists to deliver 4 days of educational sessions and training workshops. From Data-Driven Marketing to Social, Mobile, Display, Search and Email, this year's comprehensive agenda will help you maximize your marketing efforts and ROI. Register today!


Jeanniey Mullen

Jeanniey Mullen, a recognized women-in-business and tech, is known for her entrepreneurial style and her ability to build, shape, and grow brands into well-known dominant, successful entities. Jeanniey is a pioneer in email, mobile, and digital marketing; publishing; and brand-building. She now leads her own agency, YellowBean LLC, focused on assisting companies of all sizes with driving innovation and growth. Most recently, Jeanniey was the Global EVP, CMO, and subsequently Chief Growth Officer for Zinio, where she worked to define and implement strategies creating explosive growth through strategic partnerships with publishers, technology companies, brands, and consumers during her five-year tenure. Jeanniey has authored and contributed to multiple books, blogs, and magazine articles. She is a regular columnist for ClickZ, a blogger for Huffington Post, and a frequent keynote speaker. A serial networker, in 2005 Jeanniey founded the Email Experience Council, which was sold to the Direct Marketing Association in 2008. She sits on the Advisory Board for IndieFlix, and on the International Executive Council of the Internet Marketing Association. Jeanniey is recognized as both a Top CMO and Top Author on Twitter, and was most recently featured as Mover and Shaker by the Professional Woman's Magazine, and a featured Woman in Technology by The Legacy Series Magazine.

COMMENTSCommenting policy

comments powered by Disqus

Get ClickZ Email newsletters delivered right to your inbox. Subscribe today!



Featured White Papers

BigDoor: The Marketers Guide to Customer Loyalty

The Marketer's Guide to Customer Loyalty
Customer loyalty is imperative to success, but fostering and maintaining loyalty takes a lot of work. This guide is here to help marketers build, execute, and maintain a successful loyalty initiative.

Marin Software: The Multiplier Effect of Integrating Search & Social Advertising

The Multiplier Effect of Integrating Search & Social Advertising
Latest research reveals 68% higher revenue per conversion for marketers who integrate their search & social advertising. In addition to the research results, this whitepaper also outlines 5 strategies and 15 tactics you can use to better integrate your search and social campaigns.


    Information currently unavailable


    • Interactive Product Manager
      Interactive Product Manager (Western Governors University) - Salt Lake CityWestern Governors University, one of the 20 largest universities...
    • SEO Senior Analyst
      SEO Senior Analyst (University of Phoenix (Apollo Education Group)) - San FranciscoSEO Senior Analyst   Position Summary...
    • SEM & Biddable Media Manager
      SEM & Biddable Media Manager (Kepler Group LLC) - New YorkAs an Optimization & Innovation Manager at Kepler Group, you will be on the bleeding...