Is E-mail Really Measurable?

  |  January 24, 2008   |  Comments

The three issues standing between e-mail marketers and accurate campaign reporting

It's long been known figures for e-mail delivery, bounce, open, and click vary from sender to sender. To make matters worse, it's not just the value measurements that vary, but even the very definitions.

This issue has come to the attention of some industry heavy hitters. JupiterResearch is leading the creation of a new industry group to bring some consistency to e-mail metrics. The Email Measurement Accuracy Coalition (EMAC) formed recently with a remit to: "establish a consistent methodology and framework for the accurate calculation of e-mail delivery, in order to inform the computation of critical e-mail marketing metrics such as open, click-through-rate and conversion."

I feel the pain of the mess and inconsistency that are e-mail marketing metrics. This is a laudable and important goal I fully support, but I'm not holding my breath for results.

Here's why.

Delivery

Delivery is at the heart of the coalition's remit. Unfortunately, it's perhaps the least-defined term in e-mail marketing.

The first issue is bounces, specifically which bounces should count as failures. This topic alone has generated long, ongoing discussions without solid agreement. Even with accurate bounce measurement, there are many situations in which e-mail goes undelivered without ever generating a bounce (Hotmail is notorious for this).

Beyond bounces, there's the question of delivery to the inbox or the bulk folder. This is an important distinction. Delivery to the bulk folder is substantially different from delivery to the inbox. However, no systems are available to accurately measure inbox delivery across ISPs. Goodmail reports this information for a few ISPs, but it doesn't seem practical to base industry standard metrics on a single provider's systems, particularly a paid service that only functions with a very small number of ISPs.

Without inbox delivery measurement, any delivery figure is at best an estimate and hardly solid grounds for consistent, accurate measurement.

Opacity

Beyond the difficulty of actually defining and measuring delivery, there are political and business problems to tackle. The truth is for many e-mail providers, the perception of high delivery and response rates are a competitive advantage.

This has two effects. First is it isn't in these providers' interests to be transparent. If they are, perceived advantage evaporates. Second is some providers ensure their metrics are to their advantage. I'm not suggesting anyone's being dishonest. It's just that, as Disraeli said, "there are three kinds of lies: lies, damned lies, and statistics." And this is all about statistics.

A provider whose delivery and response rates plummeted due to metric standardization would have a pretty difficult time explaining this to its clients.

Entrenchment

Finally, many of the current metrics and measurement methods are deeply entrenched both psychologically and in the software systems we operate.

We have clients that are seriously attached to total click-throughs, unique click-throughs, and even open rates based on their current delivery definitions. Redefining these metrics would cause significant discomfort. All the figures they're familiar with and all the data that's been collected would be rendered obsolete.

On the technological side, altering service providers' systems to calculate CTRs (define) differently or classify bounces differently will take years for some of them. Instituting transitional measurements with both old and new figures calculated simultaneously to ease migration would still cause much confusion.

Despite these issues, it's not all doom and gloom. Some very smart people are working on this, and EMAC has the backing of many industry heavyweights, including many of the largest e-mail service providers. They may conclude key response metrics shouldn't be based on delivery counts but on send counts. It would be a tough sell, but it would circumvent many thorniest issues.

I await with interest their first pronouncements. My only hope is they truly address the issues and don't get bogged down in committee and produce something so watered down it only serves to further muddy already cloudy waters.

Until next time,

Derek

Derek is off this week. Today's column ran earlier on ClickZ.

Want more e-mail marketing information? ClickZ E-Mail Reference is an archive of all our e-mail columns, organized by topic.

ClickZ Live New York What's New for 2015?
You spoke, we listened! ClickZ Live New York (Mar 30-Apr 1) is back with a brand new streamlined agenda. Don't miss the latest digital marketing tips, tricks and tools that will make you re-think your strategy and revolutionize your marketing campaigns. Super Saver Rates are available now. Register today!

ABOUT THE AUTHOR

Derek Harding

Derek is the managing director of J-Labs, Javelin Marketing Group's technology skunkworks, a role that draws on his 20 years of experience and leadership in the fields of marketing and technology. A British expatriate based in Seattle, Washington, Derek is perhaps better known as the founder and technologist behind Innovyx, one of the first email service providers later acquired by the Omnicom Group. An industry veteran and thought-leader, Derek is a regular expert author, contributor, conference speaker, and takes an active role in a number of industry and trade groups.

COMMENTSCommenting policy

comments powered by Disqus

Get ClickZ Email newsletters delivered right to your inbox. Subscribe today!

COMMENTS

UPCOMING EVENTS

UPCOMING TRAINING

Featured White Papers

Google My Business Listings Demystified

Google My Business Listings Demystified
To help brands control how they appear online, Google has developed a new offering: Google My Business Locations. This whitepaper helps marketers understand how to use this powerful new tool.

5 Ways to Personalize Beyond the Subject Line

5 Ways to Personalize Beyond the Subject Line
82 percent of shoppers say they would buy more items from a brand if the emails they sent were more personalized. This white paper offer five tactics that will personalize your email beyond the subject line and drive real business growth.

WEBINARS

    Information currently unavailable

Jobs

    • Lead Generation Specialist
      Lead Generation Specialist (The Oxford Club) - BaltimoreThe Oxford Club is seeking a talented writer/marketer to join our growing email lead-generation...
    • Health Marketing Editor
      Health Marketing Editor (Agora Inc.) - BaltimoreCome flex your intellectual muscle as part of Agora, Inc’s (http://agora-inc.com/) legal team...
    • Technical Business Analyst
      Technical Business Analyst (OmniVista Health) - BaltimoreOmniVista Health is looking to add a Technical Business Analyst to our expanding team...