Delivery Reports: What They Mean, How to Read Them

Because delivery performance is so important to your e-mail program’s success, you need to know whether the message that goes out at your end is getting delivered, not just to the destination Internet service providers, but to your subscribers’ inboxes.

Two sets of reports, used together, can give you the most complete picture of your true delivery results. These are the delivery reports by your e-mail service provider (ESP) or your e-mail sending software, and reports generated by delivery-monitoring services.

Delivery Reporting From Your ESP

Delivery tracking starts with the delivery reports from your e-mail service provider. Most ESP delivery reporting is based on feedback from a receiver, such as an ISP. It follows this equation: “Sent Messages” minus “Bounces/Undeliverables” equals “Delivery.”

This reporting assumes that the message got delivered if the ISP doesn’t indicate otherwise. Most ESPs count both inbox and spam-folder placement as delivered.

Conventional wisdom says your total bounce rate should be between 1 and 5 percent if you’re doing proper list hygiene and have no delivery challenges. However, bounce rates don’t tell the whole story.

Good ESP delivery reporting also breaks down metrics by domain, allowing you to identify where you might have specific problems that need to be resolved.

Recently, many senders experienced delivery challenges at Yahoo. Domain-level reporting quickly identified this challenge was specific to Yahoo and allowed senders to focus on resolving the problems without interrupting sending to other domains.

Besides bounce metrics, you should also know your “engagement” metrics at each domain. Delivery reports won’t report messages that went to the bulk or spam folder instead of the inbox. However, they will show open, click, and conversion metrics.

If your engagement metrics at one major ISP are dramatically lower compared to others, your messages likely aren’t reaching the inbox.

“Seedbox” Monitoring Patrols the Inbox

Seedbox-monitoring services, such as Pivotal Veracity and Return Path, also report delivery performance, but differently from your ESP.

They measure success by inbox placement. Reports show what percentages of your e-mails reach the inbox, go to the bulk folder, and go missing (undelivered).

Unlike ESP numbers, these numbers aren’t absolute. Typically, they are sample sets of 10 to 20 “seed” e-mail addresses sent to each ISP the service monitors, for a total of 200 to 300 seed accounts.

These services assume that when the first message for a specific e-mail campaign is received at a seed address, all the other seed addresses should receive the message, too. Your delivery rate begins at 1 percent delivered and climbs as more seeds receive e-mails.

These services benchmark multiple senders and typically report an aggregate expected inbox delivery rate of 80 percent. This means 20 percent of messages never reach the inbox, although your ESP often counts these as delivered messages.

As mentioned above, these services use sample sets of messages instead of total numbers. This doesn’t limit their effectiveness. I trust and often recommend using these tools. However, where the seeds are placed during the message send can affect the results.

Seeds sent first or in a test send might predict 100 percent inbox delivery. Seeds sent later might predict 100 percent missing for specific ISPs, depending on what happened during your send. Seeds dispersed randomly can report 30 percent missing e-mail — a metric that doesn’t truly reflect the total number of messages, only the number of seeds received.

For example, Hotmail will throttle the total number of messages you send on a given day based on your reputation score. If your seeds are at the start of your send, all might be delivered before throttling happens.

If throttling kicks in, your ESP might report large bounce rates and low opens/click/conversions compared to other domains, even though your seeds gave you the all-clear.

Conversely, if throttling occurred near the end of your send, your seeds at the end of the file might report full blocking. Your ESP might report only slightly lower engagement metrics, while your seed monitor shows complete blocks.

If you believe you have a blocking issue at a major ISP, (the message is undelivered, not bulk-foldered), find the bounce message for that seed in your ESP’s bounce report and verify that the message was blocked.

Beware: Giving Equal Weight to All ISPs

Seedbox monitoring tools often have the same number of seeds for smaller ISPs, such as Lycos, Excite, and Juno, as they do for major ISPs like Yahoo, Hotmail, and AOL, even when sender lists skew heavily toward the majors.

So, you might have a 30 percent non-inbox delivery rate, according to the seed-monitoring tool, at ISPs that make up only between 1 and 3 percent of your total list demographics.

You shouldn’t take nondelivery lightly at those ISPs. But keep it in perspective when comparing overall scores, because the delivery problem is much smaller than represented. Sometimes, the seed tool allows you to weight the percentage and number of seeds accurately, but many senders often don’t change the default setting.

What’s a Sender to Do?

Understand the difference in reporting techniques. Don’t simply rely on one versus the other as a true measure of your delivery rates. I advise senders to use both types of monitoring, because each has strengths and limitations. When resolving delivery problems, more information is always better than less information.

ESP monitoring is often incomplete. While using real numbers, it might not present a complete picture of your message placement.

Seedbox monitoring can uncover delivery challenges quickly, but it doesn’t always indicate the proper magnitude of the problems. However, the alerts tell you about potential problems and should prompt you to investigate further.

When these reports are used together, you will have more information and be empowered to make the changes required to receive full inbox placement and drive higher results.

Until next time, keep on deliverin’!