Understand ISP-Level E-Mail Filtering

Got filtered? Of course you have. Your messages have been filtered by ISPs for the past few years.

E-mail filtering is a necessary part of fighting spam. As ClickZ News reported earlier this year, AOL receives roughly 2 billion email messages a day, of which about 75 percent are blocked and another 4 to 7 percent are sent to the bulk folder.

Dealing with this influx of spam is a major issue for ISPs. It’s a tremendous drain on servers and human resources. It’s also a major source of customer complaints, dissatisfaction, and churn. Minimizing the flow of spam to inboxes has become a major point of differentiation as ISPs battle to acquire and retain customers.

Long ago, smart marketers moved past the anger phase, all the way to acceptance. They’re actively deploying a number of techniques to minimize ISP filtering.

Like the search engines’ secretive page-ranking algorithms, ISPs don’t share the specific types of filtering they deploy nor at what thresholds. Through dealing with ISPs over time, it’s apparent there are quite a few key components. Though they may be approached differently at various companies, all are quite commonly used.

ISPs deploy a number of methods to reduce the volume of spam. They may include looking at:

• Sender reputation (e.g., blacklists)
• Sender authenticity
• Volume of messages sent
• Volume/percentage of invalid addresses (hard bounces)
• Message content
• Spam complaints

Let’s consider a fictitious, consumer-oriented ISP with 5-10 million subscribers. Typical email filtering happens at multiple levels. It begins before the message even arrives at the ISP’s servers. At the top of the filtering pyramid are:

• Blacklists. To check a sender’s reputation, the ISP can tap a wealth of information collected at publicly available blacklists or internal blacklists built over time. Public blacklists come in many flavors. An ISP can select those that most closely match its policies. Private internal blacklists are usually built through customer feedback, primarily from spam complaints. Blacklists allow an ISP to quickly identify potential connecting sources as undesirable and to refuse a connection prior to accepting an incoming message.
• SMTP transactions. Much can be determined about a mail source from the way it talks to a receiving server. Spam-like behavior, such as cramming hundreds of addresses into an address line, can be detected at this step. A check of authentication, such as SPF (define) records to check for phishing attempts, occurs here. Other anti-spam tactics, such as a reverse DNS (define) check, can verify the connecting IP belongs to the domain it says it does.
• Mail streams. The ISP also looks at the behavior of the message stream as a whole. For example, if the message is sent to an unusually high number of invalid addresses, it’s most likely a dictionary attack (define). If recipients who already received this message are complaining en masse through a “report spam” button, the message is probably spam. The rest of the transmissions are interrupted or redirected to the spam folder.
• Top-level filters. Once the server connections are made and the message content arrives, a top-level content filter can weed out obvious spam by scanning the message body. Obvious red flags are picked up at this step, including sexually explicit material, offers for brand-name prescription drugs, blatant attempts to obfuscate words (“m0rtG@|ge”), and common spammer tricks, such as falsifying header information.
• User-level filter. Filtering at this level is personal and created by the end recipient. Through a “learning filter,” typically a Bayesian filter, the ISP can identify patterns in the type of email the user usually receives, as well as patterns in messages that user typically designates as spam. Through these patterns, a fairly accurate judgment call can be made as to whether the incoming email is legitimate or spam.

Large ISPs tend to handle all these steps in-house. Some ISPs contract with anti-spam companies to provide some or all of these services. Most ISPs have unique filtering methods they find work for their needs. They even use their unique methods as selling points.

In upcoming columns, we’ll look at best practices and third-party services that help monitor and resolve ISP filtering issues.

Until next month, keep on deliverin’.

Want more email marketing information? ClickZ E-Mail Reference is an archive of all our email columns, organized by topic.