When Will AI Help Solve Spam?

  |  January 22, 2004   |  Comments

Targeting spam with artificial intelligence and concept technologies.

I was at a project in Vermont for five weeks and had a problem that isn't new to anyone reading this column. I was forced to use a dial-up connection to access my email. Was it slow! I used a "download message headers only" feature to view only the message envelopes, saving me from downloading all that spam. But Outlook's filters made a mistake. My client tried to send an email three times before I realized why I never got it: The word "debt" appeared in the message's body copy. I have a filter that zaps all email containing words such as "debt" or "mortgage." This email wasn't spam. It just happened to use that spam-like word.

What's the solution? Maybe it's artificial intelligence (AI).

Regular readers know my pre-marketing background is in AI and its e-commerce applications. I've been involved in the development of many technologies over the years. A few could really help fight the spam wars. Yet no one really seems to be using them for that purpose.

The problem with current spam filters is they operate on keywords and heuristics. Outlook's rules engine and similar spam filters use keyword matching (searching for words) to identify terms that indicate spam. A lot of spam these days uses characters such as "$" instead of "S" in subject lines. They do this to elude keyword-matching filters.

Service providers also use heuristics, such as the number of email messages sent from a particular address to addresses within their services, to determine if something might be spam. The problem with this, of course, is nonspam email (such as newsletters) are also sent in bulk and exhibit the same heuristics as spam. Obviously the keyword/heuristic approach isn't working.

Several technologies already exist that can be refitted to help fight spam. Let's look at a few.

Content Abstraction

Content abstraction (natural language processing) technology looks at unstructured content (e.g., a news article) and creates abstracts that convey, in a short paragraph, the essence of the article. News services use this technology to create automatic abstracts to display in search results.

Two types of technologies are commonly used for this: those that output human-readable text and those that generate a "concept fingerprint" of the article. A human can't understand this fingerprint, but the computer can. Active Navigation outputs human readable abstracts, whereas Autonomy use the concept fingerprint.

Why are these technologies interesting as anti-spam weapons? They can identify concepts and work on unstructured email (e.g., a message body). They're perfect for understanding the essence of what an email communication is about. In the above example, these technologies would have understood my client's email containing the word "debt" was a business proposal I had to review, not a solicitation for a credit card.

These technologies are concept-based, not keyword-based. That means it doesn't matter if someone calls it "Viagra," "V_i_a_g_r_a," or "that little blue pill." The concept engine understands the email wants to sell you some type of drug.

Neural Networks

Neural networks are a technology used by credit card companies to help identify credit card fraud. The basic premise is you train a neural network to identify patterns inherent to fraudulent behavior. The system analyzes new purchase patterns and raises a red flag if it thinks a purchase may be fraudulent. Rules-based systems (like the one in Outlook) are also used to weed out common purchase situations that are most likely fraudulent.

Two weeks ago, my credit card company called me while I was in England. Within the same week I used the card to buy plane tickets in New York, subway cards in London, and several large purchases in Germany. A combination of neural-network pattern matching and rules-based purchase patterns contributed to the company calling to ensure the card wasn't stolen.

Bayesian Networks

Bayesian networks are currently used to help determine if something is spam, but they're based on keyword tokens and heuristics surrounding the tokens (how close together they appear, where they appear, etc.), not on concepts. Bayesian networks, like neural networks, would do a terrific job if inputs were concept-based instead of keyword-based.

Putting It All Together

Combined with concept fingerprint technologies, neural networks can be trained to identify which concepts are likely spam. As new email arrives, the content abstraction system can send the concept to the neural network, then to the rules-based system. These can weed out unwanted email.

Obviously, spam is far from over. Keyword-based rules engines and heuristic models to identify spam aren't as effective as we'd hoped. We must use different technologies that identify spam based on message concept, not just its use of words. Using concept fingerprints to identify spam will help eradicate much more spam.

No matter what they call that little blue pill, they're still trying to sell it to you.

What are your thoughts? Let me know!

Until next time...


ClickZ Live New York Want to learn more?
Attend ClickZ Live New York March 30 - April 1. With over 15 years' experience delivering industry-leading events, ClickZ Live brings together over 60 expert speakers to offer an action-packed, educationally-focused agenda covering all aspects of digital marketing. Register today!


Jack Aaronson

Jack Aaronson, CEO of The Aaronson Group and corporate lecturer, is a sought-after expert on enhanced user experiences, customer conversion, retention, and loyalty. If only a small percentage of people who arrive at your home page transact with your company (and even fewer return to transact again), Jack and his company can help. He also publishes a newsletter about multichannel marketing, personalization, user experience, and other related issues. He has keynoted most major marketing conferences around the world and regularly speaks at Shop.org and other major industry shows. You can learn more about Jack through his LinkedIn profile.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!




Featured White Papers

A Buyer's Guide to Affiliate Management Software

A Buyer's Guide to Affiliate Management Software
Manage your performance marketing with the right solution. Choose a platform that will mutually empower advertisers and media partners!

Google My Business Listings Demystified

Google My Business Listings Demystified
To help brands control how they appear online, Google has developed a new offering: Google My Business Locations. This whitepaper helps marketers understand how to use this powerful new tool.


    • Website Optimizer - SEO, CRO, Analytics
      Website Optimizer - SEO, CRO, Analytics (Marcel Digital) - ChicagoMarcel Digital, an award winning interactive marketing agency established in 2003...
    • Director of Marketing
      Director of Marketing (Patron Technology) - New YorkDirector of Marketing We are seeking a Director of Marketing to manage and build our marketing...
    • Senior Interactive Producer
      Senior Interactive Producer (Ready Set Rocket) - New YorkWhat You'll Do As a member of our team, the Senior Producer reports directly to our...