When Will AI Help Solve Spam?

  |  January 22, 2004   |  Comments

Targeting spam with artificial intelligence and concept technologies.

I was at a project in Vermont for five weeks and had a problem that isn't new to anyone reading this column. I was forced to use a dial-up connection to access my email. Was it slow! I used a "download message headers only" feature to view only the message envelopes, saving me from downloading all that spam. But Outlook's filters made a mistake. My client tried to send an email three times before I realized why I never got it: The word "debt" appeared in the message's body copy. I have a filter that zaps all email containing words such as "debt" or "mortgage." This email wasn't spam. It just happened to use that spam-like word.

What's the solution? Maybe it's artificial intelligence (AI).

Regular readers know my pre-marketing background is in AI and its e-commerce applications. I've been involved in the development of many technologies over the years. A few could really help fight the spam wars. Yet no one really seems to be using them for that purpose.

The problem with current spam filters is they operate on keywords and heuristics. Outlook's rules engine and similar spam filters use keyword matching (searching for words) to identify terms that indicate spam. A lot of spam these days uses characters such as "$" instead of "S" in subject lines. They do this to elude keyword-matching filters.

Service providers also use heuristics, such as the number of email messages sent from a particular address to addresses within their services, to determine if something might be spam. The problem with this, of course, is nonspam email (such as newsletters) are also sent in bulk and exhibit the same heuristics as spam. Obviously the keyword/heuristic approach isn't working.

Several technologies already exist that can be refitted to help fight spam. Let's look at a few.

Content Abstraction

Content abstraction (natural language processing) technology looks at unstructured content (e.g., a news article) and creates abstracts that convey, in a short paragraph, the essence of the article. News services use this technology to create automatic abstracts to display in search results.

Two types of technologies are commonly used for this: those that output human-readable text and those that generate a "concept fingerprint" of the article. A human can't understand this fingerprint, but the computer can. Active Navigation outputs human readable abstracts, whereas Autonomy use the concept fingerprint.

Why are these technologies interesting as anti-spam weapons? They can identify concepts and work on unstructured email (e.g., a message body). They're perfect for understanding the essence of what an email communication is about. In the above example, these technologies would have understood my client's email containing the word "debt" was a business proposal I had to review, not a solicitation for a credit card.

These technologies are concept-based, not keyword-based. That means it doesn't matter if someone calls it "Viagra," "V_i_a_g_r_a," or "that little blue pill." The concept engine understands the email wants to sell you some type of drug.

Neural Networks

Neural networks are a technology used by credit card companies to help identify credit card fraud. The basic premise is you train a neural network to identify patterns inherent to fraudulent behavior. The system analyzes new purchase patterns and raises a red flag if it thinks a purchase may be fraudulent. Rules-based systems (like the one in Outlook) are also used to weed out common purchase situations that are most likely fraudulent.

Two weeks ago, my credit card company called me while I was in England. Within the same week I used the card to buy plane tickets in New York, subway cards in London, and several large purchases in Germany. A combination of neural-network pattern matching and rules-based purchase patterns contributed to the company calling to ensure the card wasn't stolen.

Bayesian Networks

Bayesian networks are currently used to help determine if something is spam, but they're based on keyword tokens and heuristics surrounding the tokens (how close together they appear, where they appear, etc.), not on concepts. Bayesian networks, like neural networks, would do a terrific job if inputs were concept-based instead of keyword-based.

Putting It All Together

Combined with concept fingerprint technologies, neural networks can be trained to identify which concepts are likely spam. As new email arrives, the content abstraction system can send the concept to the neural network, then to the rules-based system. These can weed out unwanted email.

Obviously, spam is far from over. Keyword-based rules engines and heuristic models to identify spam aren't as effective as we'd hoped. We must use different technologies that identify spam based on message concept, not just its use of words. Using concept fingerprints to identify spam will help eradicate much more spam.

No matter what they call that little blue pill, they're still trying to sell it to you.

What are your thoughts? Let me know!

Until next time...

Jack

ClickZ Live Toronto Twitter Canada MD Kirstine Stewart to Keynote Toronto
ClickZ Live Toronto (May 14-16) is a new event addressing the rapidly changing landscape that digital marketers face. The agenda focuses on customer engagement and attaining maximum ROI through online marketing efforts across paid, owned & earned media. Register now and save!

ABOUT THE AUTHOR

Jack Aaronson

Jack Aaronson, CEO of The Aaronson Group and corporate lecturer, is a sought-after expert on enhanced user experiences, customer conversion, retention, and loyalty. If only a small percentage of people who arrive at your home page transact with your company (and even fewer return to transact again), Jack and his company can help. He also publishes a newsletter about multichannel marketing, personalization, user experience, and other related issues. He has keynoted most major marketing conferences around the world and regularly speaks at Shop.org and other major industry shows. You can learn more about Jack through his LinkedIn profile.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!

COMMENTS

UPCOMING EVENTS

Featured White Papers

ion Interactive Marketing Apps for Landing Pages White Paper

Marketing Apps for Landing Pages White Paper
Marketing apps can elevate a formulaic landing page into a highly interactive user experience. Learn how to turn your static content into exciting marketing apps.

eMarketer: Redefining Mobile-Only Users: Millions Selectively Avoid the Desktop

Redefining 'Mobile-Only' Users: Millions Selectively Avoid the Desktop
A new breed of selective mobile-only consumers has emerged. What are the demos of these users and how and where can marketers reach them?

Jobs

    • Contact Center Professional
      Contact Center Professional (TCC: The Contact Center) - Hunt ValleyLooking to join a workforce that prides themselves on being routine and keeping...
    • Recruitment and Team Building Ambassador
      Recruitment and Team Building Ambassador (Agora Inc.) - BaltimoreAgora, www.agora-inc.com, continues to expand! In order to meet the needs of our...
    • Design and Publishing Specialist
      Design and Publishing Specialist (Bonner and Partners) - BaltimoreIf you’re a hungry self-starter, creative, organized and have an extreme...