When Will AI Help Solve Spam?

I was at a project in Vermont for five weeks and had a problem that isn’t new to anyone reading this column. I was forced to use a dial-up connection to access my email. Was it slow! I used a “download message headers only” feature to view only the message envelopes, saving me from downloading all that spam. But Outlook’s filters made a mistake. My client tried to send an email three times before I realized why I never got it: The word “debt” appeared in the message’s body copy. I have a filter that zaps all email containing words such as “debt” or “mortgage.” This email wasn’t spam. It just happened to use that spam-like word.

What’s the solution? Maybe it’s artificial intelligence (AI).

Regular readers know my pre-marketing background is in AI and its e-commerce applications. I’ve been involved in the development of many technologies over the years. A few could really help fight the spam wars. Yet no one really seems to be using them for that purpose.

The problem with current spam filters is they operate on keywords and heuristics. Outlook’s rules engine and similar spam filters use keyword matching (searching for words) to identify terms that indicate spam. A lot of spam these days uses characters such as “$” instead of “S” in subject lines. They do this to elude keyword-matching filters.

Service providers also use heuristics, such as the number of email messages sent from a particular address to addresses within their services, to determine if something might be spam. The problem with this, of course, is nonspam email (such as newsletters) are also sent in bulk and exhibit the same heuristics as spam. Obviously the keyword/heuristic approach isn’t working.

Several technologies already exist that can be refitted to help fight spam. Let’s look at a few.

Content Abstraction

Content abstraction (natural language processing) technology looks at unstructured content (e.g., a news article) and creates abstracts that convey, in a short paragraph, the essence of the article. News services use this technology to create automatic abstracts to display in search results.

Two types of technologies are commonly used for this: those that output human-readable text and those that generate a “concept fingerprint” of the article. A human can’t understand this fingerprint, but the computer can. Active Navigation outputs human readable abstracts, whereas Autonomy use the concept fingerprint.

Why are these technologies interesting as anti-spam weapons? They can identify concepts and work on unstructured email (e.g., a message body). They’re perfect for understanding the essence of what an email communication is about. In the above example, these technologies would have understood my client’s email containing the word “debt” was a business proposal I had to review, not a solicitation for a credit card.

These technologies are concept-based, not keyword-based. That means it doesn’t matter if someone calls it “Viagra,” “V_i_a_g_r_a,” or “that little blue pill.” The concept engine understands the email wants to sell you some type of drug.

Neural Networks

Neural networks are a technology used by credit card companies to help identify credit card fraud. The basic premise is you train a neural network to identify patterns inherent to fraudulent behavior. The system analyzes new purchase patterns and raises a red flag if it thinks a purchase may be fraudulent. Rules-based systems (like the one in Outlook) are also used to weed out common purchase situations that are most likely fraudulent.

Two weeks ago, my credit card company called me while I was in England. Within the same week I used the card to buy plane tickets in New York, subway cards in London, and several large purchases in Germany. A combination of neural-network pattern matching and rules-based purchase patterns contributed to the company calling to ensure the card wasn’t stolen.

Bayesian Networks

Bayesian networks are currently used to help determine if something is spam, but they’re based on keyword tokens and heuristics surrounding the tokens (how close together they appear, where they appear, etc.), not on concepts. Bayesian networks, like neural networks, would do a terrific job if inputs were concept-based instead of keyword-based.

Putting It All Together

Combined with concept fingerprint technologies, neural networks can be trained to identify which concepts are likely spam. As new email arrives, the content abstraction system can send the concept to the neural network, then to the rules-based system. These can weed out unwanted email.

Obviously, spam is far from over. Keyword-based rules engines and heuristic models to identify spam aren’t as effective as we’d hoped. We must use different technologies that identify spam based on message concept, not just its use of words. Using concept fingerprints to identify spam will help eradicate much more spam.

No matter what they call that little blue pill, they’re still trying to sell it to you.

What are your thoughts? Let me know!

Until next time…

Jack