Targeting spam with artificial intelligence and concept technologies.
I was at a project in Vermont for five weeks and had a problem that isn't new to anyone reading this column. I was forced to use a dial-up connection to access my email. Was it slow! I used a "download message headers only" feature to view only the message envelopes, saving me from downloading all that spam. But Outlook's filters made a mistake. My client tried to send an email three times before I realized why I never got it: The word "debt" appeared in the message's body copy. I have a filter that zaps all email containing words such as "debt" or "mortgage." This email wasn't spam. It just happened to use that spam-like word.
What's the solution? Maybe it's artificial intelligence (AI).
Regular readers know my pre-marketing background is in AI and its e-commerce applications. I've been involved in the development of many technologies over the years. A few could really help fight the spam wars. Yet no one really seems to be using them for that purpose.
The problem with current spam filters is they operate on keywords and heuristics. Outlook's rules engine and similar spam filters use keyword matching (searching for words) to identify terms that indicate spam. A lot of spam these days uses characters such as "$" instead of "S" in subject lines. They do this to elude keyword-matching filters.
Service providers also use heuristics, such as the number of email messages sent from a particular address to addresses within their services, to determine if something might be spam. The problem with this, of course, is nonspam email (such as newsletters) are also sent in bulk and exhibit the same heuristics as spam. Obviously the keyword/heuristic approach isn't working.
Several technologies already exist that can be refitted to help fight spam. Let's look at a few.
Content abstraction (natural language processing) technology looks at unstructured content (e.g., a news article) and creates abstracts that convey, in a short paragraph, the essence of the article. News services use this technology to create automatic abstracts to display in search results.
Two types of technologies are commonly used for this: those that output human-readable text and those that generate a "concept fingerprint" of the article. A human can't understand this fingerprint, but the computer can. Active Navigation outputs human readable abstracts, whereas Autonomy use the concept fingerprint.
Why are these technologies interesting as anti-spam weapons? They can identify concepts and work on unstructured email (e.g., a message body). They're perfect for understanding the essence of what an email communication is about. In the above example, these technologies would have understood my client's email containing the word "debt" was a business proposal I had to review, not a solicitation for a credit card.
These technologies are concept-based, not keyword-based. That means it doesn't matter if someone calls it "Viagra," "V_i_a_g_r_a," or "that little blue pill." The concept engine understands the email wants to sell you some type of drug.
Neural networks are a technology used by credit card companies to help identify credit card fraud. The basic premise is you train a neural network to identify patterns inherent to fraudulent behavior. The system analyzes new purchase patterns and raises a red flag if it thinks a purchase may be fraudulent. Rules-based systems (like the one in Outlook) are also used to weed out common purchase situations that are most likely fraudulent.
Two weeks ago, my credit card company called me while I was in England. Within the same week I used the card to buy plane tickets in New York, subway cards in London, and several large purchases in Germany. A combination of neural-network pattern matching and rules-based purchase patterns contributed to the company calling to ensure the card wasn't stolen.
Bayesian networks are currently used to help determine if something is spam, but they're based on keyword tokens and heuristics surrounding the tokens (how close together they appear, where they appear, etc.), not on concepts. Bayesian networks, like neural networks, would do a terrific job if inputs were concept-based instead of keyword-based.
Putting It All Together
Combined with concept fingerprint technologies, neural networks can be trained to identify which concepts are likely spam. As new email arrives, the content abstraction system can send the concept to the neural network, then to the rules-based system. These can weed out unwanted email.
Obviously, spam is far from over. Keyword-based rules engines and heuristic models to identify spam aren't as effective as we'd hoped. We must use different technologies that identify spam based on message concept, not just its use of words. Using concept fingerprints to identify spam will help eradicate much more spam.
No matter what they call that little blue pill, they're still trying to sell it to you.
What are your thoughts? Let me know!
Until next time...
This Year's Premier Digital Marketing Event is #CZLSF
ClickZ Live San Francisco (Aug 11-14) brings together the industry's leading practitioners and marketing strategists to deliver 4 days of educational sessions and training workshops. From Data-Driven Marketing to Social, Mobile, Display, Search and Email, this year's comprehensive agenda will help you maximize your marketing efforts and ROI. Register today!
Jack Aaronson, CEO of The Aaronson Group and corporate lecturer, is a sought-after expert on enhanced user experiences, customer conversion, retention, and loyalty. If only a small percentage of people who arrive at your home page transact with your company (and even fewer return to transact again), Jack and his company can help. He also publishes a newsletter about multichannel marketing, personalization, user experience, and other related issues. He has keynoted most major marketing conferences around the world and regularly speaks at Shop.org and other major industry shows. You can learn more about Jack through his LinkedIn profile.
The Marketer's Guide to Customer Loyalty
Customer loyalty is imperative to success, but fostering and maintaining loyalty takes a lot of work. This guide is here to help marketers build, execute, and maintain a successful loyalty initiative.
The Multiplier Effect of Integrating Search & Social Advertising
Latest research reveals 68% higher revenue per conversion for marketers who integrate their search & social advertising. In addition to the research results, this whitepaper also outlines 5 strategies and 15 tactics you can use to better integrate your search and social campaigns.