Text Mining: MI5's (and Your) Secret Weapon

  |  February 22, 2005   |  Comments

If your customers are talking to you, shouldn't you listen to what they're saying? Text mining can help.

Years ago, I received my first large data file. It contained around 200,000 records. All had the names, addresses, and purchase histories of a retail catalog company's customers. Soon after I received the file, a question arose regarding how the company was marketing some of its products.

Being young and naive, I thought it would be no problem to use that data file to call appropriate customers and ask them questions about the company's marketing. After all, the file contained customers' phone numbers. I figured, "Who better to answer the questions?"

The customers were surprised by the calls but happy to answer my questions. The information we collected was helpful to the company. Fortunately, I didn't get in trouble for using the data in this manner.

Consumer Feedback

Customers are more than happy to give you feedback regarding your products and services. Some companies' customers actually submit unsolicited feedback. The companies could use the data to improve their products and services, yet often they don't.

Consumer packaged good, hotel, and airline companies are examples of organizations that receive unsolicited feedback all the time, usually in the form of complaint letters. Airlines receive thousands upon thousands of letters each year. What do they do with them?

Almost nothing.

Do they really take the time to read all the letters they receive? Of course not. Is there useful information contained somewhere in all of those letters? Of course there is!

Text Mining

It's probably not cost-effective to individually read and monitor all consumer complaint letters at companies that receive very large quantities. Fortunately, that's not necessary.

Consumer data can be analyzed with software known as text mining applications. The leader in this field is Autonomy. The algorithms underlying the software were first put into use by MI5, the U.K.'s Security Service, in defense applications. Today, they're available to any company interested in using them to mine available text data.

SPSS offers another text mining program. SPSS focuses on practical applications, so the software has a specialized module for analyzing text captured in surveys. SAS also offers a text mining product.

How It Works

Technical details of how text mining works involve mathematical discussions of things such as vectors, cosines between vectors, and cluster analysis. Thankfully, all these things are programmed into the software, allowing users to focus on the information generated from the analysis.

Here's a very simplified description: Text mining involves counting the number of times specific words appear in documents and comparing documents against one another in terms of word count. This helps group similar documents together. In an analysis of airline complaint letters, resultant groups might include "weather incidents," "baggage handling," and "flight attendant compliments."

Initial and Ongoing Use

The initial application of this kind of software helps a company learn what natural groupings of complaint letters exist in its data. By themselves, these results could be very useful.

On an ongoing basis, this kind of analysis could keep track of the quantity of letters received in each of those categories and determine when a new category appears (e.g., "leg room problems"). If the company applied it in a more in-depth manner, it could monitor the number of letters regarding a specific product or problem (e.g., "flight 435 out of MCI airport").

Works for Them; Works for You

The MI5's original techniques monitored information from many organizations and determined if "chatter" in certain categories was increasing. The same techniques can be used to determine if complaints in certain categories are on the rise.

Many companies have large data repositories they could use to improve their products and services, whether with an initial text-mining analysis or an ongoing analysis. If your customers are talking to you, shouldn't you listen to what they're saying?

ClickZ Live Chicago Join the Industry's Leading eCommerce & Direct Marketing Experts in Chicago
ClickZ Live Chicago (Nov 3-6) will deliver over 50 sessions across 4 days and 10 individual tracks, including Data-Driven Marketing, Social, Mobile, Display, Search and Email. Check out the full agenda and register by Friday, Sept 5 to take advantage of Super Saver Rates!

ABOUT THE AUTHOR

Brian Teasley Brian Teasley is the leader of Teasley, a consultancy that helpsadvertisers, marketers and advertising agencies use data and analysis toimprove their marketing campaigns. Brian has over 14 years experience inengineering and marketing, and has worked for numerous Fortune 100companies. Brian also teaches a marketing course at New York University. Heholds a M.S. degree in Applied Statistics from Iowa State University and aBA in Mathematics and a BA in Mathematics and Statistics from St. OlafCollege.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!

COMMENTS

UPCOMING EVENTS

Featured White Papers

IBM: Social Analytics - The Science Behind Social Media Marketing

IBM Social Analytics: The Science Behind Social Media Marketing
80% of internet users say they prefer to connect with brands via Facebook. 65% of social media users say they use it to learn more about brands, products and services. Learn about how to find more about customers' attitudes, preferences and buying habits from what they say on social media channels.

Marin Software: The Multiplier Effect of Integrating Search & Social Advertising

The Multiplier Effect of Integrating Search & Social Advertising
Latest research reveals 68% higher revenue per conversion for marketers who integrate their search & social advertising. In addition to the research results, this whitepaper also outlines 5 strategies and 15 tactics you can use to better integrate your search and social campaigns.

Resources

Jobs

    • Digital Marketing Analyst
      Digital Marketing Analyst (GovLoop) - Washington D.C.Are you passionate about audience acquisition? Love effective copy and amazingly effective...
    • Product Specialist
      Product Specialist (Agora Inc. ) - BaltimoreDescription: The Product Specialist is hyper-focused on the customer experience and ensures that our...
    • Partnerships Senior Coordinator
      Partnerships Senior Coordinator (Zappos.com, Inc.) - Las VegasZappos IP, Inc. is looking for a Partnerships Senior Coordinator! Why join us? Our...