Text Mining: MI5's (and Your) Secret Weapon

  |  February 22, 2005   |  Comments

If your customers are talking to you, shouldn't you listen to what they're saying? Text mining can help.

Years ago, I received my first large data file. It contained around 200,000 records. All had the names, addresses, and purchase histories of a retail catalog company's customers. Soon after I received the file, a question arose regarding how the company was marketing some of its products.

Being young and naive, I thought it would be no problem to use that data file to call appropriate customers and ask them questions about the company's marketing. After all, the file contained customers' phone numbers. I figured, "Who better to answer the questions?"

The customers were surprised by the calls but happy to answer my questions. The information we collected was helpful to the company. Fortunately, I didn't get in trouble for using the data in this manner.

Consumer Feedback

Customers are more than happy to give you feedback regarding your products and services. Some companies' customers actually submit unsolicited feedback. The companies could use the data to improve their products and services, yet often they don't.

Consumer packaged good, hotel, and airline companies are examples of organizations that receive unsolicited feedback all the time, usually in the form of complaint letters. Airlines receive thousands upon thousands of letters each year. What do they do with them?

Almost nothing.

Do they really take the time to read all the letters they receive? Of course not. Is there useful information contained somewhere in all of those letters? Of course there is!

Text Mining

It's probably not cost-effective to individually read and monitor all consumer complaint letters at companies that receive very large quantities. Fortunately, that's not necessary.

Consumer data can be analyzed with software known as text mining applications. The leader in this field is Autonomy. The algorithms underlying the software were first put into use by MI5, the U.K.'s Security Service, in defense applications. Today, they're available to any company interested in using them to mine available text data.

SPSS offers another text mining program. SPSS focuses on practical applications, so the software has a specialized module for analyzing text captured in surveys. SAS also offers a text mining product.

How It Works

Technical details of how text mining works involve mathematical discussions of things such as vectors, cosines between vectors, and cluster analysis. Thankfully, all these things are programmed into the software, allowing users to focus on the information generated from the analysis.

Here's a very simplified description: Text mining involves counting the number of times specific words appear in documents and comparing documents against one another in terms of word count. This helps group similar documents together. In an analysis of airline complaint letters, resultant groups might include "weather incidents," "baggage handling," and "flight attendant compliments."

Initial and Ongoing Use

The initial application of this kind of software helps a company learn what natural groupings of complaint letters exist in its data. By themselves, these results could be very useful.

On an ongoing basis, this kind of analysis could keep track of the quantity of letters received in each of those categories and determine when a new category appears (e.g., "leg room problems"). If the company applied it in a more in-depth manner, it could monitor the number of letters regarding a specific product or problem (e.g., "flight 435 out of MCI airport").

Works for Them; Works for You

The MI5's original techniques monitored information from many organizations and determined if "chatter" in certain categories was increasing. The same techniques can be used to determine if complaints in certain categories are on the rise.

Many companies have large data repositories they could use to improve their products and services, whether with an initial text-mining analysis or an ongoing analysis. If your customers are talking to you, shouldn't you listen to what they're saying?

ClickZ Live Toronto On the heels of a fantastic event in New York City, ClickZ Live is taking the fun and learning to Toronto, June 23-25. With over 15 years' experience delivering industry-leading events, ClickZ Live offers an action-packed, educationally-focused agenda covering all aspects of digital marketing. Register today!

ClickZ Live San Francisco Want to learn more? Join us at ClickZ Live San Francisco, Aug 10-12!
Educating marketers for over 15 years, ClickZ Live brings together industry thought leaders from the largest brands and agencies to deliver the most advanced, educational digital marketing agenda. Register today and save $500!

ABOUT THE AUTHOR

Brian Teasley Brian Teasley is the leader of Teasley, a consultancy that helpsadvertisers, marketers and advertising agencies use data and analysis toimprove their marketing campaigns. Brian has over 14 years experience inengineering and marketing, and has worked for numerous Fortune 100companies. Brian also teaches a marketing course at New York University. Heholds a M.S. degree in Applied Statistics from Iowa State University and aBA in Mathematics and a BA in Mathematics and Statistics from St. OlafCollege.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!

COMMENTS

UPCOMING EVENTS

Featured White Papers

Gartner Magic Quadrant for Digital Commerce

Gartner Magic Quadrant for Digital Commerce
This Magic Quadrant examines leading digital commerce platforms that enable organizations to build digital commerce sites. These commerce platforms facilitate purchasing transactions over the Web, and support the creation and continuing development of an online relationship with a consumer.

Paid Search in the Mobile Era

Paid Search in the Mobile Era
Google reports that paid search ads are currently driving 40+ million calls per month. Cost per click is increasing, paid search budgets are growing, and mobile continues to dominate. It's time to revamp old search strategies, reimagine stale best practices, and add new layers data to your analytics.

WEBINARS

Resources

Jobs

    • GREAT Campaign Project Coordinator
      GREAT Campaign Project Coordinator (British Consulate-General, New York) - New YorkThe GREAT Britain Campaign is seeking an energetic and creative...
    • Paid Search Senior Account Manager
      Paid Search Senior Account Manager (Hanapin Marketing) - BloomingtonHanapin Marketing is hiring a strategic Paid Search Senior Account Manager...
    • Paid Search Account Manager
      Paid Search Account Manager (Hanapin Marketing) - BloomingtonHanapin Marketing is hiring an experienced Paid Search Account Manager to...