Text Mining: MI5's (and Your) Secret Weapon
If your customers are talking to you, shouldn't you listen to what they're saying? Text mining can help.
If your customers are talking to you, shouldn't you listen to what they're saying? Text mining can help.
Years ago, I received my first large data file. It contained around 200,000 records. All had the names, addresses, and purchase histories of a retail catalog company’s customers. Soon after I received the file, a question arose regarding how the company was marketing some of its products.
Being young and naive, I thought it would be no problem to use that data file to call appropriate customers and ask them questions about the company’s marketing. After all, the file contained customers’ phone numbers. I figured, “Who better to answer the questions?”
The customers were surprised by the calls but happy to answer my questions. The information we collected was helpful to the company. Fortunately, I didn’t get in trouble for using the data in this manner.
Consumer Feedback
Customers are more than happy to give you feedback regarding your products and services. Some companies’ customers actually submit unsolicited feedback. The companies could use the data to improve their products and services, yet often they don’t.
Consumer packaged good, hotel, and airline companies are examples of organizations that receive unsolicited feedback all the time, usually in the form of complaint letters. Airlines receive thousands upon thousands of letters each year. What do they do with them?
Almost nothing.
Do they really take the time to read all the letters they receive? Of course not. Is there useful information contained somewhere in all of those letters? Of course there is!
Text Mining
It’s probably not cost-effective to individually read and monitor all consumer complaint letters at companies that receive very large quantities. Fortunately, that’s not necessary.
Consumer data can be analyzed with software known as text mining applications. The leader in this field is Autonomy. The algorithms underlying the software were first put into use by MI5, the U.K.’s Security Service, in defense applications. Today, they’re available to any company interested in using them to mine available text data.
SPSS offers another text mining program. SPSS focuses on practical applications, so the software has a specialized module for analyzing text captured in surveys. SAS also offers a text mining product.
How It Works
Technical details of how text mining works involve mathematical discussions of things such as vectors, cosines between vectors, and cluster analysis. Thankfully, all these things are programmed into the software, allowing users to focus on the information generated from the analysis.
Here’s a very simplified description: Text mining involves counting the number of times specific words appear in documents and comparing documents against one another in terms of word count. This helps group similar documents together. In an analysis of airline complaint letters, resultant groups might include “weather incidents,” “baggage handling,” and “flight attendant compliments.”
Initial and Ongoing Use
The initial application of this kind of software helps a company learn what natural groupings of complaint letters exist in its data. By themselves, these results could be very useful.
On an ongoing basis, this kind of analysis could keep track of the quantity of letters received in each of those categories and determine when a new category appears (e.g., “leg room problems”). If the company applied it in a more in-depth manner, it could monitor the number of letters regarding a specific product or problem (e.g., “flight 435 out of MCI airport”).
Works for Them; Works for You
The MI5’s original techniques monitored information from many organizations and determined if “chatter” in certain categories was increasing. The same techniques can be used to determine if complaints in certain categories are on the rise.
Many companies have large data repositories they could use to improve their products and services, whether with an initial text-mining analysis or an ongoing analysis. If your customers are talking to you, shouldn’t you listen to what they’re saying?