The Butler Did It
A look at the process followed and the conclusions reached by one acknowledged leader in the analysis of customer data.
Observed customer data can be a valuable resource for making mission critical business decisions. Ask experts in the industry and they’ll tell you analyzing customer interactions with a Web site can yield great insights. Or you can just Ask Jeeves.
Ask Jeeves Inc., the Internet search company that operates the Ask.com Web site, Teoma.com, and Ask Jeeves Kids, has a reputation as an industry leader in utilizing customer data analysis as a decision making tool. The research and analysis team plays a significant role in decisions made across the entire organization. By tirelessly crunching the numbers, the team is able to predict the results of every proposed business decision in advance of implementation. In this column, we’ll look at one example of how Ask Jeeves staffers analyze user data and made decisions.
The research and analysis staff at Ask Jeeves recently completed a detailed customer analysis that played an integral role in optimizing the positioning of search results sets on the Ask Jeeves reply page. The goal? Maximizing user satisfaction and revenue.
“There is more science than art to our products at Ask Jeeves. We believe that analyzing the behavior of our users and making refinements based on that analysis is the best method to optimize the performance our products,” said Jim Lanzone, vice president of product management.
For those of you who haven’t used Ask Jeeves, it’s a search engine with a twist. Users can type in a search term as they would with a search engine (e.g. Tampa Bay Devil Rays) or ask a question in natural language (e.g. Where can I buy tickets for the Tampa Bay Devil Rays?). In either case, users are directed to a reply page containing a maximum of five search results sets. Results sets are either editor-selected, technology-driven or paid. It is important to note that the make-up of every reply page is determined by the users’ search terms or questions. Therefore, each reply page can display anywhere from one to a maximum of five results sets. Positioning of the results sets is critical to ensuring both user satisfaction and revenue generation.
It’s a tender balance to simultaneously provide the answers to users questions while also making the cash register ring. A change in the positioning of the results sets is no willy-nilly decision. Again, the objective of this analysis was to optimize the positioning of search results sets on the Ask Jeeves reply page to maximize user satisfaction and revenue.
The research and analysis team, led by Shane McGilloway, began by reviewing observed customer data mined from user logs to determine the results sets that are selected most frequently. Logic would suggest that the results sets that appear most frequently would also be the most frequently selected by users. McGilloway’s team found some surprises in the analysis:
| Results Set | Coverage | Share of Picks (All Users) |
| A | 81% | 25% |
| B | 54% | 23% |
| C | 92% | 45% |
| D | 31% | 6.0% |
| E | 32% | 1.0% |
Results Sets have been labeled A through E for confidentiality purposes.
Coverage is defined as the percentage of reply pages that contain a specific results set. Coverage of 81 percent for Results Set A means 81 percent of all reply pages contain Results Set A.
Share of Picks is defined as the percentage of users that select a result from within a specific results set. Share of Picks of 25 percent for Results Set A means 25 percent of users select a result from Results Set A.
Note the gap between Coverage and Share of Picks for Results Set A. Although Results Set A appears on 81 percent of all reply pages, users select a result from Results Set A just 25 percent of the time. Also note Results Set B was picked almost as frequently as Results Set A (23 percent vs. 25 percent), even though Results Set B appears on just 54 percent of all reply pages.
Furthermore, Results Sets A and B always appear above the fold. Logic would suggest appearing above the fold would increase the likelihood of a results set being selected. This further amplifies the gap between Coverage and Share of Picks for Results Set A. In essence, users are scrolling past Results Set A to access results sets featured below the fold. Conversely, this makes a strong statement about the popularity of Results Set C, which typically appears below the fold.
One other point that may come as a surprise is that Results Set C generates revenue while Results Set A is non-revenue generating. That’s right; users are selecting revenue generating results sets more frequently non-revenue generating results sets.
Once McGilloway had a view of how all users selected the results sets, he concentrated his efforts on how the most valuable users accessed the results sets. Like most businesses, Ask Jeeves relies on their most valuable users to generate a significant portion of their revenue. Therefore, analyzing the behavior of Jeeves’ most valuable users is as important as looking at the entire universe of users. Making any changes without first taking into account the impact on the most valuable users could have devastating results.
McGilloway identifies the most valuable users using a derivative of an RFM model, essentially an FM model. The methodology behind McGilloway’s FM model is tailored to Jeeves’ business model and therefore confidential.
“We determined that in our business we gain little value from the R variable and began focusing on the F and M variables exclusively,” said McGilloway. It is safe to say that the FM model has proven extremely effective for Jeeves’ research and analysis purposes.
The research and analysis team repeated the previous analysis, this time utilizing observed customer data exclusively from Jeeves’ most valuable users, as defined by the FM model.
| Results Set | Coverage | Share of Picks (All Users) | Share of Picks (Most Valuable Users) |
| A | 81% | 25% | 21% |
| B | 54% | 23% | 25% |
| C | 92% | 45% | 46% |
| D | 31% | 6% | 7% |
| E | 32% | 1% | 1% |
Note the gap between Coverage and Share of Picks for Results Set A is even more pronounced among the most valuable users. Also note that among most valuable users, Results Set B was picked more frequently than Results Set A (25 percent vs. 21 percent), even though Results Set B appears on reply pages much less frequently than Results Set A (54 percent vs. 81 percent).
The behavior exhibited by the most valuable users coincides with the behavior exhibited by all users. That’s good news, as it makes the recommendation phase relatively easy. When the behavior of most valuable users does not coincide with the behavior of all users, making recommendations becomes much more challenging.
Remember, the objective of McGilloway’s analysis was to optimize the positioning of search results sets on the Ask Jeeves reply page to maximize user satisfaction and revenue. The recommendations therefore are as follows:
- Move Results Set C above the fold, and move Results Set A below the fold. All users and most valuable users alike select Results Set C most frequently. The kicker is that it generates revenue. Moving Results Set C into the space currently occupied by Results Set A will increase both user satisfaction and revenue. This is a no-brainer.
- Increase Coverage for Results Set B — Results Set B has proven a popular selection among all users and most valuable users. Given increased coverage, it has the potential to increase in frequency of selection by users.
Ask Jeeves will be making refinements to its reply page in the coming weeks based on these recommendations. When these changes happen, you’ll know how they used customer data analysis to make the refinements.
