Trick Questions

Analysts are problem solvers. They are Sudoku workers. They are detective novel readers. They enjoy puzzles. In fact, they enjoy puzzles so much that it’s easy to distract them from their core job of providing organizational and customer insight.

Take a simple question:

“Can you tell me how many people retweeted this comment, didn’t click on the link, and used the special promotion code anyway?”

The typical marketing analyst immediately starts thinking about how to collect the data, how to join tweet stream info to click-throughs and shopping cart data, and what the dashboard should look like. They were tricked into doing too much work that’s going to have too little value at the cost of their real work.

Trick questions will trip you up every time. Here are three typical trick questions. See how well you fair. (Answers below.)

Trick Question No. 1

A disabled airplane falls from the sky and comes to rest directly on the point where Iran, Afghanistan, and Pakistan meet. Where do they bury the survivors?

Trick Question No. 2

You’re driving a bus up 7th Avenue in New York. At 27th Street, four students get on from the Fashion Institute of Technology. At 29th, three doctors get on from Doctors Without Borders and two get off at the very next stop for St. Jude’s Children’s Research. At 35th Street, six freshly groomed riders get on from the Eyebrow Threading Salon and three of the FIT students get off at 38th Street for FedEx Shipping Center. All of the remaining doctors get off at 40th Street to go to the Garment Center Congregation. What’s the name of the bus driver?

Trick Question No. 3

You’re out bear hunting with some friends and one morning you leave your camp site and head due south for five miles. You then turn due west, spot a bear, and shoot it. You haul your prize five miles back to your camp site and call it a day. What color is the bear?

The first question is an example of misdirection. You do not bury survivors. They walk away. I feel that conversion rate is another example of misdirection.

Don’t get me wrong – conversion rates are very important to monitor and master all other things being equal. But all other things are never equal. If you just spent a metric boatload of money on a display ad campaign, then your traffic is going to go up and your conversion is going to drop precipitously and the unwary conversion expert will be left wondering what happened to the shopping cart navigation.

The second question is an example of analyzing the available data before knowing what you are solving for. The name of the bus driver is deeply buried at the very beginning of the question by all that data about the stops and the passengers. That, and the fact that 7th Avenue in New York is one-way and runs north-to-south. Tricked you!

The point of this one is to know the business problem that needs work before jumping into a vat of data just for the fun of it. Yes, you can API all of the tweets you like, but when all is said and done, what is the business value of knowing that the world’s record is 7,196 tweets per second?

Don’t get caught up in the thrill of analyzing the data at the expense of solving real problems.

These first two questions are best served verbally. Hearing “Where did they bury the survivors?” is very different from reading it. One glance back at the bus driver question and you are not amused. The answer is obvious. Hearing the question is a different kettle of fish.

But the third question about the bear is not so easy. Curiously, it actually does contain all the information you need to determine that the bear is white. You just need to use a little lateral thinking and some high school geometry.

There is only one place on Earth where you can head due south for five miles, due west for five miles, and then walk five miles back to camp. An equilateral triangle has three equal length legs and equal angles of 60 degrees. That precludes heading due south and due west…except at the North Pole, where legs of equal length and angles of equal degree prevail. Ergo, your bear is white.

In a recent post on online behavior called Web Analytics Tools: Question Generation Machines, Yahoo Web Analytics Account Manager Emer Kirrane wrote that the most important question in web analytics is “Why?”

“Web Analytics is not about the tool (though not every vendor may concede this). It is about collecting, slicing, manipulating and attempting to understand data, and then using your findings to improve your website, your customers’ experience, your revenue and so on.”

Sadly, those who do not understand marketing analytics continue to “…use statistics as a drunken man uses lamp-posts – for support rather than for illumination.” (Andrew Lang) Those analysts who feel too junior to the task, succumb to the whiles of inquisitors and never come into their own as insight generators.

For them, the solution is in the response, “I would be delighted to help you accomplish your goals. But first, I need to know what your goals are.”

“Why?” it turns out, is the inverse of a trick question.

Jim is off today. This column was originally published on July 21, 2011 on ClickZ.