The Survey Says...

  |  January 2, 2004   |  Comments

Navigating online survey pitfalls.

We recently published a customer-focused shopping survey. I'd like to share some of what we've learned about creating surveys that provide accurate and meaningful information.

Will Do or Have Done?

If your toilet were broken and you needed a plumber right away, where would you look for one? Most would answer, "the yellow pages." Recently, marketing guru Roy H. Williams conducted a national study for a client who was certain everyone selected a plumber based on who she found in the yellow pages.

Williams began the survey with the question: "Have you called a plumber in the last 12 months?'' If the respondent answered no, the survey was over. If she answered yes, Williams asked, "Who did you call?" If the respondent hesitated for more than four seconds, the survey was over. If she quickly remembered the name, Williams asked, "Why did you choose that plumber?" then, "Where did you get that number?"

If the respondent's answer didn't mention the yellow pages, Williams asked a final question: "Did you, at any time, look in the phonebook?"

Only 17.8 percent turned to the phonebook to find a plumber.

What people do is often quite different from what they think they'll do. As I've said frequently, people rationalize decisions with facts, but they make decisions based on feelings. Emotion is central to any decision-making process.

Ask someone if he'll take your survey. He'll immediately disconnect from the emotional side of his brain, the area where decisions are made. He'll tap into the logical side of his brain, the side that gives considered answers. The logical side knows nothing of emotion, and its answers aren't reliable decision predictors.

My colleague Jim Novo tells a story of when he conducted intent-to-buy research for the Home Shopping Network:

On average, there was an inverse relationship between stated intent and action. The customers who stated they were highly likely to buy were the least likely to actually purchase again, and vice versa. You simply can't blindly trust what customers tell you without looking at their actual behavior.

The only dependable survey responses are those based on questions that ask what's already been done. You're far more likely to receive meaningful replies. You know for certain the answers are based on what respondents have actually done.

Don't ask what will happen, ask what has happened.

Inaccuracies of Multiple Choice

Multiple-choice questions plague research firms. Multiple choice may seem like the easiest and most cost-effective way to conduct a survey. Problem is, people tend to choose the answer that sounds best, not the one that accurately reflects their behavior or intentions.

Encourage respondents to describe their experiences without offering a list of choices. It's preferable to ask fewer questions that elicit dependable answers than to prejudice the survey with multiple-choice questions.

Wording Questions

How questions are worded can influence answers. Take this old story for example: Two men ask their rabbi whether it's permissible to smoke while praying. The first asks, "Is it permissible to smoke while praying?" "No," answers the rabbi, "it is not permissible to do anything but concentrate while praying." The second man asks, "Is it permissible to pray while smoking?" "Yes," answers the rabbi, "it is always permissible to pray."

Here's another example. The Associated Press reported witnesses who were asked, "How fast were the cars going when they smashed into each other?" gave a much higher speed than those who were asked, "How fast were the cars going when they made contact?"

Give your questions to an outsider. Ask how each question might predispose or shape the answer.

How Large a Sample?

Sample a group large enough to give results with a 95 percent level of confidence or accuracy. Ninety-five percent confidence means if the survey ran 100 times, results would be within 5 percent of the first survey's results 95 times out of 100.

The sample group needn't be large. With fewer than 1,500 qualified respondents, Gallup can report with 95 percent confidence which president American voters will elect. With 384 qualified respondents, you can measure 1 million people with 95 percent confidence. To understand what qualifies a respondent, find out what a statistician would consider a random sample of your audience.

Subject to Interpretation

A survey's value is based on data reliability, not data interpretation. To ensure you understand a survey's results, be suspicious of interpretation. Look at the data to see if you'd arrive at the same conclusions. Be careful to distinguish between factual results and interpretation of results. Interpretation can be filtered through bias.

To quote Frank Lloyd Wright, "The truth is more important than the facts." Just because it's easy to do online surveys with the multitude of tools available doesn't mean they should be taken lightly.


Bryan Eisenberg

Bryan Eisenberg is co-founder and chief marketing officer (CMO) of IdealSpot. He is co-author of the Wall Street Journal, Amazon, BusinessWeek, and New York Times best-selling books Call to Action, Waiting For Your Cat to Bark?, and Always Be Testing, and Buyer Legends. Bryan is a keynote speaker and has keynoted conferences globally such as Gultaggen,, Direct Marketing Association, MarketingSherpa, Econsultancy, Webcom, the Canadian Marketing Association, and others for the past 10 years. Bryan was named a winner of the Marketing Edge's Rising Stars Awards, recognized by eConsultancy members as one of the top 10 User Experience Gurus, selected as one of the inaugural iMedia Top 25 Marketers, and has been recognized as most influential in PPC, Social Selling, OmniChannel Retail. Bryan serves as an advisory board member of several venture capital backed companies such as Sightly, UserTesting, Monetate, ChatID, Nomi, and BazaarVoice. He works with his co-author and brother Jeffrey Eisenberg. You can find them at

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!



Featured White Papers

2015 Holiday Email Guide

2015 Holiday Email Guide
The holidays are just around the corner. Download this whitepaper to find out how to create successful holiday email campaigns that drive engagement and revenue.

Three Ways to Make Your Big Data More Valuable

Three Ways to Make Your Big Data More Valuable
Big data holds a lot of promise for marketers, but are marketers ready to make the most of it to drive better business decisions and improve ROI? This study looks at the hidden challenges modern marketers face when trying to put big data to use.