The Survey Says...

  |  January 2, 2004   |  Comments

Navigating online survey pitfalls.

We recently published a customer-focused shopping survey. I'd like to share some of what we've learned about creating surveys that provide accurate and meaningful information.

Will Do or Have Done?

If your toilet were broken and you needed a plumber right away, where would you look for one? Most would answer, "the yellow pages." Recently, marketing guru Roy H. Williams conducted a national study for a client who was certain everyone selected a plumber based on who she found in the yellow pages.

Williams began the survey with the question: "Have you called a plumber in the last 12 months?'' If the respondent answered no, the survey was over. If she answered yes, Williams asked, "Who did you call?" If the respondent hesitated for more than four seconds, the survey was over. If she quickly remembered the name, Williams asked, "Why did you choose that plumber?" then, "Where did you get that number?"

If the respondent's answer didn't mention the yellow pages, Williams asked a final question: "Did you, at any time, look in the phonebook?"

Only 17.8 percent turned to the phonebook to find a plumber.

What people do is often quite different from what they think they'll do. As I've said frequently, people rationalize decisions with facts, but they make decisions based on feelings. Emotion is central to any decision-making process.

Ask someone if he'll take your survey. He'll immediately disconnect from the emotional side of his brain, the area where decisions are made. He'll tap into the logical side of his brain, the side that gives considered answers. The logical side knows nothing of emotion, and its answers aren't reliable decision predictors.

My colleague Jim Novo tells a story of when he conducted intent-to-buy research for the Home Shopping Network:

On average, there was an inverse relationship between stated intent and action. The customers who stated they were highly likely to buy were the least likely to actually purchase again, and vice versa. You simply can't blindly trust what customers tell you without looking at their actual behavior.

The only dependable survey responses are those based on questions that ask what's already been done. You're far more likely to receive meaningful replies. You know for certain the answers are based on what respondents have actually done.

Don't ask what will happen, ask what has happened.

Inaccuracies of Multiple Choice

Multiple-choice questions plague research firms. Multiple choice may seem like the easiest and most cost-effective way to conduct a survey. Problem is, people tend to choose the answer that sounds best, not the one that accurately reflects their behavior or intentions.

Encourage respondents to describe their experiences without offering a list of choices. It's preferable to ask fewer questions that elicit dependable answers than to prejudice the survey with multiple-choice questions.

Wording Questions

How questions are worded can influence answers. Take this old story for example: Two men ask their rabbi whether it's permissible to smoke while praying. The first asks, "Is it permissible to smoke while praying?" "No," answers the rabbi, "it is not permissible to do anything but concentrate while praying." The second man asks, "Is it permissible to pray while smoking?" "Yes," answers the rabbi, "it is always permissible to pray."

Here's another example. The Associated Press reported witnesses who were asked, "How fast were the cars going when they smashed into each other?" gave a much higher speed than those who were asked, "How fast were the cars going when they made contact?"

Give your questions to an outsider. Ask how each question might predispose or shape the answer.

How Large a Sample?

Sample a group large enough to give results with a 95 percent level of confidence or accuracy. Ninety-five percent confidence means if the survey ran 100 times, results would be within 5 percent of the first survey's results 95 times out of 100.

The sample group needn't be large. With fewer than 1,500 qualified respondents, Gallup can report with 95 percent confidence which president American voters will elect. With 384 qualified respondents, you can measure 1 million people with 95 percent confidence. To understand what qualifies a respondent, find out what a statistician would consider a random sample of your audience.

Subject to Interpretation

A survey's value is based on data reliability, not data interpretation. To ensure you understand a survey's results, be suspicious of interpretation. Look at the data to see if you'd arrive at the same conclusions. Be careful to distinguish between factual results and interpretation of results. Interpretation can be filtered through bias.

To quote Frank Lloyd Wright, "The truth is more important than the facts." Just because it's easy to do online surveys with the multitude of tools available doesn't mean they should be taken lightly.

ClickZ Live San Francisco This Year's Premier Digital Marketing Event is #CZLSF
ClickZ Live San Francisco (Aug 11-14) brings together the industry's leading practitioners and marketing strategists to deliver 4 days of educational sessions and training workshops. From Data-Driven Marketing to Social, Mobile, Display, Search and Email, this year's comprehensive agenda will help you maximize your marketing efforts and ROI. Register today!


Bryan Eisenberg

Bryan Eisenberg is coauthor of the Wall Street Journal, Amazon, BusinessWeek, and New York Times bestselling books "Call to Action," "Waiting For Your Cat to Bark?," and "Always Be Testing." Bryan is a professional marketing speaker and has keynoted conferences globally such as SES,, Direct Marketing Association, MarketingSherpa, Econsultancy, Webcom, SEM Konferansen Norway, the Canadian Marketing Association, and others. In 2010, Bryan was named a winner of the Direct Marketing Educational Foundation's Rising Stars Awards, which recognizes the most talented professionals 40 years of age or younger in the field of direct/interactive marketing. He is also cofounder and chairman emeritus of the Web Analytics Association. Bryan serves as an advisory board member of SES Conference & Expo, the eMetrics Marketing Optimization Summit, and several venture capital backed companies. He works with his coauthor and brother Jeffrey Eisenberg. You can find them at

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!



Featured White Papers

BigDoor: The Marketers Guide to Customer Loyalty

The Marketer's Guide to Customer Loyalty
Customer loyalty is imperative to success, but fostering and maintaining loyalty takes a lot of work. This guide is here to help marketers build, execute, and maintain a successful loyalty initiative.

Marin Software: The Multiplier Effect of Integrating Search & Social Advertising

The Multiplier Effect of Integrating Search & Social Advertising
Latest research reveals 68% higher revenue per conversion for marketers who integrate their search & social advertising. In addition to the research results, this whitepaper also outlines 5 strategies and 15 tactics you can use to better integrate your search and social campaigns.



    • Interactive Product Manager
      Interactive Product Manager (Western Governors University) - Salt Lake CityWestern Governors University, one of the 20 largest universities...
    • SEO Senior Analyst
      SEO Senior Analyst (University of Phoenix (Apollo Education Group)) - San FranciscoSEO Senior Analyst   Position Summary...
    • SEM & Biddable Media Manager
      SEM & Biddable Media Manager (Kepler Group LLC) - New YorkAs an Optimization & Innovation Manager at Kepler Group, you will be on the bleeding...