The Survey Says...

  |  January 2, 2004   |  Comments

Navigating online survey pitfalls.

We recently published a customer-focused shopping survey. I'd like to share some of what we've learned about creating surveys that provide accurate and meaningful information.

Will Do or Have Done?

If your toilet were broken and you needed a plumber right away, where would you look for one? Most would answer, "the yellow pages." Recently, marketing guru Roy H. Williams conducted a national study for a client who was certain everyone selected a plumber based on who she found in the yellow pages.

Williams began the survey with the question: "Have you called a plumber in the last 12 months?'' If the respondent answered no, the survey was over. If she answered yes, Williams asked, "Who did you call?" If the respondent hesitated for more than four seconds, the survey was over. If she quickly remembered the name, Williams asked, "Why did you choose that plumber?" then, "Where did you get that number?"

If the respondent's answer didn't mention the yellow pages, Williams asked a final question: "Did you, at any time, look in the phonebook?"

Only 17.8 percent turned to the phonebook to find a plumber.

What people do is often quite different from what they think they'll do. As I've said frequently, people rationalize decisions with facts, but they make decisions based on feelings. Emotion is central to any decision-making process.

Ask someone if he'll take your survey. He'll immediately disconnect from the emotional side of his brain, the area where decisions are made. He'll tap into the logical side of his brain, the side that gives considered answers. The logical side knows nothing of emotion, and its answers aren't reliable decision predictors.

My colleague Jim Novo tells a story of when he conducted intent-to-buy research for the Home Shopping Network:

On average, there was an inverse relationship between stated intent and action. The customers who stated they were highly likely to buy were the least likely to actually purchase again, and vice versa. You simply can't blindly trust what customers tell you without looking at their actual behavior.

The only dependable survey responses are those based on questions that ask what's already been done. You're far more likely to receive meaningful replies. You know for certain the answers are based on what respondents have actually done.

Don't ask what will happen, ask what has happened.

Inaccuracies of Multiple Choice

Multiple-choice questions plague research firms. Multiple choice may seem like the easiest and most cost-effective way to conduct a survey. Problem is, people tend to choose the answer that sounds best, not the one that accurately reflects their behavior or intentions.

Encourage respondents to describe their experiences without offering a list of choices. It's preferable to ask fewer questions that elicit dependable answers than to prejudice the survey with multiple-choice questions.

Wording Questions

How questions are worded can influence answers. Take this old story for example: Two men ask their rabbi whether it's permissible to smoke while praying. The first asks, "Is it permissible to smoke while praying?" "No," answers the rabbi, "it is not permissible to do anything but concentrate while praying." The second man asks, "Is it permissible to pray while smoking?" "Yes," answers the rabbi, "it is always permissible to pray."

Here's another example. The Associated Press reported witnesses who were asked, "How fast were the cars going when they smashed into each other?" gave a much higher speed than those who were asked, "How fast were the cars going when they made contact?"

Give your questions to an outsider. Ask how each question might predispose or shape the answer.

How Large a Sample?

Sample a group large enough to give results with a 95 percent level of confidence or accuracy. Ninety-five percent confidence means if the survey ran 100 times, results would be within 5 percent of the first survey's results 95 times out of 100.

The sample group needn't be large. With fewer than 1,500 qualified respondents, Gallup can report with 95 percent confidence which president American voters will elect. With 384 qualified respondents, you can measure 1 million people with 95 percent confidence. To understand what qualifies a respondent, find out what a statistician would consider a random sample of your audience.

Subject to Interpretation

A survey's value is based on data reliability, not data interpretation. To ensure you understand a survey's results, be suspicious of interpretation. Look at the data to see if you'd arrive at the same conclusions. Be careful to distinguish between factual results and interpretation of results. Interpretation can be filtered through bias.

To quote Frank Lloyd Wright, "The truth is more important than the facts." Just because it's easy to do online surveys with the multitude of tools available doesn't mean they should be taken lightly.

ClickZ Live Chicago Join the Industry's Leading eCommerce & Direct Marketing Experts in Chicago
ClickZ Live Chicago (Nov 3-6) will deliver over 50 sessions across 4 days and 10 individual tracks, including Data-Driven Marketing, Social, Mobile, Display, Search and Email. Check out the full agenda and register by Friday, Oct 3 to take advantage of Early Bird Rates!

ABOUT THE AUTHOR

Bryan Eisenberg

Bryan Eisenberg is coauthor of the Wall Street Journal, Amazon, BusinessWeek, and New York Times bestselling books "Call to Action," "Waiting For Your Cat to Bark?," and "Always Be Testing." Bryan is a professional marketing speaker and has keynoted conferences globally such as SES, Shop.org, Direct Marketing Association, MarketingSherpa, Econsultancy, Webcom, SEM Konferansen Norway, the Canadian Marketing Association, and others. In 2010, Bryan was named a winner of the Direct Marketing Educational Foundation's Rising Stars Awards, which recognizes the most talented professionals 40 years of age or younger in the field of direct/interactive marketing. He is also cofounder and chairman emeritus of the Web Analytics Association. Bryan serves as an advisory board member of SES Conference & Expo, the eMetrics Marketing Optimization Summit, and several venture capital backed companies. He works with his coauthor and brother Jeffrey Eisenberg. You can find them at BryanEisenberg.com.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!

COMMENTS

UPCOMING EVENTS

Featured White Papers

IBM: Social Analytics - The Science Behind Social Media Marketing

IBM Social Analytics: The Science Behind Social Media Marketing
80% of internet users say they prefer to connect with brands via Facebook. 65% of social media users say they use it to learn more about brands, products and services. Learn about how to find more about customers' attitudes, preferences and buying habits from what they say on social media channels.

An Introduction to Marketing Attribution: Selecting the Right Model for Search, Display & Social Advertising

An Introduction to Marketing Attribution: Selecting the Right Model for Search, Display & Social Advertising
If you're considering implementing a marketing attribution model to measure and optimize your programs, this paper is a great introduction. It also includes real-life tips from marketers who have successfully implemented attribution in their organizations.

WEBINARS

Jobs

    • Web Writer
      Web Writer (Money Map Press) - BaltimoreDo you have a passion for the markets and investing, and writing? Do you want to spend your days providing...
    • Web Production Specialist
      Web Production Specialist (Money Map Press) - BaltimoreMoney Map Press is looking for a self-starter to perform and oversee the production of daily...
    • Internet Marketing Campaign Manager
      Internet Marketing Campaign Manager (Straight North, LLC) - Downers GroveWe are looking for a talented Internet Marketing Campaign Manager...