Navigating online survey pitfalls.
We recently published a customer-focused shopping survey. I'd like to share some of what we've learned about creating surveys that provide accurate and meaningful information.
Will Do or Have Done?
If your toilet were broken and you needed a plumber right away, where would you look for one? Most would answer, "the yellow pages." Recently, marketing guru Roy H. Williams conducted a national study for a client who was certain everyone selected a plumber based on who she found in the yellow pages.
Williams began the survey with the question: "Have you called a plumber in the last 12 months?'' If the respondent answered no, the survey was over. If she answered yes, Williams asked, "Who did you call?" If the respondent hesitated for more than four seconds, the survey was over. If she quickly remembered the name, Williams asked, "Why did you choose that plumber?" then, "Where did you get that number?"
If the respondent's answer didn't mention the yellow pages, Williams asked a final question: "Did you, at any time, look in the phonebook?"
Only 17.8 percent turned to the phonebook to find a plumber.
What people do is often quite different from what they think they'll do. As I've said frequently, people rationalize decisions with facts, but they make decisions based on feelings. Emotion is central to any decision-making process.
Ask someone if he'll take your survey. He'll immediately disconnect from the emotional side of his brain, the area where decisions are made. He'll tap into the logical side of his brain, the side that gives considered answers. The logical side knows nothing of emotion, and its answers aren't reliable decision predictors.
My colleague Jim Novo tells a story of when he conducted intent-to-buy research for the Home Shopping Network:On average, there was an inverse relationship between stated intent and action. The customers who stated they were highly likely to buy were the least likely to actually purchase again, and vice versa. You simply can't blindly trust what customers tell you without looking at their actual behavior.
The only dependable survey responses are those based on questions that ask what's already been done. You're far more likely to receive meaningful replies. You know for certain the answers are based on what respondents have actually done.
Don't ask what will happen, ask what has happened.
Inaccuracies of Multiple Choice
Multiple-choice questions plague research firms. Multiple choice may seem like the easiest and most cost-effective way to conduct a survey. Problem is, people tend to choose the answer that sounds best, not the one that accurately reflects their behavior or intentions.
Encourage respondents to describe their experiences without offering a list of choices. It's preferable to ask fewer questions that elicit dependable answers than to prejudice the survey with multiple-choice questions.
How questions are worded can influence answers. Take this old story for example: Two men ask their rabbi whether it's permissible to smoke while praying. The first asks, "Is it permissible to smoke while praying?" "No," answers the rabbi, "it is not permissible to do anything but concentrate while praying." The second man asks, "Is it permissible to pray while smoking?" "Yes," answers the rabbi, "it is always permissible to pray."
Here's another example. The Associated Press reported witnesses who were asked, "How fast were the cars going when they smashed into each other?" gave a much higher speed than those who were asked, "How fast were the cars going when they made contact?"
Give your questions to an outsider. Ask how each question might predispose or shape the answer.
How Large a Sample?
Sample a group large enough to give results with a 95 percent level of confidence or accuracy. Ninety-five percent confidence means if the survey ran 100 times, results would be within 5 percent of the first survey's results 95 times out of 100.
The sample group needn't be large. With fewer than 1,500 qualified respondents, Gallup can report with 95 percent confidence which president American voters will elect. With 384 qualified respondents, you can measure 1 million people with 95 percent confidence. To understand what qualifies a respondent, find out what a statistician would consider a random sample of your audience.
Subject to Interpretation
A survey's value is based on data reliability, not data interpretation. To ensure you understand a survey's results, be suspicious of interpretation. Look at the data to see if you'd arrive at the same conclusions. Be careful to distinguish between factual results and interpretation of results. Interpretation can be filtered through bias.
To quote Frank Lloyd Wright, "The truth is more important than the facts." Just because it's easy to do online surveys with the multitude of tools available doesn't mean they should be taken lightly.
Revolutionize your digital marketing campaigns at ClickZ Live San Francisco (August 10-12)!
Educating marketers for over 15 years, our action-packed, educationally-focused agenda offers 9 tracks to cover every aspect of digital marketing. Join over 500 digital marketers and expert speakers from leading brands. Register today!
Bryan Eisenberg is co-founder and chief marketing officer (CMO) of IdealSpot. He is co-author of the Wall Street Journal, Amazon, BusinessWeek, and New York Times best-selling books Call to Action, Waiting For Your Cat to Bark?, and Always Be Testing, and Buyer Legends. Bryan is a keynote speaker and has keynoted conferences globally such as Gultaggen, Shop.org, Direct Marketing Association, MarketingSherpa, Econsultancy, Webcom, the Canadian Marketing Association, and others for the past 10 years. Bryan was named a winner of the Marketing Edge's Rising Stars Awards, recognized by eConsultancy members as one of the top 10 User Experience Gurus, selected as one of the inaugural iMedia Top 25 Marketers, and has been recognized as most influential in PPC, Social Selling, OmniChannel Retail. Bryan serves as an advisory board member of several venture capital backed companies such as Sightly, UserTesting, Monetate, ChatID, Nomi, and BazaarVoice. He works with his co-author and brother Jeffrey Eisenberg. You can find them at BryanEisenberg.com.
US Consumer Device Preference Report
Traditionally desktops have shown to convert better than mobile devices however, 2015 might be a tipping point for mobile conversions! Download this report to find why mobile users are more important then ever.
E-Commerce Customer Lifecycle
Have you ever wondered what factors influence online spending or why shoppers abandon their cart? This data-rich infogram offers actionable insight into creating a more seamless online shopping experience across the multiple devices consumers are using.