Poor Study Results Drive Ad Research Foundation Initiative

Advertiser demand for quick and dirty online studies and a dearth of participants has corrupted the quality of some online research, a problem the Advertising Research Foundation is hoping to change with its newly launched Online Research Quality Council. As calls for transparency in online measurement increase, the goal of the new ARF committee is to create standards for research firms and their clients when it comes to Web panel-based research studies.

“The biggest issue that is perplexing advertisers….is the inability to replicate the studies,” said ARF President and CEO Robert Barocci. What researchers and their advertiser clients are finding, said Barocci, is it’s been difficult to duplicate the results from one panel “so that people in the business can have confidence that the results they’re getting are valid and predictable.”

There are a couple factors contributing to this dilemma that the ARF aims to understand through the council’s work. One is over-participation by survey respondents. “Those people are making a living taking online surveys,” said Barocci, who suggested the cash and prize incentives awarded to online survey participants might create bias in favor of the advertiser in question.

“I think some of the recruiting techniques are a little unusual,” Barocci said, noting “It used to be believed that if you pay people, they’d be biased going in.”

Among other things, the Online Research Quality Council will address this perceived prejudice. A meeting is set for September 10th to discuss a preliminary proposal for guidelines for Web panel measurement.

“We’re hoping to get a plan outlined or agreed to at that meeting,” said Barocci, who said he’d be pleased to see a hundred or more people discussing the issues at the September get-together. He’d like for official standards to be unveiled during the annual ARF convention in April, he added. The council steering committee is comprised of six execs representing research firms including TNS, Ipsos Interactive Services, NPD Group, comScore and Harris Interactive.

Some believe a high demand from advertisers for cheap, speedy online studies is resulting in erosion in the quality of Web panel research.

“I’ve witnessed the pressure that panel companies are under from advertisers to provide very low cost” studies, said Tom Kelly, COO of Safecount, a recently re-launched arm of WPP’s research division Kantar. Rather than collecting data from a set panel of respondents, the company conducts live Web intercept studies, inviting individuals to take brief five-minute surveys.

Safecount is also dealing with a lack of Web survey participants, driven by privacy concerns. According to Kelly, online research firms have experienced a drop in survey cooperation rates over the past few years.

“We need to understand the impact of non-response,” said Kelly.

Although many in the ad industry have expressed a lack of confidence in Web panels for some time, Barocci suggested concern among ARF members “has grown dramatically” since the organization’s meeting last fall focused on the decline in Web survey response and cooperation rates.

Barocci is quick to differentiate between ARF’s initiative and the Interactive Advertising Bureau’s recent demand for audits of Nielsen//NetRatings’ and comScore’s methodologies. The impetus for the IAB’s efforts is “a currency issue,” he said; the IAB’s goal is to shed light on methods for measuring Web site audiences, which in turn influence how ads are priced.

Still, there does seem to be a common theme in the recent calls for transparency in online research and measurement. “The research space online is maturing,” said Kelly.

But in order for the ARF to accomplish its mission, Kelly believes the organization will need cooperation from the advertiser community.

“People in the client space need to become stakeholders in this effort,” he said. “I don’t think any groups or mandates will work unless there’s advertiser buy-in.”