ARF Research Council Sets Goals to Improve Web Research Studies

In keeping with its mission to improve the quality of Web panel-based research studies, the Advertising Research Foundation’s recently-launched Online Research Quality Council met this week to organize working committees and set out a timeframe for planned projects. The group has organized a Steering Committee comprised of research company executives, a Client Advisory Board made up of advertiser representatives, as well as four committees that will focus on research quality, metrics benchmarks, and raising funds for vetting research.

“We have to work fast because global standards and metrics are required [now]…advertisers need them,” ARF President and CEO Robert Barocci told ClickZ News.

The council’s initial meeting, which took place September 10, attracted 125 execs from research firms and their advertiser clients. Participants from research outfits Burke, Inc., Global Market Insite and Knowledge Networks presented what is commonly known as “research on research,” examining what is and isn’t working when it comes to studies driven by online surveys.

A presentation by Burke COO Jeffrey Miller, for instance, suggested threats to Web research quality include non-response errors, fraudulent respondents and “conditioning” of highly active survey respondents.

“This is a world that has exploded dramatically because it’s fast and cheap,” said Barocci of online surveys. “We think that fast, cheap and good can be achieved… but the high price of low quality needs to be addressed.”

Indeed, one of the main drivers of the council’s formation last month is the conception that advertiser demand for quick and dirty online studies has eroded the quality of some online research. One factor, as pointed out by Burke, is the tendency for some survey participants to participate in multiple studies on a regular basis.

“There are these people out there called ‘heavy responders’ that do maybe 10 studies a month. Intuitively, you say, ‘That’s probably biasing the results,’ ” said Barocci. Still, there may be a benefit to enlisting frequent survey respondents who are more dedicated to the process than others. “Nobody’s ever tested that,” he added.

Among the four council committees formed, the Funding and Outreach Committee is intended to drum up cash to pay for just such testing. “[The ARF] doesn’t have enough money in our primary coffers” for that type of research, Barocci said.

Three other committees organized by the group will isolate survey quality definitions and measures, and analyze metrics used to gauge market research quality. The Define Quality Committee is tasked with determining standards for panel management processes, sampling, survey design, data processing, study replication and accuracy.

The council’s Quality Matters Committee will pinpoint key elements of market research quality, and what can or should be done to limit the negative impact of study processes set up by suppliers and advertisers. Another subgroup focused on metrics will measure strengths and weaknesses of panel and non-panel sampling methods, and determine whether a common set of metrics can be employed by all parties.