Despite Reservations, ComScore and NetRatings Carry on with Audits

Despite its contention that audience measurement data from Web site publishers is flawed, as presented in its just-published white paper, comScore is following through on the auditing process it promised to undergo last month. In fact, the firm recently hired a chief research officer to shepherd the process. Its fellow leader in the audience measurement space, Nielsen/NetRatings, is knee deep in its full-audit, too. This amounts to a lot of extra work for something some say will do little to alleviate discrepancies between the measurement firms and site publishers.

Since a resolute IAB missive was sent in April requesting both companies have their methodologies audited, each has agreed to complete the process in the hopes of obtaining accreditation from the Media Ratings Council.

NetRatings Chief of Measurement Science Mainak Mazumdar has devoted 30 to 40 percent of his time to it in the past few months. That’s in addition to his regular responsibilities managing the company’s media programs. ComScore meanwhile has six staffers conducting various functions for the audit. Yesterday the company announced the hire of a chief research officer, Josh Chasin, who will head up the process.

Firms including PriceWaterhouse Coopers and Deloitte and Touche, perhaps better known for their financial investigations than their site measurement methodology inspections, typically conduct the pre- and full-audits. Both comScore and NetRatings have bestowed their audit honors on Ernst and Young.

“Fundamentally, they’re not researchers,” said Magid Abraham, CEO of comScore, of auditing firms like Ernst and Young. “They may be well-versed in technology and statistics, but they don’t deal with a media research service everyday. My understanding is their job is to verify certain assertions made by the service providers,” he continued. “Ultimately whether those assertions lead to accreditation or not….That’s something they may be counsel to, but the MRC research committee is the final decision maker.”

The MRC acts as a liaison between the company undergoing the audit and the CPA firm performing it; once the audit is complete, MRC members employ audit documents to evaluate the methodology in question, determining whether or not to accredit it.

In the pre-audit phase, the MRC assesses a company’s operations to isolate things that need adjustment before the actual audit. “You basically know the answers before you pass the test,” said Mazumdar.

For its pre-audit, completed in ’06, a NetRatings team of around four people fed Ernst and Young details on things such as how their data are reported and how they recruit for their panel of site visitors, providing logins and passwords to the NetRatings system, and speaking to auditors over the phone.

None of the pre-auditing took place in person, though. “They’re not sitting here next to me and looking at our meters and the processes and everything else,” Mazumdar told ClickZ News.

ComScore is nearly done with its pre-audit phase, which has involved providing details on such things as how the company compiles enumeration and demographics data.

Out of these pre-audits come reports from the CPA firm containing preliminary evaluations of the measurement methodologies. When NetRatings received theirs, it was about 50 pages long.

What comes next is what Mazumdar refers to as “the most important piece” of the process: the research plan. Started last year for NetRatings, the research phase entails the MRC analyzing various aspects of the methodologies, from sample selections and weighting to incentive schemes employed to get people to join its audience panel. It’s the MRC’s job to determine whether Ernst and Young’s findings satisfy its standards. The expediency of that process, however, is subject to the MRC’s schedule.

The full audit and accreditation could take many months, even over a year. Audits of some measurement methods for other media have taken several years.

The impetus for all of this is an ongoing battle pitching the measurement firms’ panel-based data against publisher servers’ site log data; disparities are typical in the measurement of how many and what types of people are visiting sites. As noted in the IAB letter clamoring for the audits, “our members’ server logs continue to diverge starkly from your companies’ sample-based assessments, by 2x to 3x magnitudes in some cases – far beyond any legitimate margin of sampling error.”

Certain standards, such as how an at-work unique visitor is defined, are also cloudy. In addition, both firms are adjusting their panel-based systems to include randomly-selected users and recruited users, methods the official audits and MRC analysis could actually help develop.

Though it has focused on audits for comScore and Netratings, the IAB has also stated plans to request that site publishers have their internal systems audited. “There are no double standards,” IAB SVP and GM Sheryl Draizen told ClickZ News last month.

“We’re very happy the IAB asked for an audit of all parties,” said comScore’s Abraham. After publishers are audited, it might actually resolve the discrepancies, he said, adding, ”I think the benefit [of the audit] is way exaggerated in terms of resolving the discrepancy with server side data.”

Indeed, the white paper comScore published earlier this week presented findings that around 30 percent of U.S. computer users clear their first-party cookies each month, suggesting publisher servers end up tracking the same user as multiple users each month. The company contended, “site-server data can grossly overstate the number of unique visitors to a site and, as such, are simply not a reasonable surrogate for measuring the true number of people that visit a site.”

Whether audits of the measurement firms and of publishers will settle the industry contention remains to be seen; however, stated Abraham, “An audit is a good exercise in transparency, and the learning will benefit everyone involved.”