ComScore Blog Study Sparks Controversy, Vitriol

Hostilities flared this week between the two best-known blog networks after comScore released a blog readership study that was co-sponsored by Six Apart and blog network Gawker Media.

The turmoil highlighted both the problems of panel-based media research and the increasingly high stakes of blog advertising.

The report aims to measure the size and characteristics of the blog audience, as well as to rank the most highly trafficked sites in this category. Its findings, perhaps predictably, have been both hailed and criticized by those with a stake in blog advertising models.

Media buyers have been clambering for just the sort of demographic profiling it offers, and blog publishers should benefit overall from its discovery that the blog audience is both richer and younger than the overall Internet audience.

Yet the research has been challenged by several prominent bloggers, including — most loudly — Jason Calacanis, publisher of Gawker rival Weblogs Inc. Network (WIN).

Calacanis accused comScore of bias and inconsistency, pointing specifically to discrepancies between blog traffic rankings offered by the report versus those sites’ own stats packages. He calls into question the number of sites published by report sponsor Gawker Media that fall in the top 20 sites ranked by unique visitors and visits. And he’s particularly appalled at the report’s rating of Gawker.com over WIN’s Engadget.com.

“[The report] contradicts [Gawker Media publisher Nick Denton’s] own statistics that he has publicly available on his site, that’s the smoking gun right there,” Calacanis told ClickZ News. On his blog, Calacanis has accused comScore of bias based on the personal relationship between Denton and the report’s author, Rick Bruner. He has also called for comScore to publicly release its data to the blogosphere in Excel format.

Denton posted findings of the study to his blog on Monday, hours before its release. He also shared figures showing the specific traits of Gawker Media’s audience, which he said he requested for sales purposes.

Denton dismissed Calacanis’ objections.

“I know it galls Jason Calacanis that his sites are about as memorable as Burger King franchises, and that none register among the top blogs, except Pete Rojas’s Engadget,” he said.

Denton added, “But Jason Calacanis misses the big picture. The study finally provides evidence for what we’ve all hoped for: that blog readers are younger and richer than average, and, one hopes, thinner.”

Calacanis noted Denton and report author Rick Bruner are friends. Bruner, who is director of research at DoubleClick, was never an employee of comScore, but worked on the report pro bono. He said he approached comScore to produce the report. ComScore agreed, but wanted a sponsor. “I reached out to Nick and the folks at Six Apart, they were interested and we got the sponsorship and proceeded,” said Bruner.

The controversy over the report highlights more than just the colorful personalities of the individuals at its center. It also points to the considerable problems of online audience measurement.

“These are some technicalities that can lead to differences,” said comScore SVP Dan Hess. “There are other data sets out there that seek to estimate audiences; many use number of links to a site, or users of a particular toolbar that visit a site, data points that can be interesting, but not based on panels built specifically for media research.”

In a blog post responding to criticism of the report, Bruner wrote, “My opinion is the best information market research can give us is this: ‘Is it bigger than a breadbox?’ This research study satisfactorily answers that question for the blogosphere: Yes. There is no flawless methodology in market research. It’s an inexact science. Samples get biased, corners are cut, yadda-yadda-yadda. It’s always directional, at best.”

ComScore uses data collected from a panel of 1.5 million U.S. Internet users. Bruner added the relative rankings of Gawker.com and WIN’s Engadget.com could be explained by a larger international audience for the WIN property.

As the debate rages, comScore stands by its data.

“Sponsorship has no bearing on the production of the research,” said Hess. “The same standards of objective research apply. We conducted the study using objective and best available methods at the time we did it, and various approaches and methods that we used are disclosed in the report.” Hess said a small fraction of the overall work that comScore does is sponsored.

Calacanis said he wanted to sponsor the report. “I asked to be involved,” he said, and claims an employee at comScore hung up on him earlier this week when he called to discuss the findings.

Some other bloggers have echoed doubts raised by Calacanis on his own blog. Among those to publicly weigh in on it are digital media expert Jeff Jarvis, who authors BuzzMachine, and Fred Wilson, a prominent digital marketing venture capitalist, comScore investor, and blogger. Wilson commended the research firm for tackling the subject, but said, “There are some questionable stats in this report.”