Taking Stock of Our Toolbox, Part 2

I must admit the volume of responses to my last column was a little disappointing. I’m certain many folks out there are as impatient as I am about perfecting our arsenal of tools. Maybe they’re all on summer vacation. At any rate, I did get about a dozen responses which fell into three basic categories: overall comments around problems with research today, new research companies eager to present their wares, and current research companies that felt slighted by the column.

A few of the overall comments about our research today focused on the notion of credibility (or lack thereof). Simply put, syndicated research companies seem to lose credibility when the companies (or sites) they report on are also clients of theirs.

For example, although we may use Gartner research as part of our planning process, we really don’t know which of the entities being reported on are actually Gartner’s consulting-services clients. Does Gartner look more favorably on client companies? People seem to feel the same level of discomfort about the online ratings business. A few people who wrote to me wondered whether Media Metrix or Nielsen//NetRatings looked more favorably on those sites that are paying them for “custom research.”

Maybe we are just a cynical bunch. I have to believe as the industry matures and becomes more established, the research companies are going to have to stop playing both sides of the fence. I don’t know if the same quandary exists in the traditional media research space (I’ve been out of that world for a little while now), and I would love to hear your thoughts on the matter. Though it almost goes without saying, we have so many barriers to overcome in the interactive space, the very least we can do is ensure the data we are getting is as objective and accurate as possible.

The one part of our syndicated toolbox we didn’t cover in my last column was competitive intelligence. You can guarantee to fire up any client by looking at its category and competitive activity in the marketplace. We typically begin presentations such as those with a number of caveats that put the data into context.

Just a few of the things that make accurate competitive reporting difficult: the challenge of differentiating between paid and “added-value” placements, the significant amount of barter and/or in-house ads among online publishers, the blurry/nonexistent line between advertising and content, the lack of real rate cards to accurately assess spending levels, and technological hurdles in capturing all forms of rich media. That’s why we use syndicated research to look at competitive spending directionally. If we assume the numbers are inaccurate, we can assume they’re inaccurate across the board. We use the research to report on relative differences in spending across advertisers, as well as providing a reasonably good creative “clipping” tool to analyze creative and messaging.

There are three leading services: Nielsen’s AdRelevance and TNS Media Intelligence/CMR’s AdNetTrackUS and Evaliant. You may be asking, “Which one is better?” To which I would respond, “Damned if I know!”

We’ve taken a number of our clients’ spending and run reports from both CMR and AdRelevance. There is really no rhyme or reason to the results. Sometimes the numbers are much higher (by a factor of two to three), sometimes lower, and sometimes even almost correct. If hard pressed to make a sweeping generalization, we’ve found the CMR products tend to underestimate spending while AdRelevance inflates spending (sometimes wildly).

My hope is as the technology gets better and these companies learn more accurate ways to count impressions, the data will become more accurate. I believe all the services use some type of spidering technology that collects what it believes to be ads and registers the impressions accordingly. The point at which the impression information gets translated into actual dollar volume is a particularly slippery slope and one that could use some radical improvement.

There is no doubt our toolbox has swelled with new measurement services and tools to make our marketing decisions more grounded. A careful analysis of the syndicated landscape indicates widely duplicative services with moderately little differentiation. Before signing up for any of them, make sure you have done your due diligence. Products are changing regularly, and new players are always entering the picture.

Note: One of the companies that contacted me after reading part one of this column was Hitwise, an Australian-born “competitive intelligence” service. I would place the company in the “ratings” bucket, although executives seem adamant about not being lumped in the comScore/Nielsen camp. Nevertheless, Hitwise gets its data from ISPs and tracks 10 million U.S. users — a large enough number to deliver rich insights. The outputs were interesting, and the information provided is certainly a twist on what is currently available. Give Hitwise a look.

Join us at the Jupiter ClickZ Advertising Forum in New York City on July 30 and 31.

Related reading