Talking back and forth with several readers of my column that dealt with valuating new types of media opportunities, a common question kept coming up. To quote one reader’s email, “What criteria would you use to establish the minimum number of impressions for an untested product?”
This is an issue that comes up frequently — not just with new types of media. Every time a new site comes around or a new section within a site comes out or, perhaps, a new targeting mechanism appears within an older site, we face the question of how many impressions will give us a good idea of how this media performs.
It’s easy to give a knee-jerk response to some thousands of impressions, but we have to take into account several factors.
First, you must have enough impressions to be statistically significant. This is usually easy, as they’re pretty cheap. We can get tens of thousands of impressions for several hundred dollars.
But don’t go purchasing such a small lot to be run all at one time. This will skew your test. If you buy only 50K impressions, and they run over the course of 10 minutes on a Tuesday night at 3 a.m., you could get some pretty funky performance results. You need to either buy several different lots over the course of a week’s time or purchase enough media to ensure that the site will not have the opportunity to “flush” out the buy too quickly.
Controls are problematic, particularly with unique types of media or new types of media. When trying to test the effectiveness of a new manner of geographical targeting, for instance, you really need to control that test with another — known — geographical targeting media buy. And this can be a royal pain when you’re trying to match up the relative effectiveness of the two buys on a region-by-region basis. You need to purchase an awful lot of media to make sure you get a large-enough sample in any one geographical location.
One last factor worth bringing up remains the issue of burnout, especially in vertical industries and the B2B space. As one reader put it: “We have several clients that have not been online before. Two, in particular, are traditionally B2B clients who are making their first consumer efforts. We know that their audience spends time online and can identify the sites that make sense. The weight becomes a tricky issue… How would you approach this problem?”
In this particular case, working with a narrow audience within the constraints of a narrow topic, it’s quite easy to purchase too much media. You can buy all the relevant sites and wind up getting the same poor schmoes over and over again. This is especially true with the targeting technologies.
A great example is the keyword buy. If you use one of the syndicated research-panel companies to attempt to determine the “overlap” from one site to the next, you could be gravely deceived. While, on average, 2 users out of 10 might visit both sites A and B, when 10 users on site A use the keyword “carburetor,” you can be pretty sure that the 10 “carburetor” searchers on site B will have a higher-than-normal overlap.
So how do you figure out how much media to buy? You apply two tests.
For the first test, make sure you’re purchasing enough media. Are you buying enough impressions to be significant? Does your test remain significant even when you break down all the different targeting pieces? Will the media run over the course of a period of time so that you don’t get strange time-of-day and day-of-week problems?
For the second test, make sure you aren’t buying too much media. This is best determined by watching declining response rates, the typical symptom of burnout.