The Holy Grail of Market Research

StatMarket gives an amazing, real-time reading on the technological pulse of the Net. Using data gathered by thousands of free “hit counters,” StatMarket’s computers crunch data 24X7, distilling the numbers into an amazing snapshot of what browsers people are using, which plug-ins they’ve installed, where they’re hailing from, what OS they use, and even what screen resolution they’ve got their monitor set to. The WebSideStory folks who created StatMarket are claiming a sample size of 28,797,751 people per day from 99,737 sites.

Now I’m not going to go into a long discussion of what their stats say about who is using what, and I’m not here to get into an argument on sample skew or methodology. Heck, you can go make up your own mind about that.

What I want to discuss today is data and how the finding, collecting, mining, and refining of it in the future may completely change the paradigm of web marketing and advertising.

One thing that’s become clear over the past year is how the web population is starting to look more and more like the mainstream, offline audience. Gender split is projected to be even by 2001 (Computer Economics). Fifty-one percent of those going online are over 35. Half have a high-school education or even less. And more than half planning on taking the plunge have incomes of $50,000 (stats from Intelliquest). The information superhighway is starting to look more and more like Main Street.

If there’s one thing that StatMarket proves to us, it’s that computers are very good at collecting data. Day in and day out, hour by hour, those StatMarket hit counters sit there and collect information on users. Actually, all of our sites collect volumes of data about users Just check your server logs some time.

So what? Here’s what. As the web starts to look more and more like the bricks-and-mortar world, those stats we collect start to become more and more valuable.

Looking at where people are going, what they’re doing, what product pages they spend the most time on, what they’re buying, and what they’re NOT were interesting statistics when the web didn’t resemble the consumer marketplace. Now that it looks exactly like the analog consumer world (or soon will), information like that is gold to marketers.

For years, marketing and product development decisions have been made by focus groups, consumer panels, and surveys — all good measures, but measures that had inherent flaws based on sample size, self selection, or the fact that people tend to answer differently when they know they’re being surveyed. Sure, we had market data to look at, but often that was too late to do any quick maneuvering, and it only told the story about the people that bought, not those that didn’t.

Real-time POS systems, data mining technologies, and other weapons in the high-tech retailer’s arsenal have helped to clear up the picture, but they still don’t tell the whole story of consumer behavior. While retail anthropologists have shed a little more light on the subject, their methods still rely on subjective observations of a limited sampling.

We’ve got none of these problems on the web. Instead we’ve got real-time measurement of all consumers entering a site. We can see what they’re interested in, what they ignore, what offers work, what don’t, what level of technology they use, what they’re confused about, and what they like. It’s literally the Holy Grail of market research if we’d just start to use it.

And I don’t mean just obvious uses for tweaking e-commerce sites. Today, every site’s logs have the potential for yielding valuable market data. From finding consumer trends by measuring links on personal web sites to a determining a star’s real popularity by the number of fan sites that pop up lots can be learned on a mainstream web.

The MP3 phenomenon has helped break the record industry lock on who even gets the chance to be popular. It doesn’t seem too far out to think about a future where musicians release “beta” MP3s of their work before going to the expense of recording a whole album’s worth of music. And I’ve always found I’ve been able to gauge what issues are hot with my friends and family by the subject lines of the joke mails I get.

As the web moves towards the middle, it’s imperative as marketers that we pay more attention to web stats. In fact, I don’t think that it’s too bizarre to imagine a world where sites support themselves not by advertising but by the kinds of data they sell, with ISPs offering free access to anyone who agrees to let their every move be monitored.

As upping the ante on using web measurement, data must also mean upping the ante on privacy; (something that’s becoming increasingly important with consumers) striking a balance between data and privacy is the key to reaping the riches of the market data we’ve now got at our fingertips.

Related reading