Metrics Identify Problems, Not Solve Them

A thoughtful and intriguing email arrived this week from Max Blumberg, a statistician with considerable experience in Web metrics. He writes:

I’d like to suggest that the metrics available today are still in themselves insufficient to guide Web site designers into redesigning the site so as to meet the needs of different customer segments — i.e., just because I know that focus and yield, etc. are low in this section of my site is not sufficient to tell me “how” to fix it. However, this is not to say that being guided by any metric is better than having none at all, but I’d say we are a long way from being able to translate customer Web site behavior into useful metrics that can lead us to make the required changes to our Web site. I believe that today’s metrics can help only those who already have domain expertise in this regard.

I completely agree: Metrics can help to identify a problem; they cannot, currently, present the “right” solution. Luckily for those of us who like the marketing process, there is still a need for individual thought and experience — and for the ability to reason through the problem — to find the optimal solution for each unique situation.

Max also sent some thoughts about how the future of Web metrics might address this gap. Without getting into the impressive statistical detail he gave, I’m including a few highlights of his input:

Given a whole bunch of variables (e.g., offline shopping style, clickstreams, revenue, conversions, abandoned carts, customer loyalty, RFM [recency/frequency/monetary], etc.), SEM — structured equation modeling — is able to suggest which one of those variables might have caused another.

In fact, the real benefit of SEM is in testing suggested models relating those variables. So, for example, the model might hypothesize that offline shopping style, social class, and income determine level of online purchasing acting through specific clickstreams if these are available on the site in question. Testing such a model would be a nightmare using traditional stats because of the number of possible relationships that might exist (e.g., who knows whether social class doesn’t in fact “cause” shopping style?). So one could gather a whole bunch of data from someone’s data warehouse, propose interesting models linking the variables, test them with SEM, and refine the models depending on how accurate SEM tells us they are.

Of course if research shows that e-metrics have not produced significant improvements (in ROI [return on investment], turnover, etc.), this wouldn’t necessarily mean that the metrics were bad; it could simply mean that users are not yet applying them usefully. In itself, this would be a significant and useful finding.

Interesting future directions — when the technology and infrastructure to manipulate and draw useful conclusions from data catches up with the information needs we in the online world face.

Until then, we need to use the metrics as indicators of a problem or opportunity, then test every variable possible to get the results we seek. And we need to apply good marketing principles and a keen eye for observing customer behaviors to make the best of the metrics we now have available.

It’s no cakewalk, but for those who crack the code of how their customers want to interact, it’s well worth the effort.