Building Effective Measurement Models

Heading into the annual planning season (the end of 2014 is now closer than the beginning!), it’s useful to stop and consider the underlying models that guide business decisions. Even more than the models, it’s a great time to reflect on the basis for the models themselves. This matters, as the choice of how a model is derived fundamentally sets up success metrics, guiding how process improvements are made.

Most models begin with assumptions: a certain input leads, typically through a formula or other defined relationship, to some specific output. Increase ad spending…and awareness goes up, for example. At a deeper level, there is also a choice made that governs the basis for the model itself: is it based on a known (or believed to be known) mechanism that links actions with discrete results, or on an observed relationship between the two?

Models based on assumed linkages – called “mechanistic models” – are more common in business, probably because they are intuitively comfortable: they allow people to connect what they see with what they believe. It’s easy to believe that “with increased spend on advertising, as offer will get noticed more.” A mechanistic model for ad planning might therefore focus on budget and assume a relationship between awareness and spending, expressed through intermediary metrics like reach and frequency.

By comparison, models based on observed conditions and outcomes independent of assumed linkages – called “correlation models” – do away with the “why” and instead simply relate what ultimately happened to whatever else was happening at the time. These models require lots of data but they don’t rely on an understanding of any given process. Correlation models are therefore powerful tools in discovering relationships between action taken and results that may have been otherwise ignored.

Here’s an example: email open rates. Because we’re all consumers, we tend to think in terms of consumer-observables: taste, color, location, price, convenience, etc. So, when designing an email campaign, an investment in rich graphics to catch the attention of a recipient seems logical. After all, if the graphics are striking and catch attention, then it’s reasonable to assume a higher open rate.

Not so fast. Take a look at this email open rate data from MailChimp. MailChimp has email data, obviously, and lots of it. So, they can analyze all sorts of relationships: using correlation – the study of related events – they can find unusual but nonetheless robust relationships that are associated with (note: I did not say “cause”) higher open rates. Take a look at these email open rate charts, showing open rate (y-axis) presented against time of receipt (left) and day of the week (right).


Now, back to the question of open rate and what drives it: Want to double your open rate? Send your email Tuesday, Wednesday, or Thursday so that it arrives in the afternoon of the time zone of your recipient. And you thought you needed an ad agency!

Of course, in real life it’s not always that simple (and yes, you really do need your agency!) But here’s the point: By combining mechanistic models with insights gleaned from correlation, you can improve your mechanistic model.

Who knows why open rates are higher in the afternoon? Who cares? They are. You might think (but you’d be wrong) that people would read email while having a coffee and “getting their day started” and so email rates would be higher then. But for whatever reason, they are highest midweek, afternoon.

Note as well that correlation does not imply causation: I wrote a specific section about this in the “Metrics and Measurement” section of my book, Social Customer Experience. You may want to dig deeper, and you’d certainly want to test this to make sure it applies to your business.

Regardless, as you continue refining your business models, consider adopting aspects of the study of correlation. Use this to discover apparent relationships that may challenge (or change!) your thinking on the critical drivers for your business. And for the time being send your email midweek, in the afternoon!

Related reading