The toughest part of ROI measurement is the measurement piece! In the last couple of columns I’ve been looking at some of the approaches to understanding how to evaluate the return on investment (ROI) in analytics, but how easy it is to actually to measure it?
I don’t think it is easy. As discussed in the first part of this series, measuring the investment piece is probably more straightforward than identifying and measuring the returns. Generally the accountants will tend to do that for you, as it’s mainly about measuring the costs of technology, services, and people. In fact, all too often they will be very transparent! That’s inherently often the problem in that the costs of analytics are transparent but the returns are relatively opaque. So is it possible to get more transparency into understanding the returns on analytics?
Last time I suggested that it’s useful to chunk the problem down and to start to think about the returns generated by analytics into the “tactical” returns and the “strategic” returns, tactical returns tending to be more immediate and the strategic returns tending to be longer term and perhaps less tangible.
With tactical investments designed to generate tactical returns, measurement can be easier and should be built into the business case for the investments in the first place. For example, investments in technology areas such as conversion rate optimization (CRO) are designed to produce specific results such as improvements in the conversion process and this is something that most businesses have been measuring for years. This has evolved from simple A/B testing through to more complex multi-variate testing and personalization approaches but the framework is broadly the same: The optimization process generates certain results, those results can be measured and can be compared against the total investment in that program of work.
As with all measurement processes, there is the need to be aware of the maxim “be careful what you measure because what you measure is what you will get.” A focus on the measurement and the optimization of one metric can be at the expense of another one. So for example by investing in CRO technologies that drive tangible returns, are there other areas of the business that are being impacted, such as the overall customer experience perhaps?
A balance is needed between the immediate and measurable impact of analytics and the longer-term and more intangible aspects. The latter is certainly a more complex management and measurement problem and requires structured thinking. The goal is to understand the value that these investments in technologies, services, and people deliver to the business. This means trying to answer touch questions like: What is the business value of:
- Adding one more person to my analytics team?
- Having better data integration capabilities?
- Having better data visualization technologies?
- Engaging with this particular consulting business?
These types of strategic questions need more strategic measurement frameworks and also need to be addressed in the context of the analytics maturity of the organization and the trajectory that it has traveled in. Analytics functions are there to serve the organization and it’s only by being explicit as to how they serve that it can be possible to measure their impact. In a way this is no different from what analytics functions do on a day-to-day basis; they understand what the key performance indicators (KPIs) are and they put in place the analytics capabilities to provide the tracking, diagnostics, insight, and intelligence required to help the organization achieve those objectives. The question is how well they are doing that.
“Measuring the measurers” requires a systematic approach to defining, understanding, and evaluating the value delivered by analytics. The value delivery will be different for different organizations and will need to be contextualized and defined. For example, does the analytics function have the ability to directly influence the KPIs of the organization through its activities? If not, do internal customers and stakeholder groups believe that the analytics function adds value to the business?
The approach to addressing the first issue would be for the analytics function to evaluate the expected dollar value impact of each of its recommendations or actions and record the results as well. These results may take months to materialize and that is why a system or framework is required. Understanding the internal impact of an analytics function would require some type of survey approach that could be benchmarked over time.
As I said at the beginning, I don’t pretend this is easy and I don’t think there is a “one size fits all approach” to measuring the ROI of analytics. In some cases it will be possible to measure the outputs, but more often than not it’s going to require some deep thinking about how to evaluate the outcomes. But that’s what analysts do, right?