Measuring What Matters

  |  February 1, 2011   |  Comments

Consider this framework for measuring user experience.

We're not short of numbers are we? In our digital world, we generally have access to a vast amount of data from all sorts of different digital tracking and analysis systems. And that's just the site-centric data. Increasingly, we also have user-generated data such as that from voice of the customer surveys and remote user testing programs. So the problem's not how to get ahold of data anymore, it's knowing how to manage it and to structure it in a way that allows it to tell us a story.

This is why we need to develop measurement frameworks. Measurement frameworks are a way of structuring metrics and those all-important key performance indicators around the objectives of the business. The key thing about a measurement framework is that it's coherent and helps the business to understand not only the metrics themselves, but also the relationship between metrics.

I came across a good example recently in a paper from Google that outlines a framework it uses for measuring the user experience which is called HEART. HEART stands for happiness, engagement, adoption, retention, and task success. They use this framework to understand the user experience across their various different products. The important point is that the metrics that make up the framework come from a variety of different data sources.

Happiness, for example, is measured using attitudinal metrics gathered using survey data, adoption and retention may be measured using customer data, and task success by using remote user testing methodologies. The actual metrics used to measure the user experience vary from product to product depending on the nature of the product.

For example, the way that Gmail is measured is different to the way that Google Maps is measured, but the HEART framework brings the rigour and coherence to the individual metrics and usually describes what is being measured and to some extent why. It also always helps to have a good acronym!

To develop a framework, you need to go back to the objectives and determine you need to measure and why. In an e-commerce scenario, for example, a measurement framework may be based around the transactional environment looking at sales, order, average order value, number of customers, number of orders per customer, and so on. The framework would highlight the relationship between the different variables and how a change in one might or might not influence the other.

As an example, we were recently asked by a client to help them think about how to measure the success of various initiatives. In line with many companies, they are looking to serve more of their customer service needs using the online channel. One measurement perspective would be to focus on some of the core site-centric metrics such as visits to the customer service section, number of downloads, etc.

These metrics are certainly useful in terms of understanding what's actually happening on the site but are not necessarily indicative of whether the initiative overall is successful or not. To do that we needed to develop a wider framework that looked at the customer service section in the context of the overall customer journey.

The framework included measurements that looked at to what extent customers knew about some of the online capabilities, whether they would use them or not, and what they thought of the experience. The framework essentially tells the story about the migration of customer service from offline channels to online channels and allows the organization to understand where the focus areas need to be. Does the site need to improve or does the promotion of the offline capabilities need to increase? Such a framework can help the organization to understand the most effective way to hit the initiative's goals and objectives.

With data everywhere, it's easy not to be able to see the forest for the trees. Developing multi-sourced measurement frameworks is the way to add structure to your data and to focus on the metrics that matter.


Neil Mason

Neil Mason is SVP, Customer Engagement at iJento. He is responsible for providing iJento clients with the most valuable customer insights and business benefits from iJento's digital and multichannel customer intelligence solutions.

Neil has been at the forefront of marketing analytics for over 25 years. Prior to joining iJento, Neil was Consultancy Director at Foviance, the UK's leading user experience and analytics consultancy, heading up the user experience design, research, and digital analytics practices. For the last 12 years Neil has worked predominantly in digital channels both as a marketer and as a consultant, combining a strong blend of commercial and technical understanding in the application of consumer insight to help major brands improve digital marketing performance. During this time he also served as a Director of the Web Analytics Association (DAA) for two years and currently serves as a Director Emeritus of the DAA. Neil is also a frequent speaker at conferences and events.

Neil's expertise ranges from advanced analytical techniques such as segmentation, predictive analytics, and modelling through to quantitative and qualitative customer research. Neil has a BA in Engineering from Cambridge University and an MBA and a postgraduate diploma in business and economic forecasting.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!



Featured White Papers

2015 Holiday Email Guide

2015 Holiday Email Guide
The holidays are just around the corner. Download this whitepaper to find out how to create successful holiday email campaigns that drive engagement and revenue.

Three Ways to Make Your Big Data More Valuable

Three Ways to Make Your Big Data More Valuable
Big data holds a lot of promise for marketers, but are marketers ready to make the most of it to drive better business decisions and improve ROI? This study looks at the hidden challenges modern marketers face when trying to put big data to use.