Measuring Outcomes and Not Just Outputs

  |  February 15, 2011   |  Comments

Are you measuring just activity levels or real outcomes? A look at some differences between the two.

People who run non-transactional websites have a tough job evaluating the success or return on investment. In contrast, if you sell stuff online or have a clearly defined transactional environment, then it's relatively easy to assess whether things are working or not. You can count the number of transactions or you can use metrics like the conversion ratio to work out how effective the site is in turning opportunity into value. But what about all the cases, like information sites or support sites, where there aren't any clearly defined transactions? What happens then?

A useful approach to measuring the effectiveness of non-transactional websites is taken from models built to evaluate the success of government policies and other intangibles. It differentiates the measurement of performance between measuring "outputs" and measuring "outcomes."

Outputs

Outputs are activities that happen on the site. These are probably the closest things that you might get to "conversion points" in a non-transactional environment. They are the tangible pieces of activity which are believed to be valuable. If you take the example of a public information website whose purpose is to support a campaign to raise awareness of a certain type of disease and encourage people to get themselves screened or tested, then the measureable outputs of the website could be things like:

  • The actual number of visits to the website which reflects the awareness of the campaign
  • The number of pages viewed by website visitors, which could be a proxy for the amount of information being consumed
  • The downloading of certain PDFs which could be interpreted as getting information further into the home
  • The number of contacts made via a Contact Us page or using an e-mail link to signify the number of people who are interested in getting further information

Virtually all of these outputs could be measured using a properly configured Web analytics system, but what the Web analytics system can't tell you is what the result of the visit to the website ended up being and whether the website was successful in its objective of getting potentially vulnerable people to get themselves tested. This is why it's also important to measure outcomes.

Outcomes

The outcomes are what happen as a result of the visit to the website. Often, non-transactional websites are trying to influence subsequent attitudes and behaviors and these are difficult to measure at the best of times and are impossible to measure using on-site behavioral data such as Web analytics systems.

In my example of the public information website above, the desired outcome is that potentially vulnerable people read the relevant information on the website and then make an informed decision to put themselves forward for screening. The approach in measuring this could be two-fold:

  • Understand the impact of the visit to the website in terms of how it has changed attitudes
  • Understand the role and influence the website had in terms of subsequent behavior

The methodology in both approaches is likely to revolve around some kind of survey or other user feedback mechanisms. In the first instance, you could run an exit survey and ask a series of questions about whether the user thought that the information was useful and also what they were likely to do next as a result of their visit. One measure of success could be the proportion of people who said that they were more likely to go to see a doctor to discuss their condition. One problem, though, might be that some people would end up going to a doctor unnecessarily!

The second approach could be to try and understand what influence the website had in a person's decision to go for screening at the time they are actually being screened. This is possibly more subjective, but it also can take into account other influences that may have happened along the way and give some insight into the context in which those types of decisions are made. The true test would be to understand what number or proportion of people who went for screening would have done if had they not been able to access the relevant information over the Web.

At a time when many organizations are trying to shift information provision or customer contact from other channels onto the Web, it's important to differentiate between measuring just activity levels and real outcomes. Measuring outcomes is hard to do and often the outcomes will manifest themselves in places other than the website, such as in the call center. It therefore requires a more holistic approach to measurement than what comes out the end of a Web analytics system.

Tags:

ClickZ Live Chicago Join the Industry's Leading eCommerce & Direct Marketing Experts in Chicago
ClickZ Live Chicago (Nov 3-6) will deliver over 50 sessions across 4 days and 10 individual tracks, including Data-Driven Marketing, Social, Mobile, Display, Search and Email. Check out the full agenda and register by Friday, Oct 3 to take advantage of Early Bird Rates!

ABOUT THE AUTHOR

Neil Mason

Neil Mason is SVP, Customer Engagement at iJento. He is responsible for providing iJento clients with the most valuable customer insights and business benefits from iJento's digital and multichannel customer intelligence solutions.

Neil has been at the forefront of marketing analytics for over 25 years. Prior to joining iJento, Neil was Consultancy Director at Foviance, the UK's leading user experience and analytics consultancy, heading up the user experience design, research, and digital analytics practices. For the last 12 years Neil has worked predominantly in digital channels both as a marketer and as a consultant, combining a strong blend of commercial and technical understanding in the application of consumer insight to help major brands improve digital marketing performance. During this time he also served as a Director of the Web Analytics Association (DAA) for two years and currently serves as a Director Emeritus of the DAA. Neil is also a frequent speaker at conferences and events.

Neil's expertise ranges from advanced analytical techniques such as segmentation, predictive analytics, and modelling through to quantitative and qualitative customer research. Neil has a BA in Engineering from Cambridge University and an MBA and a postgraduate diploma in business and economic forecasting.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!

COMMENTS

UPCOMING EVENTS

Featured White Papers

IBM: Social Analytics - The Science Behind Social Media Marketing

IBM Social Analytics: The Science Behind Social Media Marketing
80% of internet users say they prefer to connect with brands via Facebook. 65% of social media users say they use it to learn more about brands, products and services. Learn about how to find more about customers' attitudes, preferences and buying habits from what they say on social media channels.

Marin Software: The Multiplier Effect of Integrating Search & Social Advertising

The Multiplier Effect of Integrating Search & Social Advertising
Latest research reveals 68% higher revenue per conversion for marketers who integrate their search & social advertising. In addition to the research results, this whitepaper also outlines 5 strategies and 15 tactics you can use to better integrate your search and social campaigns.

Resources

Jobs

    • Digital Marketing Analyst
      Digital Marketing Analyst (GovLoop) - Washington D.C.Are you passionate about audience acquisition? Love effective copy and amazingly effective...
    • Product Specialist
      Product Specialist (Agora Inc. ) - BaltimoreDescription: The Product Specialist is hyper-focused on the customer experience and ensures that our...
    • Partnerships Senior Coordinator
      Partnerships Senior Coordinator (Zappos.com, Inc.) - Las VegasZappos IP, Inc. is looking for a Partnerships Senior Coordinator! Why join us? Our...