Measuring Outcomes and Not Just Outputs

  |  February 15, 2011   |  Comments

Are you measuring just activity levels or real outcomes? A look at some differences between the two.

People who run non-transactional websites have a tough job evaluating the success or return on investment. In contrast, if you sell stuff online or have a clearly defined transactional environment, then it's relatively easy to assess whether things are working or not. You can count the number of transactions or you can use metrics like the conversion ratio to work out how effective the site is in turning opportunity into value. But what about all the cases, like information sites or support sites, where there aren't any clearly defined transactions? What happens then?

A useful approach to measuring the effectiveness of non-transactional websites is taken from models built to evaluate the success of government policies and other intangibles. It differentiates the measurement of performance between measuring "outputs" and measuring "outcomes."

Outputs

Outputs are activities that happen on the site. These are probably the closest things that you might get to "conversion points" in a non-transactional environment. They are the tangible pieces of activity which are believed to be valuable. If you take the example of a public information website whose purpose is to support a campaign to raise awareness of a certain type of disease and encourage people to get themselves screened or tested, then the measureable outputs of the website could be things like:

  • The actual number of visits to the website which reflects the awareness of the campaign
  • The number of pages viewed by website visitors, which could be a proxy for the amount of information being consumed
  • The downloading of certain PDFs which could be interpreted as getting information further into the home
  • The number of contacts made via a Contact Us page or using an e-mail link to signify the number of people who are interested in getting further information

Virtually all of these outputs could be measured using a properly configured Web analytics system, but what the Web analytics system can't tell you is what the result of the visit to the website ended up being and whether the website was successful in its objective of getting potentially vulnerable people to get themselves tested. This is why it's also important to measure outcomes.

Outcomes

The outcomes are what happen as a result of the visit to the website. Often, non-transactional websites are trying to influence subsequent attitudes and behaviors and these are difficult to measure at the best of times and are impossible to measure using on-site behavioral data such as Web analytics systems.

In my example of the public information website above, the desired outcome is that potentially vulnerable people read the relevant information on the website and then make an informed decision to put themselves forward for screening. The approach in measuring this could be two-fold:

  • Understand the impact of the visit to the website in terms of how it has changed attitudes
  • Understand the role and influence the website had in terms of subsequent behavior

The methodology in both approaches is likely to revolve around some kind of survey or other user feedback mechanisms. In the first instance, you could run an exit survey and ask a series of questions about whether the user thought that the information was useful and also what they were likely to do next as a result of their visit. One measure of success could be the proportion of people who said that they were more likely to go to see a doctor to discuss their condition. One problem, though, might be that some people would end up going to a doctor unnecessarily!

The second approach could be to try and understand what influence the website had in a person's decision to go for screening at the time they are actually being screened. This is possibly more subjective, but it also can take into account other influences that may have happened along the way and give some insight into the context in which those types of decisions are made. The true test would be to understand what number or proportion of people who went for screening would have done if had they not been able to access the relevant information over the Web.

At a time when many organizations are trying to shift information provision or customer contact from other channels onto the Web, it's important to differentiate between measuring just activity levels and real outcomes. Measuring outcomes is hard to do and often the outcomes will manifest themselves in places other than the website, such as in the call center. It therefore requires a more holistic approach to measurement than what comes out the end of a Web analytics system.

Tags:

ClickZ Live Toronto On the heels of a fantastic event in New York City, ClickZ Live is taking the fun and learning to Toronto, June 23-25. With over 15 years' experience delivering industry-leading events, ClickZ Live offers an action-packed, educationally-focused agenda covering all aspects of digital marketing. Early Bird Rates expire May 29. Register today and save!

ABOUT THE AUTHOR

Neil Mason

Neil Mason is SVP, Customer Engagement at iJento. He is responsible for providing iJento clients with the most valuable customer insights and business benefits from iJento's digital and multichannel customer intelligence solutions.

Neil has been at the forefront of marketing analytics for over 25 years. Prior to joining iJento, Neil was Consultancy Director at Foviance, the UK's leading user experience and analytics consultancy, heading up the user experience design, research, and digital analytics practices. For the last 12 years Neil has worked predominantly in digital channels both as a marketer and as a consultant, combining a strong blend of commercial and technical understanding in the application of consumer insight to help major brands improve digital marketing performance. During this time he also served as a Director of the Web Analytics Association (DAA) for two years and currently serves as a Director Emeritus of the DAA. Neil is also a frequent speaker at conferences and events.

Neil's expertise ranges from advanced analytical techniques such as segmentation, predictive analytics, and modelling through to quantitative and qualitative customer research. Neil has a BA in Engineering from Cambridge University and an MBA and a postgraduate diploma in business and economic forecasting.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!

COMMENTS

UPCOMING EVENTS

Featured White Papers

Gartner Magic Quadrant for Digital Commerce

Gartner Magic Quadrant for Digital Commerce
This Magic Quadrant examines leading digital commerce platforms that enable organizations to build digital commerce sites. These commerce platforms facilitate purchasing transactions over the Web, and support the creation and continuing development of an online relationship with a consumer.

Paid Search in the Mobile Era

Paid Search in the Mobile Era
Google reports that paid search ads are currently driving 40+ million calls per month. Cost per click is increasing, paid search budgets are growing, and mobile continues to dominate. It's time to revamp old search strategies, reimagine stale best practices, and add new layers data to your analytics.

WEBINARS

Resources

Jobs

    • SEO Specialist
      SEO Specialist (HeBS Digital) - NEW YORK                             ...
    • GREAT Campaign Project Coordinator
      GREAT Campaign Project Coordinator (British Consulate-General, New York) - New YorkThe GREAT Britain Campaign is seeking an energetic and creative...
    • Paid Search Senior Account Manager
      Paid Search Senior Account Manager (Hanapin Marketing) - BloomingtonHanapin Marketing is hiring a strategic Paid Search Senior Account Manager...