Your Job Is Impossible

  |  July 17, 2014   |  Comments

The job of a digital analyst is incredibly difficult - and a paper from Google and Microsoft drives that point home, with a focus on the near impossibility of measuring return on investment.

Yes, you always knew it and now there's proof - from Google and Microsoft no less.

The paper is called "On the Near Impossibility of Measuring the Returns to
" and is about a year and a half old.

For those of you who do not have the time (I'm looking at you - all of you!) here's the TL;DR version:

Using 25 online field experiments, representing $2.8 million in ad spend, Randall A. Lewis (economic research scientist, Google) and Justin M. Rao (economic researcher, Microsoft) showed that you cannot measure the causal impact of choice variables on profit even given access to reliable signals.

Attribution? Fuhgettaboutit.

In their own words:

"We find that even when ad delivery and consumer purchases can be measured at the individual level, linked across purchasing domains, and randomized to ensure exogenous exposure, forming reliable estimates on the returns to advertising is exceedingly difficult, even with millions of observations. As an advertiser, the data are stacked against you."

There's just too much noise in the system. To arrive at statistical significance, you'd need to run so many ads that marketplace conditions will have changed between start and finish. You'd need millions of person-weeks of trials to "reliably distinguish disparate hypotheses such as 'the ad had no effect' (-100% ROI from 'the ad was profitable for the firm' (ROI>0%)."

your-job-is-impossible-1Even worse, "Answering questions such as 'was the ROI 15% or -5%' ... or 'was the annualized ROI at least 5%,' ... typically require at least hundreds of millions of independent person-weeks - nearly impossible for a campaign of any realistic size."

And that's just for return on investment (ROI). "Determining the profit maximizing level of ROI is far harder, as it requires one to estimate the shape of the underlying profit function. We briefly discuss the (rather incredible) difficulties of this enterprise."

The paper has gratifying graphs to occupy your left and right brains, such as this figure depicting an ROI model where, "c* gives the optimal spend level, and ch (h for 'high') gives the spend level where ROI is exactly 0%. Any point past ch and the firm has negative earnings on the spend whereas any point to the left of c∗ the firm is under-advertising."

There are scads of equations and pages of scholarly references, but the conclusion is that old John Wannamaker is plumb out of luck: "The advertising market as a whole may have incorrect beliefs about the causal impact of advertising on consumer behavior."

So What's a Digital Analyst to Do?

The same thing professionals in other industries do: We make it up as we go along.

People practice medicine. There is no definitive definition of "healthy."
People practice law. You may win a case, but it can be appealed.
People practice accounting. Everybody is open to an audit.
People practice coding. There is no ideal program.

We have Marketing Mix Modeling at the top of the food chain and Optimization at the bottom. At the high end, you can get rather sophisticated juggling budgeted buckets of money to shift a little more from television to search pay-per-click (PPC) or from online display to retargeting. At the bottom, you can employ complex testing tools and tweak user experience design for incremental improvements in outcomes.

alphabet-soupBut in the middle, there is attribution, where the goal is to create a model that can account for so many variables, it simply tells you where, when, and how to spend your money.

This assumes that if we learn how to stir this alphabet soup well enough and stir it long enough, it will eventually spell out all the works of Shakespeare.

In the words of Lewis and Rao, this is not impossible, but exceedingly difficult.

In the words of George Box, "All models are wrong, but some are useful."

If you find a useful model, congratulations! Use it as long as it remains useful, but no longer. And please heed that other piece of good advice, "In statistics, as in art, never fall in love with your model."

If I have gored your ox, please let me know. There are mathematicians, software vendors, book writers, and pundits who have helped us reach this moment of great expectation and Big Data enchantment. If you are among them, I would love to understand if there is a LaGrange Point between Marketing Mix Modeling and Optimization that I just don't see at the moment.

Images via Shutterstock.

ClickZ Live Toronto On the heels of a fantastic event in New York City, ClickZ Live is taking the fun and learning to Toronto, June 23-25. With over 15 years' experience delivering industry-leading events, ClickZ Live offers an action-packed, educationally-focused agenda covering all aspects of digital marketing. Register today!

ClickZ Live San Francisco Want to learn more? Join us at ClickZ Live San Francisco, Aug 10-12!
Educating marketers for over 15 years, ClickZ Live brings together industry thought leaders from the largest brands and agencies to deliver the most advanced, educational digital marketing agenda. Register today and save $500!


Jim Sterne

Jim Sterne is an international consultant who focuses on measuring the value of the Web as a medium for creating and strengthening customer relationships. Sterne has written eight books on using the Internet for marketing, is the founding president and current chairman of the Digital Analytics Association and produces the eMetrics Summit and the Media Analytics Summit.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!



Featured White Papers

Gartner Magic Quadrant for Digital Commerce

Gartner Magic Quadrant for Digital Commerce
This Magic Quadrant examines leading digital commerce platforms that enable organizations to build digital commerce sites. These commerce platforms facilitate purchasing transactions over the Web, and support the creation and continuing development of an online relationship with a consumer.

Paid Search in the Mobile Era

Paid Search in the Mobile Era
Google reports that paid search ads are currently driving 40+ million calls per month. Cost per click is increasing, paid search budgets are growing, and mobile continues to dominate. It's time to revamp old search strategies, reimagine stale best practices, and add new layers data to your analytics.




    • GREAT Campaign Project Coordinator
      GREAT Campaign Project Coordinator (British Consulate-General, New York) - New YorkThe GREAT Britain Campaign is seeking an energetic and creative...
    • Paid Search Senior Account Manager
      Paid Search Senior Account Manager (Hanapin Marketing) - BloomingtonHanapin Marketing is hiring a strategic Paid Search Senior Account Manager...
    • Paid Search Account Manager
      Paid Search Account Manager (Hanapin Marketing) - BloomingtonHanapin Marketing is hiring an experienced Paid Search Account Manager to...