Forecasting Techniques, Part 2: Qualitative Methods

Last time, we started to look at forecasting and investigated a set of methods called time-series techniques. Using these techniques, you forecast purely based on the patterns in the historical data of what you’re trying to forecast. For example, you forecast the number of visits to the site next week based on the volume of traffic over the past few weeks. These techniques don’t take any external factors into account; they rely mainly on identifying underlying trends and seasonality.

What happens if you’re launching a major new TV advertising campaign next week? Won’t that affect site traffic? You hope so, but by how much?

This is where explanatory forecasting techniques come into play. They’ve been used for many years in offline marketing analysis to understand the marketing activity’s effect on important outcomes, such as sales and brand awareness. These techniques build a model where the thing you’re interested in, such as visits, registrations, leads, or sales, is explained quantitatively by external factors such as TV advertising, promotions, price, and so on. This branch of techniques is often called econometrics. One of the most popular methods is regression analysis.

Use of any advanced analytical method is as much an art as a science. This is certainly true when it comes to techniques such as regression analysis. Regression analysis is widely accessible through programs such as Excel and, to a limited extent, even PowerPoint. A former colleague used talk about these types of tools as “like putting guns in the hands of children.” Though a bit arrogant, the point is valid. These algorithms can be quite dangerous if there isn’t the right kind of care and thought about how they’re used and what results say.

The trouble with having these kinds of techniques available is there’s a tendency for them to be used in inappropriate ways or when the model or forecast construction isn’t fully thought through.

I got myself into trouble many years ago when cutting my teeth on econometric modeling. We were doing some modeling work on price elasticity, trying to forecast the effect of a proposed price increase on a client’s sales. The model’s results suggested a massive detrimental result, bigger than anything the client had ever seen before.

The client naturally didn’t like the results and said there must be something wrong with the model. I countered that the model was technically correct according to all the diagnostic statistics, but I’d look at it again. At this point, I noticed an event I hadn’t taken into account in the model the first time. This event made the brand look like it was far more sensitive to price changes than it really was. When we factored this event in, price sensitivity became something more appropriate. We were able to make a much more sensible forecast.

I took two key lessons from that experience:

• If the model looks wrong, it probably is.
• Modeling is like baking a cake.

The first lesson is the law of common sense. Though you’re trying to look for insight through these more advanced analytical techniques, the results should still make some sense.

The second lesson is you must ensure you have all the right ingredients to get the right result. If you want to forecast sales, for example, you must ensure you’re capturing as many of the likely effects on sales as possible in the model. If you don’t, either the wrong effects will inadvertently come through or large errors will be associated with the forecasts.

Is there any use for these types of techniques in online marketing? Much of online marketing analysis is based on direct response or tracking individuals over time. This level of granularity is fantastic. It allows us to get deep into the analysis of individual visitor behavior.

I wonder, though, whether sometimes we can’t see the forest for the trees. How do TV advertising and press ads influence online behavior? What can we infer about the synergistic effects of multichannel marketing? These are all areas in which modeling techniques will help us better understand the return on investment on marketing spend, as they have in the offline world for a number of years.