AnalyticsROI MarketingHow to Get Started With a Pre-Post Analysis

How to Get Started With a Pre-Post Analysis

The pre-post analysis, while a fundamental type of market research experiment, has countless tricky applications. Luckily, Michael stresses the key points.

The pre-post analysis is one of the most widespread and most useful types of research tools available to Web marketers. The pre-post analysis is the market research version of the before-and-after pictures you see in weight-loss-product commercials. Want to know if something works on your site? Measure it before (pre) and after (post) implementation, and see what happens.

When to Use a Pre-Post Analysis

The pre-post analysis is a fundamental type of market research experiment, and it has countless applications, such as:

  • Evaluating the overall results of a site face-lift
  • Comparing the quality of specific site content
  • Measuring the effectiveness of advertising and other promotional content
  • Measuring the effects of site changes on customer satisfaction

But rather than focus on any one of these specific applications, I’m going to stress a few key points that are critical, regardless of whatever kind of pre-post analysis you’re conducting.

Getting Started

In general, the first step is to decide at the outset on a relatively small number of outcomes to measure. These could be based on traffic or survey data, for example. An alternative would be to measure every single thing you think might be relevant to your analysis, and fish out the interesting data later. The latter approach might be reasonable in some limited circumstances, but you run the risk of finding a lot of spurious information. (I’ll say more about this below.)

Naturally, you need to collect the data. (This would be a good time to reread my column on sampling! I won’t repeat too much of what I said there, but let me stress again the importance of obtaining a random sample of your real users. This will avoid bias, provide projectable results, and allow you to do a more thorough statistical analysis.)

The Timeline

What should the “pre” and “post” be? One strategy that often works well is to consider equal-length time periods before and after the time of the site change. Choosing two adjacent time periods this way should minimize the effect of any long-term trends in the data. Of course, short-term time trends (e.g., booming sales near the holidays) can throw a monkey wrench into the works, too! If your site change occurs smack-dab in the middle of an obvious short-term trend, you might be better off choosing pre and post time periods that fall outside the trend.

The Analysis

Inevitably, you or someone you work with will want to know if the differences between the pre and the post are “significant,” meaning statistically significant. (You, of course, understand the difference between practical significance and statistical significance, having read my column on basic statistics!)

The T-Test

The usual statistical method for comparing the pre- to the post-analysis is called the two-sample t-test. For each outcome of interest, you can perform a t-test to decide whether there is a statistically significant difference between the new version of the site versus the old. This column is a formula-free zone, so here’s a link to a detailed explanation of the t-test.

There is one subtle point here. Suppose you are measuring a long list of site features for your pre-post analysis. Even if there is no real difference between the pre and the post, it is likely that just by chance alone you would get a “significant difference” between pre and post for a few elements. If you get only a few significant differences, you might wonder whether they are really meaningful. This is a statistical version of the argument I mentioned earlier for picking just a few outcomes to study. There are ways to take this into account in your analysis, but they can be pretty tricky. If you were looking for a really rigorous treatment of this kind of pre-post data, I’d hire an expert.

So now that we’ve laid the general groundwork for the pre-post analysis, remember that it has countless applications, which can be tough to interpret. Hopefully my tips for how to get started will make you feel better equipped than before you read this column.


US Mobile Streaming Behavior

Whitepaper | Mobile US Mobile Streaming Behavior


US Mobile Streaming Behavior

Streaming has become a staple of US media-viewing habits. Streaming video, however, still comes with a variety of pesky frustrations that viewers are ...

View resource
Winning the Data Game: Digital Analytics Tactics for Media Groups

Whitepaper | Actionable Analysis Winning the Data Game: Digital Analytics Tactics for Media Groups


Winning the Data Game: Digital Analytics Tactics f...

Data is the lifeblood of so many companies today. You need more of it, all of which at higher quality, and all the meanwhile being compliant with data...

View resource
Giving a Voice to Your Brand

Whitepaper | AI & Automation Giving a Voice to Your Brand


Giving a Voice to Your Brand

Voice commerce, or conversational commerce, is a major disruptive opportunity in US retail. It opens up the possibility of building deeper relationshi...

View resource
Mobile Messaging Masters

Whitepaper | Mobile Mobile Messaging Masters


Mobile Messaging Masters

Every year the mobile market continues to grow and develop. Cyber Monday 2018 saw $2.2 billion in sales stemming just from smartphones in the United S...

View resource