How to Get Started With a Pre-Post Analysis
The pre-post analysis, while a fundamental type of market research experiment, has countless tricky applications. Luckily, Michael stresses the key points.
The pre-post analysis, while a fundamental type of market research experiment, has countless tricky applications. Luckily, Michael stresses the key points.
The pre-post analysis is one of the most widespread and most useful types of research tools available to Web marketers. The pre-post analysis is the market research version of the before-and-after pictures you see in weight-loss-product commercials. Want to know if something works on your site? Measure it before (pre) and after (post) implementation, and see what happens.
When to Use a Pre-Post Analysis
The pre-post analysis is a fundamental type of market research experiment, and it has countless applications, such as:
But rather than focus on any one of these specific applications, I’m going to stress a few key points that are critical, regardless of whatever kind of pre-post analysis you’re conducting.
Getting Started
In general, the first step is to decide at the outset on a relatively small number of outcomes to measure. These could be based on traffic or survey data, for example. An alternative would be to measure every single thing you think might be relevant to your analysis, and fish out the interesting data later. The latter approach might be reasonable in some limited circumstances, but you run the risk of finding a lot of spurious information. (I’ll say more about this below.)
Naturally, you need to collect the data. (This would be a good time to reread my column on sampling! I won’t repeat too much of what I said there, but let me stress again the importance of obtaining a random sample of your real users. This will avoid bias, provide projectable results, and allow you to do a more thorough statistical analysis.)
The Timeline
What should the “pre” and “post” be? One strategy that often works well is to consider equal-length time periods before and after the time of the site change. Choosing two adjacent time periods this way should minimize the effect of any long-term trends in the data. Of course, short-term time trends (e.g., booming sales near the holidays) can throw a monkey wrench into the works, too! If your site change occurs smack-dab in the middle of an obvious short-term trend, you might be better off choosing pre and post time periods that fall outside the trend.
The Analysis
Inevitably, you or someone you work with will want to know if the differences between the pre and the post are “significant,” meaning statistically significant. (You, of course, understand the difference between practical significance and statistical significance, having read my column on basic statistics!)
The T-Test
The usual statistical method for comparing the pre- to the post-analysis is called the two-sample t-test. For each outcome of interest, you can perform a t-test to decide whether there is a statistically significant difference between the new version of the site versus the old. This column is a formula-free zone, so here’s a link to a detailed explanation of the t-test.
There is one subtle point here. Suppose you are measuring a long list of site features for your pre-post analysis. Even if there is no real difference between the pre and the post, it is likely that just by chance alone you would get a “significant difference” between pre and post for a few elements. If you get only a few significant differences, you might wonder whether they are really meaningful. This is a statistical version of the argument I mentioned earlier for picking just a few outcomes to study. There are ways to take this into account in your analysis, but they can be pretty tricky. If you were looking for a really rigorous treatment of this kind of pre-post data, I’d hire an expert.
So now that we’ve laid the general groundwork for the pre-post analysis, remember that it has countless applications, which can be tough to interpret. Hopefully my tips for how to get started will make you feel better equipped than before you read this column.