Where Have All the Control Groups Gone?

Email marketing is often the unsung hero of marketing portfolio success. Instead of praise and worship, email marketing gets a bad rap. Questions like “Is it dead?” “Does it work?” “Will it go on?” continue, everywhere…except among email marketers. Go figure, right?

What is comical is that it’s common to find that the budgets of the naysayers, the exploration of new channels, is often funded by…revenue generated from email marketing. The biggest challenge that faces most email marketers (and marketers in general) is attribution. There is very little way to attribute the proper revenue consideration to email, mobile, social, apps – the list continues to grow. So I ask, “Where have all the control groups gone?”

In an age where the mindset around email marketing is quantity over quality, the idea of a “control” group has all but disappeared – which is unfortunate, because it’s one of the single most telling methodologies to leverage in attribution modeling.

Where to Begin

First, you need to understand your entire marketing mix and map it out. There are going to be elements outside of your control, such as mass advertising opportunities that cannot be tracked to the individual, but it’s important to understand that they exist and play a role in the communication of a message or brand to the customer. Categorize your efforts in groups of targeted and mass media to see what dials you have the opportunity to turn.

Finding the Audience

Once you have determined where you will be messaging, you need to then focus on the targeted efforts and begin adjusting the “who.” It’s imperative that you keep two things in mind. First, the “select” for the test must be random and representative of the entire mix. No segmenting geographically or taking an alphabetical data set and slicing it after every 10,000 customers – it needs to be truly random. Second, it needs to be statistically viable; each cell of the test needs to meet the requirements for a statistically viable sample size – often 10 percent works; depending on the size of your data set you may even be able to get away with a smaller percent – but better to be safe than sorry.

Determine the Mix

Now that you have the “who,” you need to figure out exactly what they’re going to get. It’s suggested to execute these types of tests over a period of time, especially with attribution. A one-time proof does not a point make. But consistently exposing the same audience to a similar or like marketing mix will truly help you determine what impact each channel has on the likelihood to convert.

Let’s take an example here. Say you have a database of 1 million subscribers that have proper permission to receive marketing messaging in all of your four available targetable channels: direct mail, email, SMS, and push-app notifications. The hypothesis is that email is the strongest contributor to the portfolio. To keep it manageable, break the audience into four categories: email only, email + direct mail, email + direct mail + SMS, and all four channels. This would allow you to measure the impact of adding the additional channels to communication flow. Clearly the messaging needs to be on point and the timing considerations are important – but now you have something to measure.

Be Diligent

As you get everything ready to go, be sure each audience is held out in its assigned category for the duration of the test and that each category has its own set of tracking tags and categorizations so that you can easily determine who is part of which data set. While tests like this can get complex to manage, your diligence and attention to detail through the process will certainly simplify matters.

It’s important to remain the master of the test – there are likely going to be scenarios where someone wants to “just get a message out to everyone via all channels,” but these exceptions will certainly corrupt your test, and ultimately the ability to determine attribution and lift by channels.