Where Have All the Control Groups Gone?

  |  September 4, 2012   |  Comments

In an age where the mindset around email marketing is quantity over quality, the idea of a "control" group has all but disappeared.

Email marketing is often the unsung hero of marketing portfolio success. Instead of praise and worship, email marketing gets a bad rap. Questions like "Is it dead?" "Does it work?" "Will it go on?" continue, everywhere…except among email marketers. Go figure, right?

What is comical is that it's common to find that the budgets of the naysayers, the exploration of new channels, is often funded by…revenue generated from email marketing. The biggest challenge that faces most email marketers (and marketers in general) is attribution. There is very little way to attribute the proper revenue consideration to email, mobile, social, apps - the list continues to grow. So I ask, "Where have all the control groups gone?"

In an age where the mindset around email marketing is quantity over quality, the idea of a "control" group has all but disappeared - which is unfortunate, because it's one of the single most telling methodologies to leverage in attribution modeling.

Where to Begin

First, you need to understand your entire marketing mix and map it out. There are going to be elements outside of your control, such as mass advertising opportunities that cannot be tracked to the individual, but it's important to understand that they exist and play a role in the communication of a message or brand to the customer. Categorize your efforts in groups of targeted and mass media to see what dials you have the opportunity to turn.

Finding the Audience

Once you have determined where you will be messaging, you need to then focus on the targeted efforts and begin adjusting the "who." It's imperative that you keep two things in mind. First, the "select" for the test must be random and representative of the entire mix. No segmenting geographically or taking an alphabetical data set and slicing it after every 10,000 customers - it needs to be truly random. Second, it needs to be statistically viable; each cell of the test needs to meet the requirements for a statistically viable sample size - often 10 percent works; depending on the size of your data set you may even be able to get away with a smaller percent - but better to be safe than sorry.

Determine the Mix

Now that you have the "who," you need to figure out exactly what they're going to get. It's suggested to execute these types of tests over a period of time, especially with attribution. A one-time proof does not a point make. But consistently exposing the same audience to a similar or like marketing mix will truly help you determine what impact each channel has on the likelihood to convert.

Let's take an example here. Say you have a database of 1 million subscribers that have proper permission to receive marketing messaging in all of your four available targetable channels: direct mail, email, SMS, and push-app notifications. The hypothesis is that email is the strongest contributor to the portfolio. To keep it manageable, break the audience into four categories: email only, email + direct mail, email + direct mail + SMS, and all four channels. This would allow you to measure the impact of adding the additional channels to communication flow. Clearly the messaging needs to be on point and the timing considerations are important - but now you have something to measure.

Be Diligent

As you get everything ready to go, be sure each audience is held out in its assigned category for the duration of the test and that each category has its own set of tracking tags and categorizations so that you can easily determine who is part of which data set. While tests like this can get complex to manage, your diligence and attention to detail through the process will certainly simplify matters.

It's important to remain the master of the test - there are likely going to be scenarios where someone wants to "just get a message out to everyone via all channels," but these exceptions will certainly corrupt your test, and ultimately the ability to determine attribution and lift by channels.


ClickZ Live Toronto On the heels of a fantastic event in New York City, ClickZ Live is taking the fun and learning to Toronto, June 23-25. With over 15 years' experience delivering industry-leading events, ClickZ Live offers an action-packed, educationally-focused agenda covering all aspects of digital marketing. Register today!


Kara Trivunovic

As an email marketing veteran dating back to 1999, Kara Trivunovic has been actively involved in programmatic email development, execution, and strategy in a variety of senior positions on the client, agency, and provider side. She was founder and principal of The Email Advisor, a respected email marketing consultancy focusing on email strategy and channel optimization that was subsequently acquired by an enterprise email service provider in 2009. Over the course of her career, she has had the opportunity to work with a variety of brands and global organizations structuring a variety of custom email education programs, conceptualizing and implementing new and innovative email programs, optimizing contact strategies, and developing staffing and budget plans. Kara currently serves as the vice president of digital solutions at Epsilon.

COMMENTSCommenting policy

comments powered by Disqus

Get ClickZ Email newsletters delivered right to your inbox. Subscribe today!



Featured White Papers

Gartner Magic Quadrant for Digital Commerce

Gartner Magic Quadrant for Digital Commerce
This Magic Quadrant examines leading digital commerce platforms that enable organizations to build digital commerce sites. These commerce platforms facilitate purchasing transactions over the Web, and support the creation and continuing development of an online relationship with a consumer.

Paid Search in the Mobile Era

Paid Search in the Mobile Era
Google reports that paid search ads are currently driving 40+ million calls per month. Cost per click is increasing, paid search budgets are growing, and mobile continues to dominate. It's time to revamp old search strategies, reimagine stale best practices, and add new layers data to your analytics.



    • SEO Specialist
      SEO Specialist (HeBS Digital) - NEW YORK                             ...
    • GREAT Campaign Project Coordinator
      GREAT Campaign Project Coordinator (British Consulate-General, New York) - New YorkThe GREAT Britain Campaign is seeking an energetic and creative...
    • Paid Search Senior Account Manager
      Paid Search Senior Account Manager (Hanapin Marketing) - BloomingtonHanapin Marketing is hiring a strategic Paid Search Senior Account Manager...