Testing Dynamic E-Mail Campaigns

 

In creating a dynamic email campaign, the final challenge is testing. Traditionally, marketers and creatives have reviewed and approved each message variant individually prior to sending. However, dynamic campaigns produce so many variants that reviewing by hand isn’t practical. Even when it can be done, it’s often ineffective. Reviewers lose concentration and focus under a flood of similar content. The solution is to use sampling and statistical analysis to ensure consistent quality.

To marketers and content developers used to reviewing every item prior to release, this can present a huge psychological and practical hurdle. Such a change is often quite alarming, and many don’t have the expertise to develop alternative testing methodologies. Fortunately, the software industry has long faced the same testing problems, and many techniques they’ve developed can be applied to dynamic campaigns.

The first thing software development teaches is quality is baked in from the beginning, not added at the end. Testing must start right at the beginning with data validation.

As I explained a while back, data hygiene is essential. The business rules provide the framework for the data constraints: which attributes must be present and which are optional; and what are appropriate values, ranges, and formats for those attributes. These restrictions can be applied at a database level as integrity constraints (define) or tested programmatically to ensure the data are valid. To perform date comparisons, for example, it’s essential every date is in the same format.

The next area of early testing is the content. Unlike static campaigns, it’s not feasible to wait until all copy is produced to approve the content. Reviewing hundreds of paragraphs of content with no flow and no context is a challenging task, one with which even experienced copyeditors struggle. Content should be reviewed and approved as it’s produced. This is akin to the software approach of module testing, where each piece of code is tested as it’s written. Depending on the amount of content, number of variants, and level of test coverage, this may be the only time a given piece of content is viewed prior to sending.

Just as the business rules facilitate data validation, they can be used to validate final messages. Various aspects of the campaign can be analyzed to identify potential issues. You may use a very simple form of this in your standard email campaigns by comparing the final recipient count with the expected count.

Data issues with dynamic messaging increase the possibility of recipients receiving no message or multiple messages. So for dynamic messaging, the process is taken a step further. Analyze counts by recipient segment and content segment to identify problem areas. This approach can also be applied to content combinations that can be especially important if you’re dynamically including special offers in your messaging. (Sending a discount offer to 100,000 people instead of 1,000 can be costly!)

Even though it’s impractical to view every message variant, it’s still useful to view some variants. The question is, which ones? The most problematic cases are the edge cases (define). The simplest edge cases are the smallest and largest messages generated. Business rules facilitate identification of additional edge cases where recipients may be prone to receiving too much, too little, or contradictory content.

In addition to the edge cases, it’s also valuable to view some of the mainstream versions. Many content issues aren’t identified until the content is seen in situ. We’ve all had clients who’ve reviewed and approved content, only to make changes when they receive the test message. Given that, the best variants to view are those that will be seen by the largest number of recipients. This is analogous to software code coverage testing, ensuring the most common usages are the most thoroughly tested.

Due to the complexity and number of message variants, testing for a dynamic campaign is more akin to testing software than reviewing and approving a traditional campaign. Although this can be challenging, by drawing on the foundations of solid objectives and business rules, dynamic campaigns of enormous complexity can be successfully tested without having to review a thousand separate email messages.

Until next time,

Derek

Want more email marketing information? ClickZ E-Mail Reference is an archive of all our email columns, organized by topic.