The How, What, and Why of Testing for Agile Marketers

In A Day in the Life of an Agile Marketer, we presented to you a vision of the perfect agile marketer, Melanie, as she went through her day.

This perfect world vision may have made you a little sick. Who really works in a marketing environment like that?

You don’t have to become this kind of über agile marketer overnight. You can “plug in” at any of several places. Testing is one place that touches on many of the agile marketer’s practices including measurement, content development, and goal setting. I asked Rose Holston of Aviso Communications to answer some of the main questions marketers have when starting down the path toward implementing an agile online testing program.

Testing is the single most important skill you’ll develop as an agile marketer.

Testing is not hard. Becoming a testing organization is. The act of setting up a simple test requires you to increase your proficiency in a number of areas; proficiencies that are crucial for progressive, data-driven marketing. For example, managing the IT department is an important proficiency.

To be successful, you need a plan. To execute that plan, every decision should be backed up with quantitative results that help evolve the plan. The more specific you get about the outcome you are trying to affect, the more you will learn from testing. There are three basic questions to be addressed.

1. Why will we test?
2. What do we test?
3. How do we test it?

Why Will I Test?

A picture is worth a thousand words.

In order to understand the power of testing, start with a picture or baseline of what is happening today. Take a snapshot of the past 30 days.

Where are visitors coming from?

When are they visiting more often?

Did they leave right away or did they visit multiple pages?

Did they take the actions I expected?

There are several easy-to-use tools and services that are useful in delivering answers.

• Google Analytics: It provides statistics about the visitors coming to your website, where they come from, the content they visited, and if they took the actions you intended.
• Search rank reporting: This tells you where your pages rank for specific keyword phrases.
• Paid search campaigns: This is a great tool to drive traffic to your website, and helps you to test messaging and develop landing pages that work.
• Bit.ly: This URL shortener is a great tool to not only shorten URLs but to measure click-through rates for links you have to external websites.

What Do I Test?

Think about outcomes.

Once you have a baseline, or that first picture, you can make assumptions about what to change and which enhancements can be made on your website pages.

Think in terms of numbers. Analytics provide some high-level statistics that give us clues to areas that need improvement.

Bounce Rate

A bounce rate determines if a visitor leaves after viewing only one page on your site. Pages with high bounce rates may be chasing visitors away.

Exit Percentage

Like the bounce rate, the exit percentage tells us which pages visitors tend to leave from. Although visitors may be leaving a particular page in large numbers, be careful in making quick assumptions. Visitors may be getting what they were looking for.

Conversion Rate

A “conversion” is the completion of an action you want a visitor to take. This could be a downloading a paper, submitting a form, buying, etc.

How Do I Test It?

Execute the plan over and over again.

A test plan organizes your day much like a marketing plan or editorial calendar. Test planning evolves as you learn more about the behaviors of your target audience and bump up against your own resource and budget constraints. Keep in mind that test plans are an excellent tool for reporting results to the C-suite. Testing just may get you more resources and budget.

A test plan has five major components.

Hypothesis

The “what” and “why” questions are answered by how you think the outcome may be affected.

Implementation

Define the details of the test. Include images of the creative in this section.

1. Control: A control is what you are measuring against. This is the current creative, or the winner of a previous test.
2. Treatment(s): Constitute what you are going to test against the control. For example, you may be testing multiple e-mail subject lines against your original control subject line.

Use of a control and treatment doesn’t imply a split testing strategy, in which traffic is sent equally to two different pages. You can also do serial testing, in which you post changes one after the other, measuring the increase or decrease in performance after each. Serial testing doesn’t require split testing software, though it still requires the discipline to review results and roll back to a previous version if changes reduce results.

Measurement Plan

Decide which metrics you’ll use to determine if your hypothesis is true. This is your baseline picture.

Results

Estimating how long to run a test depends on your current traffic and the number of conversions you generate. You must reach a point of statistical significance. Once you have defined a timeframe to run the test, you will record your results against the measurement plan. The test has now taken shape.

Analysis

Your measurement plan and results are the picture that becomes a story that you can now put into words. At this stage, we speculate on “why” visitors reacted in the way they did.

This process documents a complete “Kaizen” exercise. Kaizen means “good change.” Your goal is to define potential “good changes” that will instruct the next series of tests. On paper, your story reads like a novel, each test becomes a chapter in the book. Your expectations should be open as the purpose of your hypothesis is to prove or disprove what you think will happen.

Note: An agile marketer will always expect the unexpected.

All great efforts run into resistance. Let us know in the comments if you’ve tried to implement tests and what barriers you’ve come up against.