Advanced Testing for Marketers

  |  February 8, 2005   |  Comments

Applying advanced test design to marketing is nothing new. Marketers must study testing techniques and understand how they work.

A number of years ago, an interesting debate raged between a senior manager and an agency analyst in a Boston ad agency.

"I'm not buying it," said the senior manager. "You mean to tell me our client, a large telecommunications company, is wasting millions of dollars because of the way they conduct their in-market tests?"

The analyst making the suggestion wasn't sure exactly what to say. It was true. The client was spending more money on testing than was necessary to obtain the answers it wanted. Delivering that news was difficult, as the agency itself had designed and conducted the tests for years.

Multivariate Testing

The concept of properly designed multivariate tests has been around since at least the early 1900s. The first practical applications were in the agricultural sector. Farmers wanted to increase crop yields, so they'd test how different combinations of variables (fertilizer, water, etc.) affected plants.

It wasn't feasible for them to test one variable at a time (known as an A/B testing). They didn't want to treat half their fields with one amount of fertilizer, the other half with another amount, then wait for harvest time for the results. And then have to wait again until the following year to find out what affect water had on the yield.

Instead, the farmers use "factorial" or "fractional factorial" tests. This is a carefully managed methodology of treating different parts of the fields with different combinations of factors (fertilizer level, water level, etc.). Results were carefully analyzed to reveal the optimal levels of all tested factors.

Marketing Applications

The telecommunications company mentioned above (I was the analyst, by the way) was in a similar situation. It was testing multiple factors, including offer levels, promises, feature combinations, and so on, and wanted to know the optimal combination of all the factors. Unlike the farmers, the client was using very carefully crafted A/B testing rather than the more useful fractional factorial tests.

Not a New Idea

Applying advanced test design to marketing isn't new. It's mentioned briefly by David Ogilvy in his classic "Ogilvy on Advertising." The practicality of designing and conducting such tests in marketing is greatly increased by the technology advancements of the last decades. Computer power and software are readily available to help marketers (and others) take advantage of the designs.

The benefits of using multivariate testing include reduced sample sizes, faster results, and potentially better results than A/B testing.

Not Trivial, But Doable

As evidenced by that Boston discussion a few years back, marketers must study testing techniques before they understand how the tests work. They can be quite beneficial. Gordon Bell wrote a good article explaining the concept of advanced test design in the marketing world.

Did I Mention Software?

Marketers and analysts can chose a number of software routes to help with the design and analysis of advanced tests. SAS offers two: a helpful interface into the relevant SAS procedures, and one from a company formerly known as JMP.

The updated version of Charles Hicks' "Fundamental Concepts in the Design of Experiments" discusses how to use SPSS and Minitab to analyze test results.

Today and Yesterday

I'm happy to report that today, the Boston agency uses these testing techniques to successfully design, execute, and analyze tests and experiments that examine many variables. The tests are conducted both in direct mail and Web site design applications.

How do I know about the history of these tools? Many years ago, I spent some time in the agricultural world.

ClickZ Live Toronto Twitter Canada MD Kirstine Stewart to Keynote Toronto
ClickZ Live Toronto (May 14-16) is a new event addressing the rapidly changing landscape that digital marketers face. The agenda focuses on customer engagement and attaining maximum ROI through online marketing efforts across paid, owned & earned media. Register now and save!


Brian Teasley Brian Teasley is the leader of Teasley, a consultancy that helpsadvertisers, marketers and advertising agencies use data and analysis toimprove their marketing campaigns. Brian has over 14 years experience inengineering and marketing, and has worked for numerous Fortune 100companies. Brian also teaches a marketing course at New York University. Heholds a M.S. degree in Applied Statistics from Iowa State University and aBA in Mathematics and a BA in Mathematics and Statistics from St. OlafCollege.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!



Featured White Papers

ion Interactive Marketing Apps for Landing Pages White Paper

Marketing Apps for Landing Pages White Paper
Marketing apps can elevate a formulaic landing page into a highly interactive user experience. Learn how to turn your static content into exciting marketing apps.

eMarketer: Redefining Mobile-Only Users: Millions Selectively Avoid the Desktop

Redefining 'Mobile-Only' Users: Millions Selectively Avoid the Desktop
A new breed of selective mobile-only consumers has emerged. What are the demos of these users and how and where can marketers reach them?


    • Contact Center Professional
      Contact Center Professional (TCC: The Contact Center) - Hunt ValleyLooking to join a workforce that prides themselves on being routine and keeping...
    • Recruitment and Team Building Ambassador
      Recruitment and Team Building Ambassador (Agora Inc.) - BaltimoreAgora,, continues to expand! In order to meet the needs of our...
    • Design and Publishing Specialist
      Design and Publishing Specialist (Bonner and Partners) - BaltimoreIf you’re a hungry self-starter, creative, organized and have an extreme...