Creative testing: Why is the control so hard to beat?

After testing over 30K image and video ads over the past year, Consumer Acquisition's Brian Bowman provides some interesting insights into how Facebook operates its creative ad testing platform.

Date published
March 12, 2020 Categories

30-second summary:

  • Facebook’s split-tests appear to maintain a creative history for the control ad. This gives the control a significant advantage with Impressions Per Thousand (IPM) testing.
  • Facebook can also group variations with a similar look and feel to the control. So similar-looking ads may also start with a higher IPM.
  • Creative concepts that are visually very different from the control do not share a creative history. IPMs for these variations tend to start near zero.
  • New, “out of the box” visual concepts may require more impressions for Facebook to quantify their performance.

High-performance creative is a rare thing for social advertising. In our experience, after spending over $3 billion dollars driving UA across Facebook and Google, usually only one out of twenty ads can beat the “best performing control” (the top ad). If a piece of creative doesn’t outperform the best video, you lose money running it. Losers are killed quickly, and winners are scaled to the moon.

The reality is, a vast majority of ads fail. The chart below shows the results of over 17,100 different ads. Spend is distributed based on ad performance. As you can see, out of those 17,000 ads, only a handful drove a majority of the spend.

The high failure rate of most creative shapes creative strategy, budgets and ad testing methodology. If you can’t test ads quickly and affordably, your campaign’s financial performance is likely to suffer from a lot of non-converting spend.

But testing alone isn’t enough. You also must generate enough original creative concepts to fuel testing and uncover winners. Over the years, we’ve found that 19 out of 20 ads fail (5% success rate), you don’t just need one new creative: You need 20 new original ideas or more to sustain performance and scale!

And you need all that new creative fast because creative fatigues quickly. You may need 20 new creative concepts every month, or possibly even every week depending on your ad spend and how your title monetizes (IAA or IAP). The more spend you run through your account, the more likely it is that your ad’s performance will decline.

So, we test a lot of creative. In fact, we produce and test more than 100,000 videos and images yearly for our clients! But we don’t just test a lot of new creative ideas. We also optimize our creative testing methodology. That might sound a little “meta,” but it’s essential for us to be validate and challenge our approach and results.

When we choose a winning ad out of a pack of competing ads, we’d like to know that we’ve made the right decision. When we kill four out of five new concepts because they didn’t test well, we react by moving on and pivoting the creative strategy based on results to try and find other ideas that may perform.

Because the outcomes of our tests have consequences – sometimes big consequences – we test our testing process. We question our testing methodology and the assumptions that shape it.

How we’ve been testing creative until now

For the past few years, in an effort to streamline our Facebook and Google creative testing and reduce non-converting spend, we’ve been testing new video concepts using IPM (Impressions Per Install) as the primary metric.

For the record, using IPM is not the Facebook recommended best practice to allow ad sets to get out of the learning phase by gathering enough data to become statistically valid.

When testing creative we typically would test three to five videos along with a control video using Facebook’s split test feature.

We would show these ads to broad or 5-10% LALs (Lookalike) audiences, and restrict distribution to the Facebook newsfeed only, Android only and we’d use mobile app install bidding (MAI) to get about 100-250 installs.

If one of those new “contender” ads beat the control video’s IPM or came within 10%-15% of its performance, we would launch those potential new winning videos into the ad sets with the control video and let them fight it out to generate ROAS.

We’ve seen hints of what we’re about to describe across numerous ad accounts and have confirmed with other advertisers that they have seen the same thing. But for purposes of explanation, let’s focus on one client of ours and how their ads performed in recent creative tests.

In the two months, we produced +60 new video concepts for this client. All of them failed to beat the control video’s IPM. This struck us as odd, and it was statistically impossible.

We expected to generate a new winner 5% of time or 1 out of 20 videos – so 3 winners. Since we felt confident in our creative ideas, we decided to look deeper into our testing methods.

Traditional testing methodology includes the idea of testing a testing system or an A/A test. A/A tests are like A/B tests, but instead of testing multiple creatives, you test the same creative in each “slot” of the test.

If your testing system/platform is working as expected, all “variations”, should produce similar results assuming you get close to statistical significance.

If your A/A test results are very different, and the testing platform/methodology concludes that one variation or another significantly outperforms or underperforms compared to the other variations, there could be an issue with the testing method or quantity of data gathered.

Here’s how we set up an A/A test to validate our non-standard approach to Facebook testing.

The purpose of this test was to understand if Facebook maintains a creative history for the control and thus gives the control a performance boost making it very difficult to beat – if you don’t allow it to exit the learning phase and reach statistical relevance.

Things to note here:

We ran this test for only 100 installs. Which is our standard operating procedure for creative testing.

Once we completed our first test to 100 installs, we paused the campaign to analyze the results. Then we turned the campaign back on to scale up to 500 installs in an effort to get closer to statistical significance.

We wanted to see if more data would result in IPM normalization (in other words, if the test results would settle back down to more even performance across the variations).

However, the results of the second test remained the same. Note: the ad set(s) did not exit the learning phase and we did not follow Facebook’s best practice.

The results of this first test, while not statistically significant, were surprisingly enough to merit additional tests. So we tested on!

Second A/A test of video creative

For our second test we ran the six videos shown below. Four of them were controls with different headers; two of them were new concepts that were very similar to the control. Again, we didn’t run the hotdog dogs; they’ve been inserted to protect the advertiser’s identity and to offer you cuteness!

The IPMs for all ads ranged between 7-11 – even the new ads that did not share a thumbnail with the control. IPMs for each ad in the far right of the image.

Third A/A test of video creative

Next, we tested six videos: one control and five visually similar variations to the control but one very different to a human. IPMs ranged between 5-10. IPMs for each ad in the far right of the image.

Fourth A/A test of video creative

This was when we had our “ah ha!” moment. We tested six very different video concepts: the one control and five brand new ideas, all of which were visually very different from the control and did not share the same thumbnail.

The control’s IPM was consistent in the 8-9 range, but the IPMs for the new visual concepts ranged between 0-2. IPMs for each ad in the far right of the image.

Here are our impressions from the above tests:

IMP testing summary

Here are the line graphs from the second, third, and fourth tests.

And here’s what we think they mean:

Creative testing 2.0 recommendations:

Given the above results, those of us testing using IPM have an opportunity to re-test IPM winners that exclude the control video to determine if we’ve been killing potential winners. As such, we recommend the following three phase testing plan.

Creative testing phase 1: Initial IPM test

Creative testing phase 2: Initial ROAS test

Creative testing phase 3: ROAS scale test

We’re still testing many of our assumptions and non-standard practices

We look forward to hearing how you’re testing and to sharing more of what we uncover soon.

Brian Bowman is the founder and CEO of ConsumerAcquisition. He is an elite mobile performance marketer with expertise in user acquisition, general management and product strategy. His experience spans start-up and large company environments with a track record of navigating online businesses to profitability and high-growth. Brian has profitably managed over $1B in online advertising spend and product development for leading online brands including Disney, ABC, and Yahoo!.

Exit mobile version