How to Fail at Landing Page Testing

  |  July 23, 2013   |  Comments

If your data rate is too low, your website experiences seasonal spikes in traffic, or your website has major issues, you may be doomed to failure.

In my last column, I wrote about some of the ridiculous excuses we often hear from companies for not starting a conversion optimization program. You should obviously try to improve your conversion rate. But in many people's minds there is a mistaken perception that landing page testing is the only tool available. The fact is, there are many situations where testing is just plain wrong, and probably doomed to failure:

  • If your data rate is too low
  • If your website experiences seasonal spikes in traffic
  • If your website has major issues

If Your Data Rate Is Too Low

Say your site is fairly new and you're only seeing a few conversions per day. You're beginning to panic because you have a low conversion rate and your marketing person tells you that you should be testing so you can optimize for conversions. Unfortunately, this is one situation where your tests would most likely be inconclusive.


Testing is built on probability and statistics. So we recommend at least 10 conversion actions per day in order to conduct even a basic A/B split test. Below that level you will have too much "fuzz" and variability in your answer and will probably be unable to reach conclusive results. Or your test can stretch out over many months of data collection, leading to problems with seasonal changes in your audience's behavior.

One way to deal with a low data rate is to change what you consider to be the conversion action. If you don't have enough data to test the bottom-of-the funnel macro conversion (e.g., the sales in an e-commerce setting), go up the conversion path to find more granular and frequently appearing micro conversions, which can still ultimately impact the bottom line. For example, you can measure the number of adds-to-cart instead. These happen more frequently, and serve as a proxy for the actual down-funnel conversion that you want to measure (completed sales at the end of the checkout).

If Your Website Experiences Seasonal Spikes in Traffic

It's a month before Mother's Day, you're in the flower business, and your test-happy team wants to dust off one of those wish list items that's been sitting in the inbox for months, starting with an important landing page on your site. Just say "no"!

Testing is based on the assumption that you have a homogenous group coming to your site. If you can predict how a subset of that group acts, you can generalize that to how the rest of the users are going to act. But businesses that face seasonal shopping trends have a very different traffic mix during peak season.


For example, if you're in the flower business and a month out from Mother's Day new visitors to your site are likely price shopping, and that's the only criteria they care about: your price. These are the people who plan ahead (and probably buy their Christmas present in July as well).

Two days out from Mother's Day, the visitors who show up to your site are in a panic. These visitors are looking for express shipping and they do not care about paying extra for the flowers or the shipping. They just want to make sure that Mom gets them on time. These procrastinators (myself included) are part of a very different group.

The point is, visitors in the months leading up to the holiday are going to behave differently. If you tested for price sensitivity two months out, you'd see there was a lot of it; if you tested days out, you wouldn't see much because visitors care more about availability.

Very spiky seasonality complicates your tests. To adjust for this, you might tune your CRO for the short-term seasonal period - and only apply your findings within the same period the following year. Or you might conversely filter out any short-term peak-season impacts and treat them as anomalies, and tune for the baseline behavior throughout the rest of the off-peak year.

Here are some tips for dealing with seasonal business and testing:

  • Choose recurring, controllable, and stable traffic sources.
  • Don't change your marketing mix during the data collection.
  • Plan data collection around any major product or company announcements.
  • Average across obvious sampling biases (day-of-week, time-of-day).
  • Schedule around any known seasonal trends or important industry events.
  • Remove or mitigate data from unexpected external events.
  • Continue ongoing low-level data collection on the original recipe even after the test is completed.

If Your Website Has Major Issues

You've been testing and testing and not getting any conclusive results. You're trying to pinpoint the problem but can't seem to figure out what's going wrong. But the problem may be bigger than any one of your tests. The problem is your whole website experience.

If your site has major fundamental problems with its navigation, design, or usability, testing piece by piece is very inefficient because what the site really needs is to be rebuilt and redesigned from the ground up.

If that sounds extreme, think about the fact that conversions don't just happen in a bubble. You want your site to offer a seamless experience for users overall. If you have a crappy site, putting Band-Aids on landing pages here and there is not the answer.

When visitors are meandering around your site without clear purpose or direction, commonly using the back button, or relying on the search function over navigation, you probably have a website problem.

The overall goals of web usability are:

  • Decrease the time it takes for visitors to finish tasks.
  • Reduce the number of mistakes visitors are likely to make.
  • Shorten the visitor's learning time overall.
  • Improve the visitor's satisfaction.

Your site's major issues can be remedied by focusing on three key areas:

  1. The visual presentation of your site, including page layout, graphics, color, buttons, and the decision to use on-page animation or video.
  2. Information architecture, including content categories and how easy it is for visitors to understand how the site is organized so they can quickly find what they are looking for.
  3. The content and how it's written, making sure it's free of jargon, formatted for readers on the web, and provides the information visitors seek.

So the next time someone tells you that you need to test that, keep in mind not all environments are sufficiently testable. If you work to understand the big picture, then you can often find alternatives to test that lead to a shared goal, or you can figure out how to best deal with data that can throw your results.

Image on home page via Shutterstock.

ClickZ Live New York Want to learn more?
Attend ClickZ Live New York March 30 - April 1. With over 15 years' experience delivering industry-leading events, ClickZ Live brings together over 60 expert speakers to offer an action-packed, educationally-focused agenda covering all aspects of digital marketing. Register today!


Tim Ash

Tim Ash is CEO of SiteTuners.com, a landing page optimization firm that offers conversion consulting, full-service guaranteed-improvement tests, and software tools to improve conversion rates. SiteTuners' AttentionWizard.com visual attention prediction tool can be used on a landing page screenshot or mock-up to quickly identify major conversion issues. He has worked with Google, Facebook, American Express, CBS, Sony Music, Universal Studios, Verizon Wireless, Texas Instruments, and Coach.

Tim is a highly-regarded presenter at SES, eMetrics, PPC Summit, Affiliate Summit, PubCon, Affiliate Conference, and LeadsCon. He is the chairperson of ConversionConference.com, the first conference focused on improving online conversions. A columnist for several publications including ClickZ, he's host of the weekly Landing Page Optimization show and podcast on WebmasterRadio.fm. His columns can be found in the Search Engine Watch archive.

He received his B.S. and M.S. during his Ph.D. studies at UC San Diego. Tim is the author of the bestselling book, "Landing Page Optimization."

Connect with Tim on Google+.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Analytics newsletter delivered to you. Subscribe today!




Featured White Papers

A Buyer's Guide to Affiliate Management Software

A Buyer's Guide to Affiliate Management Software
Manage your performance marketing with the right solution. Choose a platform that will mutually empower advertisers and media partners!

Google My Business Listings Demystified

Google My Business Listings Demystified
To help brands control how they appear online, Google has developed a new offering: Google My Business Locations. This whitepaper helps marketers understand how to use this powerful new tool.


    • Website Optimizer - SEO, CRO, Analytics
      Website Optimizer - SEO, CRO, Analytics (Marcel Digital) - ChicagoMarcel Digital, an award winning interactive marketing agency established in 2003...
    • Director of Marketing
      Director of Marketing (Patron Technology) - New YorkDirector of Marketing We are seeking a Director of Marketing to manage and build our marketing...
    • Senior Interactive Producer
      Senior Interactive Producer (Ready Set Rocket) - New YorkWhat You'll Do As a member of our team, the Senior Producer reports directly to our...