5 Strategies for Improved A/B Testing

Don't miss these opportunities to improve revenue or alter user behavior.

In a metrics-driven culture, product developers and marketers often become too reliant on A/B testing. This clogs the test pipeline with non-needle moving initiatives, unnecessary validations of well-known design/product principles, or ISO-testing features that couldn’t be launched without other bundled features. In essence, companies over-test, test incorrectly, and don’t look for opportunities to streamline the testing process.

During my 10+ years working at some of the world’s top search companies, I conducted many A/B tests both on the user interface (UI) as well as back-end features. Given the usage and revenue associated with a single search page, we were careful to test almost everything. We recognized there were many small changes that could alter user behavior or revenue significantly that were counter-intuitive. One surprising example was changing the bullets in front of the sponsored links from square to round. This minor change impacted click-through rate on the search ads significantly, leading to a multi-million dollar change in search revenue. Through my years, I’ve learned several strategies making the test process speedier, more streamlined, and better optimized to increase revenue.

1. Be aggressive with the first test to gauge the level of potential lift.

With the first test, ensure users will see the change with an aggressive version or call out of the feature. This will quickly determine whether the change will significantly move the metric that you are aiming to impact. If an extremely large or highlighted version does not move the needle, a tamer implementation will more than likely not. For instance, strip a lead form nearly bare rather than see if one form field makes a difference.

2. Don’t use A/B testing in place of design best practices or product sense.

Those in charge of product development can become too reliant on A/B tests: they allow their product sensibility to take a back seat and allow test results to make every decision for them. This results in product people testing every possible variation and clogging up the test pipeline. An example was when we were integrating image results on the search page. A product manager wanted to test four or five variations of the same test, using up many of our test slots for that test cycle. Some of the variations that were proposed for testing were three large images versus five smaller images versus four medium images. General design principles tell us that people like to view items in groupings of three or five and our product intuition tells us that people like larger images rather than smaller images.

3. Test your competitors’ user interface changes to understand their impact, especially if they have a metrics or big A/B testing culture.

Your competitors are testing new features and UI changes all the time. It is a best practice from a competitive intelligence and a product understanding perspective to test these changes and differences between your site’s UI and the competitions. This will uncover metrics changes that may be caused by these different implementations. Of course, this assumes your competitors are competent and are using a reliable A/B testing framework.

4. Use isolation testing on individual features but also try some big jumps and combinations.

It is a best practice to try isolation (ISO) testing; that is testing individual features in isolation. But oftentimes when three features are bundled, the features are first ISO tested and then combo tested. If the features are supposed to be bundled together, try a combo first or test the ISO and the combo at the same time. This saves a lot of time and test slots.

5. Test identical buckets against each other to gauge significance and see if your systems are working correctly.

Often times, as product developers, we wonder if our bucket results are moving our product in the right direction or if it’s sending us a strong enough signal to validate whether the product change is positive or not. How do we know if something’s wrong with our test system? It’s best practice to occasionally run identical buckets against each other and look at the variances between the two to see if there are issues or see if there is significant variance between identical buckets due to sample size or other factors.

While A/B testing and data analysis should be a fundamental part of your product development process for Internet products and services, we all need to step back and think how we can speed up our product development and reduce the bottlenecks in launching new products and feature sets. The number of A/B tests that can run concurrently on a website is constrained by:

  • The number of statistically significant tests that can be run with a particular site’s audience size.
  • Data collection and analysis work involved.
  • Potential disruption in user experience.

It’s essential these limited test slots are used in the most efficient manner and these common pitfalls are avoided.

Subscribe to get your daily business insights

Whitepapers

US Mobile Streaming Behavior
Whitepaper | Mobile

US Mobile Streaming Behavior

5y

US Mobile Streaming Behavior

Streaming has become a staple of US media-viewing habits. Streaming video, however, still comes with a variety of pesky frustrations that viewers are ...

View resource
Winning the Data Game: Digital Analytics Tactics for Media Groups
Whitepaper | Analyzing Customer Data

Winning the Data Game: Digital Analytics Tactics for Media Groups

5y

Winning the Data Game: Digital Analytics Tactics f...

Data is the lifeblood of so many companies today. You need more of it, all of which at higher quality, and all the meanwhile being compliant with data...

View resource
Learning to win the talent war: how digital marketing can develop its people
Whitepaper | Digital Marketing

Learning to win the talent war: how digital marketing can develop its peopl...

2y

Learning to win the talent war: how digital market...

This report documents the findings of a Fireside chat held by ClickZ in the first quarter of 2022. It provides expert insight on how companies can ret...

View resource
Engagement To Empowerment - Winning in Today's Experience Economy
Report | Digital Transformation

Engagement To Empowerment - Winning in Today's Experience Economy

1m

Engagement To Empowerment - Winning in Today's Exp...

Customers decide fast, influenced by only 2.5 touchpoints – globally! Make sure your brand shines in those critical moments. Read More...

View resource