Three Tips to Keep Usability Testing on Track

Do you actively engage in usability testing for your new products or features? The point of this column isn’t to convince you that you should. If that’s an open issue for you, there are many great articles that should convince you this is important. Rather, my aim in this column is to list a few things to watch out for, based on a project we’re testing right now.

Test Internationally

Are you an international company? If so, make sure your tests are run internationally. The tests we’re working on right now were carried out in two countries, in two different languages. While many companies, to save money, believe “people are people,” we found definite differences in behavior across different countries. The trick is to then figure out what to do about it. We aren’t going to build a site for each country that’s a little different (based on their usage). But knowing what features each country liked, and how each country navigated the site gives us good insight into what features are universally understood, and which need to be made more obvious and easy to use. That way, we can still design one basic site for all users, but do so understanding which features needed some extra help to make usable universal. We also cut some features that were used only by a specific country, because it was clear that no one else really understood the feature.

Watch the Videos Yourself

Oftentimes, usability testing is outsourced to a third-party company. This is fine to do, with one major caveat. You need to make sure whatever company you use gives you access to the raw video of the users doing the tests.

This is very important for a couple of reasons. Usability testing is generally scenario or task-based. Your company writes out a bunch of scenarios and the testing company sees if users can complete the tasks. For example, “A man needs to find a graphics card that will work with his computer and his monitor.” This is the task, and the usability tests will report on where in the path the user got stuck, what he had wanted to do, and whether he was successful. That’s what the company you hire will report on, because that’s what you asked.

What they won’t generally report on are the little details that weren’t important to them (or to the task), but are vital for you. For example, the company who did our European tests for us came back with the information that the specific task was completed successfully by nine out of 10 users. But, when we watched the original videos, we saw that all the users were scrolling down the page because they expected to find the information we asked them to look for, below the fold. I know that seems odd. Everyone says, “Make sure vital information is above the fold.” But the information we were having them look for was not vital information to them, and so they immediately scrolled down to find it. Eventually, they saw that it was actually above the fold and completed the task. Seems odd, I know. But this is the point of usability testing.

While the usability company simply reports that the task was successful, they might not have noticed (or made a note of) the fact that users were scrolling in this odd way.

To be fair, some usability companies will report on things like this. But, there are a million things that we could learn by watching them, and it’s not in the scope of the company to report on all of them. They report on what you asked. For the rest of the information, you really need to watch the videos yourself and not just rely on their summary presentation.

Use Personas and Weigh Results

Usability testing should be done with a very clear sense of who your customers are. Tests in a vacuum don’t matter much. And while I’d love to believe all people are created equal or of equal value, they aren’t.

In our tests, we had an equal division of each persona. If we built our personas correctly, their usability results will be different. If they were the same, what would be the point of having different personas?

Having said that, you need to carry through the understanding of their differences all the way through to the results. One report we got back simply listed the top feedback. This wasn’t as important as the top feedback organized by persona. This is because our personas are all ranked. One is clearly the heavy buyer on the site. Another buys mainly in stores and not online. So, understanding where the feedback is coming from is important.

When you create action items based on the outcomes of the usability testing, make sure you clearly understand what changes will affect the user experiences of which personas. This will help you prioritize your work and ensure you’re working toward a more profitable site, not just a more usable one. I know that must seem like an odd statement. But, there are tons of really expensive (and cool) tools that one persona may want. But if she isn’t the persona that’s actually spending money on the site (and you don’t think that the new tools would encourage her to spend more money), then her feedback (and therefore the expensive new functionality) is less important than those who actually spend money with you.

Questions, thoughts, comments? Leave feedback below.

Until next time… Jack