Beyond Analytics: Session Instant Reply

• Tweet
• Buffer

Here's the basic scenario: Customers call your help center because something they tried to do online didn't work. They know the task they were trying to complete but probably can't remember (accurately) every detail about their sessions. A few call customer service and the rest go off to other Web sites that worked properly.

Where's the problem? Was it human error? Did the site actually break down? Did the design of the feature mislead the user into clicking the wrong thing, or -- worse -- confuse the user because of a poorly designed interface? It's anyone's guess. More important, without fully understanding the problem, how can your company fix it?

Enter the world of the instant replay. Some technology firms have entered the arena to solve problems like this. The instant replays enable solutions to ask much further-reaching questions as well.

The Basics

These technologies work by watching Web traffic on your site and storing every interaction for every user session. Tons of security checkpoints are in place to make sure this isn't a scary idea. In the above scenario, the help desk operator is able to load up the session in question and step through it page by page, watching what was typed and what was clicked. That way she (or the engineers) can really understand if the problem was human error, a logic error, or a design error. She can also see what other sessions followed the same pathways and had the same errors to gauge the problem's size.

Beyond Programming Errors

While this technology is crucial for finding real-time logic errors, there are other extremely interesting uses. In our business, we routinely do exhaustive usability studies to understand how new features or designs affect customer behavior and associated revenue. We also perform usability assessments at regular intervals to help companies benchmark how they're doing. An instant replay tool can speed up the process of both of these.

When a company wants to launch a new feature on its site, it routinely gets test users in a room and watches what how they interact with the site. What did they do? Was the feature intuitive? Did they try to enter something the company wasn't expecting? It can be a long process.

By performing these tests on a system enabled with instant replay, this process can scale up very quickly, and aggregate results can be seen quickly. How many users didn't complete the task? Was there anything in common between these users' interactions that could point to a flaw in the system (in either design or logic)?

Not only is this useful in a testing environment, but it is also useful in a live environment, as features get rolled out to users. Every new feature usually goes through a soft launch phase, during which a percentage of users get access to the new beta site. A close analysis of the errant sessions can result in a much swifter launch schedule.

Converging Traditional Analytics

The granularity provided by companies offering instant replay is extremely useful in many situations. A higher-level view (such as those provided by traditional analytic companies like Coremetrics and Omniture) is also essential. At the moment, these two classes of applications are striving to answer different business questions. As business needs evolve, expect to see these packages evolve as well. Eventually we will want to start with the highest-level view, then dig down to an individual session (and watch it step by step).

We will also want to start at the individual session and expand our view up high enough to understand the relative importance of our finds across the entire site and relate them to high-level KPIs (define). Look in the next few years for these worlds to converge.

Conclusion

Instant replay analytics provide unprecedented views into consumer behavior. They also aid in debugging the logic on sites and in understanding the bottom-line value of site features. They also make usability testing much more efficient. Combined with traditional analytic packages, these tools provide a wealth of useful data about how customers use your site. As more companies get on board with tools like these, we will hopefully see a drastic reduction in development time and an increase in site usability.

Questions, thoughts, comments? Let me know.

Until next time...

Jack