Beyond the Log Files
How to determine if you use the right metrics and tools to monitor progress.
How to determine if you use the right metrics and tools to monitor progress.
I received a lot of your input and feedback from my recent query about the analysis tools you use on the job. Most of you wrote about software or vendor-related tools: log file reporting methods, data-crunching software, and so on. I shared the answers with everyone.
A few wrote to remind us of the importance of other types of research: usability studies, customer satisfaction and other surveys, and focus groups, to name a few. I’ve discussed aspects of most of these in other columns. Often, these methods prove to be the missing link between your objectives and your performance metrics.
I’ve one last reader response I haven’t discussed, although it’s possibly the single most important data analysis factor. What is it?
An objective.
If you don’t start with a clear business plan, you have a handicap no amount of analysis can overcome.
Take an honest and objective look at what you want to accomplish. What ultimately determines if your site is successful? Don’t assume you know. Speak with your organization’s management team to ensure your visions are compatible.
Example (taking liberties by assuming the potential goals of a specific site):
The Georgia Department of Transportation’s (DOT’s) NaviGAtor site provides real-time metro Atlanta traffic information: accident locations, average speeds on major roads, and construction delays. Objective? At first glance, disseminating traffic information to drivers, probably to help plan their commute.
Look deeper. Why would the DOT want to help Atlanta drivers? Convenience?
If that’s the case, it’s important to know what percentage of Atlanta drivers use the site, how often, and if they find it helpful. But I suspect the overarching goal is to reduce pollution by reducing gridlock. If so, we’re looking at a whole different set of metrics — metrics not contained in the site’s log files.
See how a misunderstanding about objectives can lead to choosing the wrong metrics (and tools) to monitor progress?
It happens repeatedly. Site managers sift through log and transaction files and other usage data every day. They react to changes in number of hits, page load time, and session lengths. Those numbers have their uses, don’t get me wrong. But it highlights what many of you wrote in about: All the analysis tools and all the data in the world get you nowhere without a clear objective. A measurable goal. Don’t get so caught up with data and tools you forget that objective.
Within an organization, different divisions have different objectives. These determine the metrics that matter to them. Within divisions, individuals probably use differing metrics. An email marketer uses different stats than his PR counterpart, though some may overlap. The overall goal must be clear. Each department and each individual must understand how their objectives support that overall goal.
Let’s use an airline site as an example. Any will do. Here’s a link, if a visual helps. I’ll assume the site’s overall goal is profit: Shift sales and customer service from costly methods (travel agents, telephone sales) to the Web. Viewing the site as a whole, there’s a lot more going on than sales. Frequent flier info. Hotels and rental cars. Special offers. Within each of these departments or categories, a different person may be responsible for content and performance. Each uses different metrics to determine if her section meets its own objectives.
Imagine the potential for disaster if everyone’s not on the same page about the overall objective. If the special offers manager forgets the overall goal of increased ticket-sale profit, offers might not be appropriate. The hotel and rental car section manager might forget he works for an airline. Setting high-level objectives, then goals for individual departments, obviously comes before selecting metrics to monitor.
Internet newcomers assume online businesses monitor different statistics than “normal” businesses do. At one company, I faithfully evaluated revenue, new registrations, and repeat purchases every morning, only to have the CEO ask, “How many hits did we have yesterday?”
Don’t make the mistake of asking what information is available, then building from there. Start with your objective. Determine what metrics are critical to monitor progress. Then, figure out what data you need and what tools you’ll use. If you didn’t identify objectives before you designed your site, you have bigger problems that I can’t help you with.
For the record, you’re hearing this from someone who learned the hard way. Analysts love data. It’s easy to get overexcited about an abundance of data and all those reports you can create. Get back to objectives. Make sure everyone knows what they are, that individual department objectives support the overall objective, and then look for data.
Hint: You may have to look outside your log files.
Don’t forget to vote for your favorite marketing technology solutions!