Choosing Your Perfect Software Match, Part 3

This is the final installment in a series based on a discussion I had with Matthew Berk, senior analyst for Jupiter Research, which shares its parent company with ClickZ. He’s a former IT guy who is passionate about site analytics and empowering marketers with the tools to make good decisions.

In part one we learned about “Berk’s list” — the list of software and solutions he would use to run his next Web site. In part two, Berk began to name names for us. We learned his picks for best of breed in the first three of six Web measurement tool categories.

Today we resume our conversation with category four, site quality measurement.

Berk: After panel-based usability testing, I have a category called “site quality measurement.” These are tools that give marketers visibility into the scope, structure, and contents of their own sites, to know what’s out there. Most marketers have no idea how many pages their sites really have. I have one client who happened to “discover” that the company owned over 10,000 domain names, many of which were forgotten but still in use, and not updated or managed by the company’s Web operations group! It’s critical to know what’s out there, accessible by the public — your customers — and to be able to enforce quality and accuracy.

Smith: These sound like quality assurance (QA) issues. Is this a tool for the QA team?

Berk: No. Marketers also need to know what’s out there. Top on my list for this category is Watchfire. I’m also impressed with Maxamine for pure visibility, but Watchfire is hands down my favorite quality monitoring tool. These tools will go through your site and look for broken links, profanity, content that should — and should not — be there. You can eradicate links to porn sites, should they exist — and they do, even in the most unlikely places — or verify that every page of the site links to the privacy policy.

Next on the list is site availability. I call it “geek works.” But it’s useful to marketers, too. Site uptime is a typical example. It’s like I said about conversion rates [see part two –Ed.] — in the aggregate, it tells you nothing. You need to know specifically where and when problems occur for customers trying to reach the site. What if there’s network congestion that’s keeping people from getting to your site in a particular geographic area? The overall uptime rate may hide that. But marketers want to know that their sites are performing as expected at all times, all around the world.

Smith: What vendors are on this list?

Berk: Empirix, Mercury Interactive/Freshwater, and Keynote. I’d ideally select Keynote to keep my hosting and network providers honest and Freshwater to stay on top of internal issues.

Smith: That leaves us with one more category…

Berk: Webwide traffic data. In this category, I’d choose Nielsen//NetRatings [and] comScore Media Metrix. HitWise will eventually compete, but it needs to work on its data.

I’m not sold on this category, but marketers I’ve worked with love the data for competitive research. As long as you don’t worry about accuracy — which simply isn’t possible from any of these vendors because of the sampling and extrapolation methodology — and rely on the data comparatively or directionally, it can be helpful to see what else is happening in your industry. Still, I’m not convinced of the ROI [return on investment] here at all. But I include it on the list because there are situations, like in publishing, where this type of measurement can really help with setting rate cards.

Smith: I guess this just seems… well… who is really going to buy this many reporting tools? I had to beg and plead just for Brio Report Writer years ago. I gave up the fight a long time ago and started writing my own SQL code and maintaining my own database.

Berk: Familiar story. You did what you had to do to, to make the decisions you needed to make. But I have to say, this isn’t about reporting tools. Measurement is not about reporting. It’s about optimization. It’s about enabling marketers to make well-informed decisions.

Smith: But what about the cost? Who can afford this many tools, other than the largest companies?

Berk: That’s why my list is prioritized, with site analytics in the number-one spot. In the end, you need to justify the expense of these tools based on the incremental value they will allow you to wring from your site. What do you hope to optimize? What is the dollar benefit of that optimization, and does it justify the software cost? If the vendor can’t help you optimize, why are you considering it?

Smith: OK, you’ve given me a lot of information, but I have to bring up one more thing. In your report on site analytics, you say using an ASP solution is now an industry best practice. I hate the thought of someone else having my data! That scares me.

Berk: Isn’t that what IT represents to most marketers right now? For most of them, IT controls the data. But the IT group has no financial incentive to give you what you want or need. The marketers are the IT group’s internal customers, but they often aren’t treated that way. An ASP has every financial incentive to get you — its paying customer — what you need. Forget for a moment the increased accuracy, cost-efficiency, and proven value of an ASP-based site analytics tool; if you’re not on the same page with your IT group, you definitely need to be thinking ASP, end of story.

End of story, indeed! I got three columns worth of information from one brief phone call. I hope it’s been as helpful to you as it was to me. Finally, someone who understands us — and he’s a former CTO, if you can believe that. I dream of the day I’ll have these tools and achieve the nirvana Matthew strives for: to marry the 5 Ss of site analytics (source, stream/scenario, segment, self, and score) so we can all make better decisions and get on with optimizing our sites — and our ROIs.