Choosing Your Perfect Software Match, Part 1

This week I found my analytics soul mate.

You know how it is when you meet someone who’s as excited about analytics as you are? You’re enthusiastic about all the same things and can share dozens of war stories. That’s how it was when I spoke with Matthew Berk, senior analyst for Jupiter Research, which shares its parent company with ClickZ. His research focus includes site measurement tools, a subject near and dear to the hearts of this column’s readers. He’s a former IT guy, with his last role being CTO for edu.com, but don’t hold that against him. As I learned from our conversation, not all IT folks consider marketing the enemy.

Berk is passionate about site analytics and empowering marketers with the tools to make good decisions. He has a lot of experience with tactics and best practices, and he’s had to learn the hard way — in previous lives — by being burned by expensive software that didn’t live up to its claims.

As a result of his experience — and his current position — he keeps a running tab of software and solutions he would use in operating his next Web site. He refers to it as “Berk’s list.” Berk’s list is divided into six major categories. You’ve gotta love how these analysts think — I wish I were so organized. The categories are listed in order of importance to him:

1. Web site analytics tool
2. Survey tool
3. Panel-based testing tool and/or usability lab
4. Site quality measurement
5. Site availability measurement and monitoring
6. Webwide traffic data

Following is a synopsis of my interview, which has provided fodder for a three-part series. Berk will name his providers of choice in each category. Our conversation starts with Berk sharing some of the pains he’s experienced in the ever-changing world of Web site measurement tools.

Smith: I read your blog, and you make reference to “Berk’s list” — a list of analytics tools you’d use were money no object. ClickZ readers are dying for this kind of information.

Berk: (Laughs.) I got burned several times, purchasing what were then best-of-breed solutions that would supposedly give my marketers the information they needed to make decisions. The software was expensive to install and operate and provided very little by way of actionable data and, therefore, little true ROI [return on investment].

In one case, to make good on a site analytics vendor’s upfront promise of accurate visitor measurement, they asked us to spend another $50,000 to customize the application! At one point we armed the business with four separate tools to measure everything from traffic to transactions to customer data. But the decision makers could correlate none of the data. Despite having all the right measurement tools, none of them worked together in any way. I call this an “embarrassment of riches.”

When I speak to vendors as an analyst, I always adopt the perspective of a potential customer, and I am always thinking about whether or not I’d spend my budget on their services or software. Would the vendor truly help me arm decision makers with actionable data?

Smith: I’m so relieved to hear you were concerned with what the marketers needed! That’s definitely not been my experience with CTOs.

Berk: How else can the marketers make decisions? There are a couple of givens, for me, if, or maybe when, I operate another site. First of all, there are no “site operations” without properly implemented site analytics. If you don’t have the tools to make well-informed decisions, everything degenerates to decision by committee, politics, and hunch. You get a dozen people trying to make decisions about the home page based on whether or not they think the VPs will like it.

Smith: Been there… but I have to say usually when we made decisions with no data, it was because the people in technology didn’t think analytics were important.

Berk: That happens a lot, the battle between marketing and technology. The other given is that I’ll only consider tools that utilize client-side tagging.

Smith: One downside of being a practitioner is I don’t get to see all the tools out there. I only know the ones I’ve personally utilized. So I don’t know how much that narrows your list.

Berk: A lot.

Smith: So, are you going to name names for us?

Berk: Yes, I’ll name names. But first let me tell you what I consider ground zero for operating a Web site….

Sorry to cut this short, readers, but this column does have a word limit. In the next installment we’ll pick up at ground zero, and, yes, we’ll eventually get those names out of him. (Although I dread the email I’ll get from the vendors who aren’t listed!)