A Traffic Analysis Solution for Small Business: Part 2

OK, so now you know how your log files work. This week we’re going to discuss various traffic analysis techniques and make recommendations for your best traffic analysis solution. We’ll start with Java Script, tell you what to measure, and help you make the right decision.

Java Script

A big trend in traffic analysis is to put Java Script on individual pages. The Java Script can access all the same information you’d find in a log file, plus it can extract pieces of data from the web browser itself.

Specifically, some of the things it can tell you are screen resolution, screen color characteristics, and the type of plug-ins used by your web site visitors’ computer. This is particularly useful for deciding how to design your web pages. If most of your audience is looking at your site with a screen resolution of 640×480 and in 256 colors, there is no point in designing a site optimized for a 800×600 screen resolution and 16 million colors. Your pages will look very crappy to most of your site visitors.

Java Script tags are how services like HitMeter, HitBox, NetStats, Counted!, SuperStats and Web Trends Live work.

Of course, the obvious problem with the Java Script solution is that you have to put the freakin’ tag on every page. If you have a 1,000 pages on your site, that makes it a bit cumbersome to retrofit this solution. Plus the pay services are based on how much traffic you generate, so your monthly fee goes up as your site gets more popular. Yet, these solutions are easier to implement than a dedicated software package.

Internal Search Engines

Once when I was listening to Rick Boyce, former VP of advertising for Wired, talk about HotBot, he said something that stuck with me. He said that because Wired had a major search engine, it had a good snapshot of what people on the web were looking for, and used that information when deciding what content to offer next.

There is no reason why you couldn’t do the same thing in your little corner of the web. All you have to do is use a search engine on your site that logs and reports what your site visitors are searching on. The best way I’ve yet found to get this data is to use a search hosting (or remote search) service.

Companies offering search hosting services will index your site and give you access to those indexed results through a form you put on your site. These services are offered for free in exchange for allowing the company to put banner advertising on the search results page. There are also a whole bunch of these companies cropping up now that charge a monthly fee in lieu of the advertising.

The two I like the most are FreeFind and SearchButton. But you can investigate the major ones by visiting this page. It has a good listing of all the major search-hosting services.

Visitor-Solicited Information

Often we get stuck in this mindset that the only good way to understand what our site visitors want is to silently watch them. But sometimes the direct approach is best. Ask them what they like, what they don’t like and what they want. This is as simple as putting a request for feedback on every page and developing a form you ask your visitors to fill out.

What Should I Measure?

We’ve gone over a lot of information these last two weeks without my yet telling you the specific data you should be measuring. So here are the basic questions I think every site needs to be able to answer:

1. How many unique visits are you getting per month, and is it increasing or decreasing?
2. What pages are people looking at the most? And what are they looking at the least?
3. What sites are sending you the most traffic?
4. Through what page are most of your visitors coming into your site? (These are called “entry pages” by the log analysis software.)
5. Through what page are most of your visitors leaving your site? (These are called “exit pages” by the log analysis software.)
6. How many 404 errors (file not found) are you generating and what files can’t be found? (These are the broken/outdated links on your site.)
7. What search terms are people entering into the search engines to find you? (Often recorded with the referrer data in your log file.)
8. What web browser software and screen resolutions are your visitors using to view your site?
9. What search terms are people using in your internal search engine to find content within your site?

How you get to these stats is a tough call in terms of budgeting. You have to do what makes sense financially for you. But here are three ways to do it:

The Freebie Route Most hosting companies, as part of their basic hosting packages, already provide reports that give you items 1 to 7 on the above list. You can register with HitMeter and FreeFind to get items 8 and 9 for free. The downside is you can’t really configure any of this software to clean up your data; like, for example, visits generated by you updating your web pages. But at least you have a usable baseline of your traffic, and it doesn’t cost anything.

The Log Analysis Software Route This will give you items 1 to 7 plus the ability to filter artifacts polluting your data. And you’ll be able to do a whole lot of other interesting analysis. You’ll still need to use something like HitMeter and FreeFind to get items 8 and 9. The downside is good log analysis software starts at about $500, and you need a fairly meaty machine to run it on the faster the processor and the more memory you have, the better. WebTrends, a popular package, recommends a Pentium II machine with at least 128MB of RAM and 100MB of disk space.

The Java Script Route This will give you everything, except item 9. And it is relatively easy to set up: Put some code on all your pages and you’re done. But since pricing is based on your level of traffic, your monthly bill goes up as your site gets more popular. It is not unusual to pay $300 a month for a site generating 500,000 file accesses or more per month. That is fine as long as your revenue increases accordingly. But if it doesn’t, you’re stuck with a rising monthly cost.

I’m probably not an unusual small business owner in that I like my overhead costs to be fixed. So if it were me, I’d go the log analysis software route. Then, when my site is generating hundreds of thousands of dollars per month, I’d go the Java Script route because I’d be running into data processing issues. As our log files get bigger, we’ll need more computing power and data storage. And the costs of those things will likely exceed what sites like SuperStats can provide for a monthly fee.

BTW, here’s a couple general archives of log analysis software that you might find useful:

Open Directory Guide to Log Analysis Software

The CGI Resource Center – Logging Accesses and Statistics

So if you take my recommendation, that’s $500 off our budget of $2,860 remaining leaving us $2,360 for our marketing budget.

I’ll give you the details on that next week!