Indexing and Ranking: The Chicken or the Egg?

After my little sabbatical from corporate life, I decided to start from scratch and launch a new brand. All the business and potential business coming my way at the moment is by referral. I’m very lucky in that sense, as I hate knocking on doors. But my greatest wish is that those people who are referred don’t immediately burst into incessant waffling about the technologies used to build their sites, search engine spiders, meta tags, and PageRank, ad infinitum.

I often find myself talking to people who still believe ranking at search engines is all about some technical voodoo done to Web pages. I try to explain doing some basic tech-tweaking to Web pages to get them indexed isn’t difficult, but getting a decent top 10 rank is. And there’s this kind of astonishment from them when they discover that instead of instantly wanting to talk about technology, I first want to discuss marketing.

There’s a little discipline I have: I don’t take on no-hopers. The mere thought of failure makes me queasy. So talking to anyone who seems to believe that ranking at search engines is still something you achieve via a technical process, I’m sorry, but get real!

I’ve had to tool up my new business. Some technology I’m having custom-built, some I’ve bought off the shelf. I’ve got keyword analysis tools, link analysis tools, a vertical directory plotter, and a link-gathering bot. I’ve got an XML sitemap tool. I’ve got bid management and analysis software. I’ve even got a ranking tool for those poor souls who insist on having one. In fact, I’ve got pretty much everything your average SEO (define)/SEM (define) firm has. (It really doesn’t cost much to set up an SEO/SEM shop, does it?)

But when I look at all this technology, I can say of none of it, “Without this tool, I’d never get my clients ranked at search engines.” I rely on my marketing skills for that, not technology.

Link building is a marketing exercise. It’s critical for ranking at the major search engines. I often say at conferences, “Don’t be a link collector. Be a business developer.” I tend to consider business and marketing objectives when it comes to linking.

Critical as link building is, it’s still only one important factor in the ranking mix. True, links infer authority when it comes to ranking. But does authority mean quality?

Quality is a subjective thing, and search engine end users are better placed to judge quality than a link-based relevancy algorithm. Two Web pages written on a popular symphony may be completely relevant to a query. But the page written by a first-year music student may vary immensely in information quality compared to the one written by an accomplished orchestra conductor.

This is one reason end-user- behavior data and machine-learning techniques are becoming so important. Last year, I conducted a lot of research into Microsoft’s RankNet approach. Moving beyond PageRank and dynamic ranking in favor of machine learning is a big research area.

I’ve found nothing specific that says search engines are definitely looking at the following, but it’s mentioned in the literature. There’s a lot of implicit information you can get from simply monitoring a Web page’s popularity. Over time, you can measure how often it’s visited from toolbar data, proxy logs, and click-through data. How long do people stay on particular pages? Do they hit the “Back” button after a period, or do they leave via a link?

When it comes to ranking, we’re used to hearing Google say that over 100 factors are involved (I believe it moved up to 200 at some point). And Microsoft has gone on record to say that RankNet calculates 600 features for each document.

The bigger a search engine index gets, the less it can rely on pure linkage data and text to rank documents. And the harder it becomes for us to meet the new ranking criteria. That is, unless we become smarter marketers.

A friend of mine appeared on breakfast TV in the U.K. every morning last week, promoting his new self-help product. We run a Web site together that sells his products. I’ve been watching the referrer data in the site logs like a hawk. The very minute he finished one of his appearances, thousands of visitors hit the site. And 90 percent of them came through search: searches for his name, his products, and generic terms.

As people poured over his personal development and self-help products, they stayed on the pages for up to 14 minutes and usually bought two products. That kind of data doesn’t go unnoticed by search engines.

I’ve checked the personal development space both on- and offline. It’s a hugely competitive area with all types of lifestyle gurus who make you feel better about yourself, help you get rich, blah, blah, blah. But if they want to get search traffic in the kind of quantities my buddy gets, they’ll have to advertise and promote themselves at the same level he does.

That’s why I think there’s a bit of a chicken-and-egg situation that goes on sometimes with indexing and ranking. Should you worry about the crawlability of your Web site before you have an inkling of whether you stand a cat’s chance in hell of ever being able to compete and rank up with the big dogs?

Join us for Search Engine Strategies in London, February 13-15, at ExCel London.

Want more search information? ClickZ SEM Archives contain all our search columns, organized by topic.

Related reading

Brand Top Level Domains