Indexing and Ranking: The Chicken or the Egg?

  |  January 22, 2007   |  Comments

Should you worry about your site's crawlability before you worry about its ranking?

After my little sabbatical from corporate life, I decided to start from scratch and launch a new brand. All the business and potential business coming my way at the moment is by referral. I'm very lucky in that sense, as I hate knocking on doors. But my greatest wish is that those people who are referred don't immediately burst into incessant waffling about the technologies used to build their sites, search engine spiders, meta tags, and PageRank, ad infinitum.

I often find myself talking to people who still believe ranking at search engines is all about some technical voodoo done to Web pages. I try to explain doing some basic tech-tweaking to Web pages to get them indexed isn't difficult, but getting a decent top 10 rank is. And there's this kind of astonishment from them when they discover that instead of instantly wanting to talk about technology, I first want to discuss marketing.

There's a little discipline I have: I don't take on no-hopers. The mere thought of failure makes me queasy. So talking to anyone who seems to believe that ranking at search engines is still something you achieve via a technical process, I'm sorry, but get real!

I've had to tool up my new business. Some technology I'm having custom-built, some I've bought off the shelf. I've got keyword analysis tools, link analysis tools, a vertical directory plotter, and a link-gathering bot. I've got an XML sitemap tool. I've got bid management and analysis software. I've even got a ranking tool for those poor souls who insist on having one. In fact, I've got pretty much everything your average SEO (define)/SEM (define) firm has. (It really doesn't cost much to set up an SEO/SEM shop, does it?)

But when I look at all this technology, I can say of none of it, "Without this tool, I'd never get my clients ranked at search engines." I rely on my marketing skills for that, not technology.

Link building is a marketing exercise. It's critical for ranking at the major search engines. I often say at conferences, "Don't be a link collector. Be a business developer." I tend to consider business and marketing objectives when it comes to linking.

Critical as link building is, it's still only one important factor in the ranking mix. True, links infer authority when it comes to ranking. But does authority mean quality?

Quality is a subjective thing, and search engine end users are better placed to judge quality than a link-based relevancy algorithm. Two Web pages written on a popular symphony may be completely relevant to a query. But the page written by a first-year music student may vary immensely in information quality compared to the one written by an accomplished orchestra conductor.

This is one reason end-user- behavior data and machine-learning techniques are becoming so important. Last year, I conducted a lot of research into Microsoft's RankNet approach. Moving beyond PageRank and dynamic ranking in favor of machine learning is a big research area.

I've found nothing specific that says search engines are definitely looking at the following, but it's mentioned in the literature. There's a lot of implicit information you can get from simply monitoring a Web page's popularity. Over time, you can measure how often it's visited from toolbar data, proxy logs, and click-through data. How long do people stay on particular pages? Do they hit the "Back" button after a period, or do they leave via a link?

When it comes to ranking, we're used to hearing Google say that over 100 factors are involved (I believe it moved up to 200 at some point). And Microsoft has gone on record to say that RankNet calculates 600 features for each document.

The bigger a search engine index gets, the less it can rely on pure linkage data and text to rank documents. And the harder it becomes for us to meet the new ranking criteria. That is, unless we become smarter marketers.

A friend of mine appeared on breakfast TV in the U.K. every morning last week, promoting his new self-help product. We run a Web site together that sells his products. I've been watching the referrer data in the site logs like a hawk. The very minute he finished one of his appearances, thousands of visitors hit the site. And 90 percent of them came through search: searches for his name, his products, and generic terms.

As people poured over his personal development and self-help products, they stayed on the pages for up to 14 minutes and usually bought two products. That kind of data doesn't go unnoticed by search engines.

I've checked the personal development space both on- and offline. It's a hugely competitive area with all types of lifestyle gurus who make you feel better about yourself, help you get rich, blah, blah, blah. But if they want to get search traffic in the kind of quantities my buddy gets, they'll have to advertise and promote themselves at the same level he does.

That's why I think there's a bit of a chicken-and-egg situation that goes on sometimes with indexing and ranking. Should you worry about the crawlability of your Web site before you have an inkling of whether you stand a cat's chance in hell of ever being able to compete and rank up with the big dogs?

Join us for Search Engine Strategies in London, February 13-15, at ExCel London.

Want more search information? ClickZ SEM Archives contain all our search columns, organized by topic.


Mike Grehan

Mike Grehan is currently chief marketing officer and managing director at Acronym, where he is responsible for directing thought leadership programs and cross-platform marketing initiatives, as well as developing new, innovative content marketing campaigns.

Prior to joining Acronym, Grehan was group publishing director at Incisive Media, publisher of Search Engine Watch and ClickZ, and producer of the SES international conference series. Previously, he worked as a search marketing consultant with a number of international agencies handling global clients such as SAP and Motorola. Recognized as a leading search marketing expert, Grehan came online in 1995 and is the author of numerous books and white papers on the subject and is currently in the process of writing his new book From Search to Social: Marketing to the Connected Consumer to be published by Wiley later in 2014.

In March 2010 he was elected to SEMPO's board of directors and after a year as vice president he then served two years as president and is now the current chairman.

COMMENTSCommenting policy

comments powered by Disqus

ClickZ Today is our #1 newsletter.
Get a daily dose of digital marketing.



Featured White Papers

2015 Holiday Email Guide

2015 Holiday Email Guide
The holidays are just around the corner. Download this whitepaper to find out how to create successful holiday email campaigns that drive engagement and revenue.

Three Ways to Make Your Big Data More Valuable

Three Ways to Make Your Big Data More Valuable
Big data holds a lot of promise for marketers, but are marketers ready to make the most of it to drive better business decisions and improve ROI? This study looks at the hidden challenges modern marketers face when trying to put big data to use.