There’s a lady who knows
All that glitters is gold
And she’s buying a page rank of seven…
Perhaps the wittiest example of marketers’ fascination with natural search marketing is encapsulated in that play on Led Zeppelin’s “Stairway To Heaven.” Mike Grehan, the noted UK search engine expert, delivered it in a speech at a recent conference to illustrate marketers’ almost absurd obsession with page rank.
Most marketers are familiar with the term and understand the importance of developing high-quality, relevant links that point back to their Web site to gain top rankings in crawler-based algorithmic search engines. (Virtually all search engines are crawler based, as opposed to directory-based, as Yahoo was at one time. The latter present directory matches edited by humans.)
Yet the definition of “relevant” is a matter of some confusion.
I received the following email this week from a “link building” services company:
I am contacting you about cross linking. I am interested in iprospect.com because it looks like it’s relevant to a site for which I am seeking links. The site is about high quality kitchen accessories and serving trays & platters… if you approve of the site, then the intention is to exchange links.
I’ll admit I do like to cook. But to save my life, I can’t figure out how my search engine marketing firm’s Web site is relevant to “kitchen accessories.”
The company that sent the message doesn’t understand how search engine algorithms evaluate links. As a result, their client cannot possibly benefit.
So if random links from random Web sites won’t help your ranking, what’s a marketer to do? Try human edited directories, vis-À-vis Yahoo and The Open Directory Project (ODP)?
But directories are dead, right? MSN said goodbye to LookSmart when recent “conclusive” tests at its U.K. site indicated purely algorithmic results (currently supplied by Inktomi) to be “more relevant,” according to MSN Product Manager Karen Redetzki.
It’s not that long since Yahoo bowed to the algorithmic superiority of its partner Google and replaced primary directory results with algorithmic.
Yet there’s a hidden advantage in directories like Yahoo’s, ODP’s and LookSmart’s. Algorithm-based search engines adore taxonomies.
The taxonomic, hierarchical approach developed by Srinija Srinivisan at Yahoo (and copied by many other Web directories), provides many semantic clues to relevance, in addition to categorization and classification data.
Structured information is a perfect starting point as a training set for crawling purposes and for classification.
Given a set of predefined categories, an algorithmic system can learn from a set of examples. Following link data surrounding specific categories provides more reliable semantic clues. Classifing pages on a purely textual basis is somewhat unreliable. Yes, there are meta tags, but the marketers often stuff these tags with keywords not relevant to the content on their pages.
Exploiting link information in a small neighborhood around documents permits better categorization. The premise is that pages on the same topic tend to be linked to more frequently than those that are unrelated. A classification algorithm evaluates these relationships using precise statistical models. As a result, links from sites not within your “community” can be easily detected and discounted by most search engine algorithms.
Consider the tight relationship between Google and ODP. Note the directory category headings that frequently appear at the top of results. You’ll see how algorithms can heavily rely on the training data a directory structure provides. It’s been proved many times that if the categories of linked pages (to or from) are known, categorization accuracy can dramatically improve when using hyperlinks in conjunction with text evaluation.
Next year the major players, Yahoo, Google and AOL (powered by Google) — all algorithmic search engines — will be standing, along with a new contender, MSN. Yahoo owns Inktomi, so there’s already that perfect fit between a directory and a crawler. And Google has a tight bond with ODP.
Where does this leave a LookSmart-less MSN? Although MSN chose to pursue the Web equivalent of Boston’s “Big Dig” and create its own crawler index, it’s unlikely MSN will also throw a few hundred editors together to create a taxonomical and hierarchical directory to make sense of it all. All of a sudden, directory listings in ODP and Yahoo seem more important. Every major search engine is algorithmic and crawler-based. Virtually all look to human-edited directories for a taxonomic baseline.
If you look at Teoma and its underlying, fundamental resemblance to Kleinberg’s HITS algorithm, there is a way to tap into structured link data on a huge scale without a directory partner. But it’s neither easy nor cheap.
I predict algorithm-based search engines will always rely on directory or hub-type sites to a greater or lesser degree. Whether vertical or general directories, algorithmic search engines need directories to provide classification clues to deliver the most relevant results. That means marketers who want top rankings need to be listed in them. Period.
Information retrieval scientists involved in Web structural analysis have long been able to identify high-level forms of structure — Web communities — based on link topology. In simple-speak, a Web community is a collection of Web pages where each member page has more hyperlinks (in either direction) within than outside of the community.
This explains why a link from your community counts more than a link from outside. Objectively, this means quality does count more than quantity when it comes to linking partners on the Web. So most Web sites need to submit to human-edited directories first, not pursue links from random Web sites.
Since Yahoo switched from primarily directory results to algorithmic results, many marketers have asked, “Why should we pay $299 to get into the directory at all?”
From a classification and categorization point of view, that’s simple. If you want to be found on all the right keyword searches, your site must first be listed in all the right directories. Pony up the $299 to get your site listed in Yahoo Repeat the process with ODP. With Microsoft readying their own new algorithmic search engine, it could be the smartest move you make this year.
There’s plenty of life left in those old-fashioned, human-powered directories. Directory listings impact a site’s visibility and qualified search traffic. Random inbound links have little, if any, impact.