Evaluating Search Results and Protecting the Innocent

One of the many interesting things about speaking at a lot of search (and related) conferences is getting the measure of where people are in the learning curve. A lot depends on the person who’s been allocated the job of becoming the in-house SEO (define) guru. I’ve discovered many people who have other responsibilities within an organization are suddenly thrust into the world of SEO conferences and told, “We want you to become our search marketing expert.”

I feel sorry for a lot of the attendees I speak with. They’ve been told by their bosses to go to a conference and find out why their competitors always come above them on SERPs (define). And that’s it. Then they simply have to go back to the office and repeat the formula. After all, it’s only about putting keywords in special places on a page, isn’t it?Relevant information retrieval (IR) on the Web is a much more complex task than that same retrieval is in a homogenous environment, such as a digital library. The late Gerard Salton’s text retrieval work and his development of the vector space model worked perfectly in that homogenous environment. But on the Web, a heterogeneous environment in which anyone can publish content of any nature or quality, relying purely on a Web page’s text as a clue to the page’s relevance doesn’t quite cut it.Back in the day, when search engines applied the vector space model as the primary factor in identifying relevant pages, keyword stuffing was just about all you needed to rank in the top 10. Of course, by the time Jon Kleinberg and Google guys Larry Page and Sergey Brin had developed hyperlink-based algorithms, things became a little trickier. No longer did old chestnuts work anymore, like creating tiny text stuffed on the page in the same color as the background. The priority was all about obtaining in-bound links.

Getting links from authority sites became the order of the day. It was all about link quality, not just quantity. Of course, you now had to get your head around the hubs and authorities idea Kleinberg came up with and, fundamentally, what PageRank is all about.

It gets worse. Just as you think you’re getting a clearer picture, someone like me writes a column telling you why to ignore PageRank and even Kleinberg’s HITS (define), if it comes to that. You see, both algorithms provide you with what link-based ranking is theoretically about. But neither actually work in the practical sense. Some shortcuts have to be taken.

All of a sudden, the newbie at a conference is thrown into an entirely different world of science and disciplines. It comes as quite a shock sometimes, when I explain to conference attendees, that it’s not just about downloading hundreds (or thousands or even hundreds of thousands) of pages and analyzing where the keywords are. And no, it’s not just analyzing who links to your competitors and trying to get some link parity and neat anchor text.

There are so many factors to consider, in fact, that it’s difficult to know where to look. Beyond the textual data carried in documents and the link structure surrounding them, there’s some very complex data to factor in: the end user.

I’ve written frequently about how this will become more of a major ranking factor. It’s still the one that’s most intriguing to me (as from my own experience, I can tell it’s already being folded in). It’s also why I’ve said many times textbook SEO is a dying art that will soon outlive its usefulness.

I get IR with text and links. There’s been a ton written about IR and network theory applied to the Web. My own library is stacked to the ceiling with textbooks on both subjects. But it’s so much harder to get qualified information about end user behavior. he reason for that is it’s almost invisible to us in the SEM (define) industry.

On one side, we in the industry are the content creators and have some knowledge and control. On the other side, search engines and researchers strongly guard end-user data information.

Is it a waste of time for someone to come to a conference expecting to get some simple answers? Well, if you honestly believe getting a top five ranking at Google is something someone can explain to you with a few PowerPoint slides about words on a page and links, the answer is probably yes.

B whether you care about the underlying factors in ranking mechanisms at search engines or not (as I definitely do), look at your competitors’ overall marketing mix, not just their Web pages and linkage data. You’ll get many more clues about why they have better rankings than by just looking at what appears to your boss to be the obvious.

That’s the real reason I wrote this column: to protect the innocent. When your boss suggests you got to a conference and come back an expert, show him this column. Protect yourself by getting him up to the same speed as you before you hit the expense account.

If you think I make it sound like SEO could actually be a science, it’s not. Neither is knitting. But you probably couldn’t learn that in a day, either.

Meet Mike at Search Engine Strategies April 10-13 at the Hilton New York in New York City.

Want more search information? ClickZ SEM Archives contain all our search columns, organized by topic.