Download this PDF, which covers the basics of the science of information retrieval on the web.
It’s been 10 years since I wrote the second edition of a book about search engines called, "Search Engine Marketing: The Essential Best Practice Guide." It was a very big seller and, in fact, it carried on selling through to the beginning of 2010 when I took it offline.
I've decided to start this year by revisiting the chapter in the book about how search engines work. I've said many times over the years that, most books about SEO have a section called how search engines work. But rarely (if ever) do they describe the interdisciplinary approach to information retrieval (IR) covering mathematics, computer science, library science, information architecture, cognitive psychology, linguistics, and statistics - to name but a few.
Previously, I had written mainly about methods of manipulating rankings by keyword stuffing and other black hat type techniques of the time. But as I began to realize the importance of linkage data and even more so, link anchor text, I became more and more inquisitive as to what it was exactly that search engines used in their ranking technologies.
After talking to one of the pioneers in web search (Brian Pinkerton of WebCrawler), I was introduced to the work of foremost information retrieval scientist, Gerard Salton. This was a major breakthrough for me.
Salton’s work was cited in just about every IR research paper I read at the time. So I tracked down and bought a copy of his seminal work "Modern Information Retrieval" (written back in 1983, however Salton’s work in the field goes back to the early 1970s).
As a marketer, not a scientist, this was no easy read. Yet, as I began to grasp the basic concepts and drivers behind information retrieval (and the way it is applied to the web), the more I was able to understand the major challenges involved. And that led me to change, not just the amateurish and spammy techniques I’d used previously, but to thinking about SEO in an entirely different way.
And to this day, I still firmly believe that a basic understanding of the science of information retrieval on the web goes a long way toward helping search marketers dispel myths and do their jobs more professionally and proficiently.
Of course, 10 years later my personal library has grown to include a very large section of information retrieval and data mining texts as more and more become available. This is also largely due to the fact that the subject matter is so fascinating it's hard not to become engrossed.
As I revisited the chapter I'd written on how search engines work from a decade ago, I expected it to be a bit stale, but it wasn't at all. Although, I dare say, to an IR scientist, if not stale it probably seems about as elementary as it gets. I wrote the chapter placing great emphasis on trying to make it non-mathematical. By that I mean highlighting concepts and background theory rather than matrices and formulae. That said, it's extremely hard to cover the subject without references to linear algebra and other mind-numbing math.
Anyway, if you're genuinely interested in how search engines work (but really, not the anecdotal stuff generally bandied around), then it's as good a place as any to start. I mention in the introduction that it is totally unchanged from the very quirky, very British flavor it had when it was first published. A few pages were eliminated purely because they were totally irrelevant a decade later. There are a few little gems in it that I'd forgotten about.
No, the subhead above isn't a typo or spelling mistake. It's actually a conversation.
When the French author Victor Hugo had "Les Miserables" published, he was not living in Paris at that time. He was waiting to hear news from his publisher about the kind of reception his new book was having. When he could wait for news no longer, he sent a letter to his publisher that contained only the character: ?
On receiving this, his publisher knew exactly what it meant and he returned a note to him containing only the character: ! This let Victor Hugo know that his book was a huge success. It is said that this is the shortest correspondence in history.
What's that got to do with anything? It was actually a good analogy I used relating to the length of the average query at search engines at the time and how difficult it is to deal with short queries.
I'm seriously thinking about trying to find the time to update the entire book this year and make it available free to Search Engine Watch and ClickZ subscribers. More recently I've been reading about a feature-centric view of information retrieval and also learning to rank for information retrieval and natural language processing (very hot research topics). I'll be writing a couple of follow-up columns covering these subjects combined with fascinating insights into the strength of end user data and, of course, weaving that into the update of the book once I get time to make a start.
But right now, feel free to download the PDF of "How Search Engines Work." If nothing else, I hope it acts as a very basic introduction.
This was first published on SearchEngineWatch.com.
This article was originally published on searchenginewatch.com.
Join the Industry's Leading eCommerce & Direct Marketing Experts in Chicago
ClickZ Live Chicago (Nov 3-6) will deliver over 50 sessions across 4 days and 10 individual tracks, including Data-Driven Marketing, Social, Mobile, Display, Search and Email. Check out the full agenda and register by Friday, Oct 3 to take advantage of Early Bird Rates!
Mike Grehan is Publisher of Search Engine Watch and ClickZ and Producer of the SES international conference series. He is the current president of global trade association SEMPO, having been elected to the board of directors in 2010.
Formerly, Mike worked as a search marketing consultant with a number of international agencies, handling such global clients as SAP and Motorola. Recognized as a leading search marketing expert, Mike came online in 1995 and is author of numerous books and white papers on the subject. He is currently in the process of writing his new book "From Search To Social: Marketing To The Connected Consumer" to be published by Wiley in 2013.
IBM Social Analytics: The Science Behind Social Media Marketing
80% of internet users say they prefer to connect with brands via Facebook. 65% of social media users say they use it to learn more about brands, products and services. Learn about how to find more about customers' attitudes, preferences and buying habits from what they say on social media channels.
The Multiplier Effect of Integrating Search & Social Advertising
Latest research reveals 68% higher revenue per conversion for marketers who integrate their search & social advertising. In addition to the research results, this whitepaper also outlines 5 strategies and 15 tactics you can use to better integrate your search and social campaigns.
September 17, 2014
September 23, 2014
September 30, 2014
1:00pm ET/10:00am PT