Caffeine: How Google's New Approach to Indexing the Web Changes Everything

For the last several years, the ultimate question on every marketer’s mind was simple and highly focused: how do I get to number one? And by “number one,” I don’t mean, “how do I get to dominate my category and sell more of my stuff than any of my competitors.” Number one was very clearly, number one – the top spot on a Google search for a valuable keyword.

It was, of course, with good reason that marketers sought this spot. The difference between number one and number two could be hundreds of dollars a day. The quest to divine the secret of the algorithm – that complicated math problem that determined who got number one and who got number (and beyond) – was taken as seriously as any marketing challenge could be. We worked diligently to figure out if headings mattered more than image text, if inbound links were from the right places, and what, what, what should go into a meta tag.

Of course, all of this rested on a solid, but shifting platform. We were trying to figure out how the engineers at Google programmed its crawlers to create a picture of the Internet, a picture that it could (and did) change however and whenever it wanted. Whenever the index would shift, based on some supposition about what constituted relevance, rankings could be turned upside down. This happened, frequently, without a lot of warning, causing site owners to either scramble to get back on top or smugly post that their site remained high in the rankings.

Google’s latest shift in the index was a bit different though. Its latest release, called Caffeine, was talked about for a long time, and the company had long signaled its intention to move away from its old and original way of categorizing the Web to a new method that better matched the real-time nature of the Web. Google finally pulled the trigger on this new release and it seems like a big deal.

What Is Caffeine?

The single most valuable piece of technology that Google has is its ability to capture content on the Web. Over the decade or so that the company has been doing this, the Web has shifted drastically, both in terms of user behavior and content. Google’s original approach to indexing content was really focused on words and images, most of which didn’t change all that frequently. Google got by on some version of the 80/20 rule: 80 percent of the content on the Web changed very slowly, and 20 percent changed very rapidly. It could put more of its resources toward the 20 percent and stay pretty current.

That’s hardly the case today, where technology has been focused on collapsing the time gap between something happening, being captured, and being viewed. The Web is now more about real-time experiences than it is about massive amounts of mostly-static content. Google has long been talking about how it wanted to do a better job of capturing this dynamic content, and has done a few things (like putting Twitter results into results pages), but this is a real, fundamental shift in the way that the engine operates.

What Caffeine Means for You

Recent incursions by Facebook and Twitter aside, Google remains the number one way people get to sites on the Web, including the one where you sell your stuff. That means that, yes, you do still have to figure out how to get to that number one spot – or at least close. There’s no guaranteed way to get to number one. You are always going to be competing against a set of people as smart and motivated as yourself. But here are three things that you must start thinking about and implementing:

• Number 1: If your SEO (define) group doesn’t get social media, get a new SEO group. Google’s spider has always been attracted to content that changes frequently. You want to be seen as a part of the 20 (or so) percent of the Web that needs special attention. Now that dynamic content is even more important, and social media remains the best way to do this. Just make sure that the content that you’re bringing in is still relevant to your core site’s topic. Otherwise, you’re efforts won’t help your cause.
• Number 2: Experiment and explore. Caffeine, as its name implies, is fast. Like, really fast. In the past, site owners had to wait days and often weeks for Google to index and re-index your site. Not anymore. Using a massively powerful and parallel set of computers, Google is cruising the Web all the time. That means you should be able to play around with new ideas and concepts and see the results fairly quickly. Stay away from anything that feels like it’s trying to trick Google. But give some new ideas a try.
• Number 3: Watch your competitors closely. A shift in the index means that new companies will rise in the rankings and old companies will fall. A few bloggers have suggested that local companies will suffer in the Caffeine era, as national brands rise to the topic of local searches. That may be the case, as will some other new shifts that maybe we haven’t anticipated. The only way to know for sure is to pay very close attention to where you are ranking, and who is on your heels. Everyone else is going to be playing around just as much as you are, and they may discover something great.

Building an online business means that you have to pay very close attention to how you generate traffic and how much that costs. Google offers an amazing opportunity to capture lots of people, but it’s a tricky business that gets trickier each time it makes a shift to its platform. This shift is a big one, but, in this case, it’s really Google playing catch-up to a Web that has dramatically changed its nature. This should ultimately be a good thing, especially if you are a marketer who’s used to looking for new opportunities.