The End for Search Engines?

Part One

The impending closure of Go only underscores the dramatic changes that have been taking place among the major search engines over the past few months. Money is tight; new revenue is being sought anywhere, and no one seems guaranteed a future. Will your favorite search engine be around tomorrow? For searchers, such losses could mean less diversity in search results. For Web marketers, a consolidation could mean less likelihood of being found. It’s scary-sounding stuff, and no one knows the answers. However, a look back can provide us with some perspective on how the future may unfold.

We’ve lost search engines before. The demise of Go is dramatic simply because old search engines are supposed to quietly fade away, not come to a screeching halt. Open Text is a good example of this. In 1995, Open Text was one of the big search engines people depended on. It was arguably bigger than Yahoo. For example, I remember being at Internet World in 1995 and stopping by the fairly large Open Text exhibit. It was big enough to graciously allow Yahoo a table within its confines, where none other than Jerry Yang was personally answering questions.

Open Text faced the classic search engine question of whether it should develop its technology as a product for businesses, concentrate on its consumer Web site, or try to do both. The company ultimately decided to concentrate on software, and the Web search site was allowed to slowly die. By the time the Web search engine closed in mid-1997, no one really noticed.

The same is true for Magellan, which was a rival that Excite gobbled up in mid-1996. Magellan has never closed, but Excite has allowed it to wither away and be essentially unsupported. WebCrawler is another example, which Excite also acquired in 1996. WebCrawler is far more popular than Magellan, but it can hardly be considered one of the Web’s major search services any longer. There haven’t been service enhancements for ages, nor does Excite position the service toward users. However, at least it survives, unlike Point (Top 5 percent), another significant service that was acquired by Lycos in 1996, then slowly allowed to disapear into oblivion.

So, losing a major service like Go isn’t new. It’s just the sudden nature that’s shocking, especially when it comes at a time when so many major services are all struggling. AltaVista has just gone through another round of layoffs. So have Excite, LookSmart, and NBCi. Surely the end is near, and we’re going to be left with only a few search engines.

Perhaps, but our memories can also fade. Search engines have been in trouble like this before. Back in late 1996 and early 1997, the situation seemed as dire as it does today. Importantly, the portal route search engines were advised to follow for survival is what caused the current crisis they face today.

“The market for consumer-oriented search and directory services is overcrowded, and a shakeout could be in the offing in 1997,” warned Jupiter Communications in November 1996. “There are simply too many players offering similar functionality and features, competing for a limited number of advertising dollars and users,” the company advised. To survive, the search engines needed content and services to “keep users” at their sites, which would boost ad revenue.

A few months later, in April 1997, MSNBC ran an article called “Searching for Success — or Survival,” which was typical of several other pessimistic business pieces about search engines at the time. It discussed some of the early changes that search engines were making to evolve into portals, before that word was even being used (In reality, search engines were positioned as the new “online services” to rival big players like AOL, CompuServe, and Prodigy).

“1997 is ‘fish or cut bait’ for some of them,” one analyst said in the article. Funding was running low, and the analyst, like practically every other analyst quoted in that article, assumed that there would be a massive search engine consolidation that year.

In May 1997, Excite laid off half its editorial staff, prompting a Red Herring article to ask, “Can Excite make it without a deep-pockets old-media partner like Bertelsmann [which had just invested in Lycos] or Yahoo investor SOFTBANK?”

(Interestingly, Go, Excite, and Lycos all went on to get media partners, which hasn’t seemed to help them gain on Yahoo. Nevertheless, some analysts continue to hound Yahoo to find its own partner, despite its success in going it alone.)

The article went on to say, “While most observers think there’s room for at least two navigation sites, it’s up in the air which pair will be left standing.” A Jupiter analyst was also cited, saying, “A shakeout is definitely imminent.”

Doom and gloom, doom and gloom. When it comes to search engines, you can only have Coke and Pepsi and maybe RC Cola, we were told. Excite CEO George Bell was just one of several to say this, at the time: “There are a lot of ‘two’ examples out there,” he said in April 1997, during a panel on Internet navigation hosted by Silicon Valley’s Churchill Club. “There’s Pepsi and Coke, Time and Newsweek … the third always tends to struggle, the fourth tends to get bought. I think the two of us will make it.”

The turnabout came in July 1997, when Amazon announced a landmark deal to be a preferred book merchant carried by Excite and Yahoo. Suddenly, the big online-retailing deals began rolling in, just the pill the money folks said was needed to save the search industry.

“Analysts have been hounding search companies to break away from the advertising-only revenue model and into a more diversified business,” News.com reported in August 1997, in a story that discussed yet another new online-retailer deal, this time between Lycos and Barnes & Noble.

To maximize the value of those deals, search engines still needed to keep users at their sites. But how do you make the sites “sticky”? Excite had made a significant move in July 1997, when it began providing free email. Lycos and Yahoo soon followed in October 1997. That was the same month that the first mentions of the “P” word — portal — began to appear.

“[Microsoft] is expected to use the search technology to create a Web search engine that would be at the center of a revamped MSN.com Web site, positioned as an Internet ‘portal,’ in which a major portion of the content now available on the proprietary MSN service would be on the Web for anyone to see,” MSNBC wrote in October 1997, when Microsoft announced plans to run its own search engine. The idea of “portals” was so new that the MSNBC article twice surrounded the word with quotes within the article. Today, it sounds almost like Dr. Evil from “Austin Powers” carefully explaining a basic concept that everyone knows about.

By 1998, instead of the expected shakeout, the future of search engines seemed bright due to their transformation into portals. Investors loved them. They had tons of traffic that seemed certain to produce huge revenues. As for the portals themselves, accidentally call them search engines and you’d be informed they’d grown well beyond that. Search was only a small part of what they offered, you’d be told. It was almost as if the portals were embarrassed of their search engine roots.

Well, why not? Being search engines nearly cost the portals their lives. People came, searched, and went, ignoring banner ads that were a vulnerable single source of revenue. Portal features, though inconvenient to those looking just to search, were what seemed to have saved the search engines. Search was really a loss leader to them, except mainly for the saving grace of selling overpriced keyword-linked banner ads. Why did these sell? They were easy to implement, and if an advertiser bought enough, they’d still get tons of traffic, no matter how poor the click-through rate was. Advertisers, flush with investment, were happy to shell out for huge orders. Indeed, there were so many banner advertisers, along with would-be retail partners, that some complained they couldn’t get search engine sales reps to call them back.

But by the beginning of 1998, the year when we were supposed to have only two or three search engines remaining, something critical happened…

Stay tuned until next week to find out all about the power of paid listings and the end or survival of search engines.