Yahoo! Renews With Google, Changes Results

After months of speculation, Yahoo announced this week it renewed its agreement to use Google‘s results as part of its search listings. In addition, Yahoo made a substantial change to its historic barrier between human-powered and crawler-based search results.

Since its inception, Yahoo has used human editors, or “surfers,” to organize Web sites into categories. Recognizing humans can’t index everything, for years Yahoo has partnered with a third-party crawler-based search engine to provide answers for instances when no matches are within its own listings.

Historically, Yahoo paid third-party search providers for queries they handled. Since Yahoo has consistently been one of the most popular search sites on the Web, those queries translate into serious money. In 2001, Google was paid $7.1 million by Yahoo for queries it handled, according to a Yahoo proxy statement filed in March.

Fighting for Yahoo

Competition for the Yahoo contract was intense, with Inktomi and FAST also competing. Google made history by being the first provider to win Yahoo twice in a row.

Yahoo has been fickle about its partners. Open Text was the first, then AltaVista won in mid-1996. That company was dumped for Inktomi in mid-1998, largely because AltaVista was viewed as a competitor in the portal space, while Inktomi ran a behind-the-scenes business model of powering, but never competing with, portals.

When Inktomi lost to Google in 2000, it seemed due to both Google’s reputation of delivering high-quality search results and what was widely assumed to be a better business proposition. It turned out that as part of the deal, Yahoo gained a small stake in privately held Google.

Two Strikes, But One Great Hit

This time, Google had two strikes against it. First, some at Yahoo believe Google may be capturing their visitors. Google does handle search volume more than double that of Yahoo’s, as measured by search hours, and Google has nearly equal Yahoo’s 30 percent reach of the U.S. search audience, according to Nielsen//NetRatings.

A deal struck earlier this year between Yahoo and Overture precluded Google from giving Yahoo both editorial and paid listings. As a result, Yahoo, looking to maximize revenues wherever possible, was potentially looking at a situation where it had to either pay Google to keep it as a partner or go with another provider and earn money.

Both Inktomi and FAST offered the latter. Some of their crawler-based results are sold on a paid-inclusion basis. Unlike Overture, those purchasing paid inclusion are not guaranteed a particular ranking. Yahoo could have gone with one of these without violating its exclusive deal with Overture to provide the Yahoo’s U.S. Web site with guaranteed placement listings.

Despite these drawbacks, Google had one factor strongly in its favor. The company is widely acknowledged as a leader in search relevance. For some, Google is a synonym for search.

Yahoo initially partnered with Google so its own users would feel they got both the quality of Google and the unique view Yahoo’s human-powered results bring to the Web. Dropping Google could have backfired, making it seem as if Yahoo sold out search relevance for cash, regardless of the fact both Inktomi and FAST have very good relevance themselves.

Using Google More Than Ever

In a twist, Yahoo didn’t simply renew the deal for Google to be its “backup” partner, used only when Yahoo doesn’t have an answer. Instead, Google’s results will be more in evidence. Yahoo unveiled a new search results page. There’s no longer a separation between Yahoo’s human-powered listings and Google’s crawler-based results.

In the old Yahoo system, a search for a popular query, such as “cars,” would have generally brought back matches from Yahoo’s human-powered listings on the first page of results, under the “Web Site Matches” heading. Only after going through all Yahoo’s listings would users find Google’s crawler-based results, in the “Web Page Matches” area.

Now, Yahoo and Google results are mixed under a new “Web Matches” section. Any listing from Yahoo’s human-compiled categories is easily identified with a “More sites about” link below its description and a red arrow pointing to the link. Click and you’ll see where the page “lives” within the Yahoo directory — as well as find similar pages. In contrast, listings solely from Google lack the “More sites about” link.

Yahoo says it uses its own search algorithm to sort through its listings and those from Google to determine which to blend in the top results. This may be so, but that algorithm clearly takes its lead from Google. After testing 30 different queries, we found only three delivered results radically different between Yahoo and Google.

If results are so similar, how can it be said Yahoo still uses its human-powered listings? The answer is pages that rank well for queries in Google tend to be pages also listed in Yahoo In fact, when a page is listed in Yahoo, Google will likely give it a ranking boost. Google has always in that way had Yahoo listed sites in its results. Now, Yahoo is doing the opposite, blending Google-only listed sites in its results — and blending and ranking the data in a manner that seems so close to Google, result sets often have little difference between them.

Better Than Google?

A differentiation Yahoo had from Google in the past was it often presented a completely different view of the Web. I’m not saying Yahoo’s results were better or worse. They were unique, and uniqueness is to be valued. Now, Yahoo seems to speak with Google’s voice. Why not simply list to Google directly?

One reason may be Yahoo’s results tidy up Google’s listings. Both may list the same pages. Yahoo provides sites it lists with a human-reviewed title and description. Google, in contrast, automatically generates the title and description based on the page’s content. It’s not always perfect, so Yahoo’s listings may be more readable.

Blending Improves Yahoo

The new change will be most evident — and a benefit for searchers — for queries for which Yahoo has only a few matches in its own database.

For example, searching old Yahoo for “gluten free pancake recipes” delivered exactly one listing from Yahoo’s human-powered database. That listing is easily lost among the navigation and other clutter on Yahoo’s results page.

A searcher, especially a novice, might feel Yahoo doesn’t provide much selection, not realizing a further click on the “Web Page Matches” link would return nearly 3,000 matches from Google’s crawler-based results.

On the new Yahoo, this sparseness won’t happen. A user won’t be forced to click when Yahoo’s human-compiled listings have only a few matches. Google’s results will automatically fill the otherwise empty space.

More About the Deal

Yahoo released no specifics about the financial terms of the deal nor its length, saying only it is “long term.” The deal applies to the Yahoo.com site alone. Other Yahoo properties are free to choose other third-party providers, if they desire, according to Jeff Weiner, senior vice president of corporate development.

The Yahoo.com deal is nonexclusive. Yahoo can bring in an additional third party to provide results alongside Google’s.

“We are very open to working with multiple vendors,” Weiner said. “We don’t want to be dependent on any one vendor.”

Just talk, or a real direction for the portal? It could very well be real. Yahoo’s new help pages about the new search results make no mention that some listings are powered by Google. Instead, it says some results come from “third-party search engine providers.”

Why add a provider in addition to Google? This could give Yahoo the opportunity to gain paid-inclusion income from a crawler partner while maintaining the Google relationship.

To that end, Inktomi says not to count the company out of a Yahoo relationship — it could happen yet.

“We still continue to talk with them, and we’ve made a lot of progress,” said Vishal Makhijani, Inktomi vice president and general manager of Web search.