Yahoo! Reawakens the Paid Inclusion Debate

Yahoo rolled out its Site Match paid inclusion program in March. It quickly drew widespread criticism and was a terrible blow to Yahoo’s quest to win searchers from Google. It infuriated plenty of advertisers, too.

Why the outcry? Paid inclusion is hardly new. It’s been around over two years. In this column, the first in a series on paid inclusion, we’ll examine why paid inclusion blew up in Yahoo’s face and will likely be constantly cited as a drawback to its search service.

How to Solve Crawler Problems?

In mid-February, “Yahoo Search Technology” debuted, a brand new search engine to power searches on Yahoo sites around the world. The company previously used Google.

Like Google’s search engine, Yahoo’s new search engine gathers listings by crawling the Web. Crawler-based listings are equivalent to a newspaper’s editorial columns. When you search, you’re supposed get what a search engine deems most relevant to your query, based on ranking algorithm. It’s the search engine equivalent of editorial judgment.

Payment isn’t supposed to play a role in what’s ranked on top. For guaranteed ranking, buy paid-placement listings. These are the search engine advertising results that run at the top and bottom of Yahoo’s search results page.

Like Google’s (and others’) crawler, Yahoo’s crawler isn’t perfect. In particular:

• It probably won’t gather all pages from each site it visits. A site owner may feel “important” pages were missed.
  
• It may not revisit pages as often as a site owner would like.
• It may encounter problems when trying to gather pages delivered via databases and other dynamic delivery systems.

All the flaws above are true of Google, too. What’s different is how the companies pursue solutions.

Yahoo sees payment as one of several ways to correct its problems. Its new paid inclusion programs allow site owners to pay for guaranteed inclusion of their URLs and for regular revisits. Can’t pay? If you’re a nonprofit, Yahoo may provide this for free.

Google believes payment linked to crawler-based results potentially taints those results. The company seeks to solve its crawler flaws purely via unpaid means.

Who’s right? Depends on your perspective.

As we’ll see later in the series, Google’s lack of a paid solution program to help with crawler-based listings has caused the company to come under fire from frustrated site owners and marketers. A paid support program, similar to Yahoo’s paid inclusion program, may have spared Google such problems.

Yet there’s no doubt avoiding a full-blown paid inclusion program gave Google the upper hand, reputation-wise. Yahoo’s use of paid inclusion has caused some to view its results as tainted, despite strong denials that paid inclusion content is boosted in results.

The Paid Inclusion Fallout

“Questionable Results at Revamped Yahoo”

That was the headline of a Washington Post article following the program’s release. Two weeks after debuting its new search technology, paid inclusion hurt Yahoo’s reputation:

But do Internet searchers get good, accurate information? Or are the results of the search skewed to favor those who’ve paid to be in the index? The jury’s out on that one. —The Boston Globe

Thanks to paid inclusion, readers get the impression Yahoo’s search engine is on trial. Will they check to see if the charges stuck, or will they instead stick with another search engine?

Yahoo executives say the payments won’t improve a site’s ranking on the list of results that appear after a search. But at the same time, Yahoo acknowledged that there will be no distinguishing marks to alert Web surfers that a company had paid to be included. –The Wall Street Journal

If you understand the nuances of paid inclusion, you might believe paid and unpaid results can be combined without harming relevancy.

Unfortunately for Yahoo, most searchers don’t know what paid inclusion is, much less its nuances. They’ll read such an article and assume Yahoo sells results and doesn’t disclosing that fact.

New Search Engine Calls Up Resentment at Yahoo –Los Angeles Times

Resentment? Wow. Many search engine marketers hoped Yahoo’s new search engine would be an escape from the Google domination they’ve felt. In two short weeks, Yahoo transformed that potential goodwill into resentment for some.

Why Headlines Now?

Paid inclusion on Yahoo is new, but it’s been used for years by search engines, including MSN, AltaVista, and Inktomi (the last two currently Yahoo-owned). Why is Yahoo getting so much grief now?

From the advertiser perspective, because Yahoo’s basic paid inclusion program, Site Match, appears to provide little value. If you believe Yahoo’s convoluted explanations, any reasonable person would say, “Thanks, but no thanks.” I’ll explore this later in the series.

From the searcher perspective, because Yahoo’s brand new search engine faces greater competition for searchers in the new search engine wars.

Everything Yahoo does will be compared to Google. Google tells an easily understood story as to why paid inclusion should be considered wrong. From Google’s IPO filing:

Some of our competitors charge Web sites for inclusion in their indices or for more frequent updating of pages. Inclusion and frequent updating in our index are open to all sites free of charge. We apply these principles to each of our products and services. We believe it is important for users to have access to the best available information and research, not just the information that someone pays for them to see.

In contrast, Yahoo tells a complicated, hard-to-believe tale to justify paid inclusion.

The Mixed-Message Reloaded

To illustrate the difficulty in rationalizing paid inclusion, I wrote “The Mixed Message of Paid Inclusion” in 2002. It contrasts the wonderful things search engines said about their crawling capabilities with the dire tales told when pitching paid inclusion to advertisers. The mixed message hasn’t changed:

Yahoo… today announced that it has created a more comprehensive and relevant search experience for users through the deployment of its own algorithmic search technology —Yahoo press release, February 18, 2000

Sounds like good news for searchers. But wait! What are advertisers told about Yahoo-owned Overture Site Match, which feeds paid inclusion content into Yahoo search results?

Eliminate guesswork: Ensure that your pages are reviewed and included in the search index quickly and refreshed frequently. No waiting for search engines to find your site or guessing which content will be included. –Overture Site Match product page, May 18, 2004

Searchers are told Yahoo has a comprehensive, relevant search engine. You’d assume that means it’s fresh. On the other hand, site owners are told Yahoo’s search engine apparently just guesses about what to include and may not refresh content frequently.

Is it any wonder Yahoo’s gotten bad press since unveiling its new programs? Either you have a great search engine or you don’t. Playing it both ways simply doesn’t fly.

Propaganda War: Google 1, Yahoo 0

Google certainly had a field day watching Yahoo try to justify its program. Google had just come off one of the worst publicity periods ever (before the Google Gmail announcement in April and the privacy concerns that followed). Last December’s upset over Google ranking changes spawned a number of anti-Google discussions and articles.

Yahoo’s complicated paid inclusion program came to the rescue. After weeks of having its results questioned, Google got to sit back and watch Yahoo’s program put under the microscope.

Google brought company cofounder, Larry Page, to major publications in unprecedented fashion. Gone were the typically cautious, limited comments regarding competition. It’s Search War now. The gloves are off, and, by not having paid inclusion, Google appears to occupy the high ground. Page confidently shot out quotes like these:

Any time you accept money to influence the results, even if it is just for inclusion, it is probably a bad thing. –The New York Times

It’s really tricky when people start putting things in the search results. –The Wall Street Journal

The final evidence of Yahoo’s failure on the propaganda front? Daniel Brandt launched Yahoo Watch.

Brandt created anti-Google site Google Watch back in 2002, when Google’s incredible rise in popularity began to raise concerns. Until now, Brandt was content to hold Google solely responsible for ills that often also apply to its competitors.

Brandt always said he might target other services if he felt they warranted it. Yahoo’s paid inclusion programs drew his ire. Google is no doubt pleased to find Yahoo taking some of the blame it formerly took alone.

Going Forward

With so much bad publicity over paid inclusion, you’d wonder why Yahoo would do it at all. As we’ll examine, money is certainly a major factor. We’ll consider Yahoo’s defense: paid inclusion can help search relevancy.

I’ll also cover the “level playing field” myth Google has so effectively employed. Google’s results aren’t necessarily “purer” than Yahoo’s because Google lacks paid inclusion. Plenty of commercial sites vie for position on both. It’s just that Google doesn’t earn money directly from its editorial results.

I expect Yahoo will fight back against charges it’s less pure than Google by pointing out Google’s AdSense program potentially gives that company as much incentive to skew results as does Yahoo’s paid inclusion.

AdSense puts Google ads on external Web pages. Obviously, if Google drives traffic to pages carrying its ads, the company may earn more. That’s something Google strongly denied it would do when AdSense launched. It repeated the denial recently, when I looked at allegations Google favored sites with AdSense content. As with Yahoo, an incentive for favoritism is there.

I’d planned to cover the AdSense issue in this series. I was particularly surprised when Yahoo raised the issue itself, when I spoke with the company recently. I don’t recall Yahoo ever employing a “Google has incentive to be bad” argument before. I suspect it may emerge as a new line of defense if paid inclusion criticisms continue.

The series will also explore paid inclusion’s future. Will other search engines embrace it? Will Yahoo determine it needs in-line disclosure of paid inclusion URLs (as I certainly hope it will)? I’ll get into these questions as this series progresses.

Want more search information? ClickZ SEM Archives contain all our search columns, organized by topic.