The Mixed Message of Paid Inclusion

  |  June 5, 2002   |  Comments

Search engines are sending out mixed (if not downright contradictory) messages.

I remain a defender of paid inclusion. There's an advantage to paid programs that allow site owners to expedite controlled entry into a search engine's database. Getting the home page of a new site listed right away can benefit search engines, their users, and site owners. Similarly, the ability to feed search engines content they might typically be unable to access can be helpful to all.

Paid inclusion can be a difficult concept for search engines to "sell" to site owners, especially for crawler-based search engines. After all, aren't they supposed to pick up pages across the Web naturally? Why should a site owner pay to make this happen? You should, say the search engines on their paid inclusion pages, because we don't get all the pages, we don't index them quickly, and we don't refresh them on a regular basis.

This message might convince site owners they'd better sign up for the various paid inclusion programs, but it also works against a completely different message search engines want potential portal partners and the public to believe: They have fresh, relevant, and comprehensive listings of what's on the Web.

Let's examine what crawler-based search engines are saying and the mixed message paid inclusion produces. We'll start with the pioneer of paid inclusion, Inktomi.


Inktomi pitches its "Search Submit" program by saying "traditional search engines" (including Inktomi):

  • Refresh their indexes less than once a month

  • Crawl only a portion of your site (usually between 30 and 50 percent)
  • May not fully leverage your metadata

On the other hand, Inktomi tells potential partners it provides "highly relevant search results," "large, regionally specific databases chosen from a master database of two billion URLs," and "rapid content refresh cycles."

So which is it? An out-of-date index that gets only parts of Web sites or quality search?


At FAST we're told we should consider the paid inclusion program resold by its partners because "typical search engines take as long as 45 days to update their listings. And, most listings include only the home page and maybe additional section pages... With FAST PartnerSite, all of your site's key public pages are available."

On the page targeting potential search partners, FAST says, "We are committed to providing our users with the largest and freshest catalog on the Net."

FAST tells potential search partners it has the largest and freshest catalog, meanwhile it tells site owners most of their pages will be ignored or not frequently refreshed.


AltaVista explains its paid inclusion program helps end "the need to continually submit URLs to AltaVista."

To potential users it says, "AltaVista offers the world's fastest, most comprehensive search service."

Why does the most comprehensive service require site owners to continually submit their URLs?


Even new player Teoma has inconsistencies. Here's its paid inclusion pitch: "With Site Submit, we guarantee your most important web pages will be included in the database powering and As a member of the Site Submit program, we'll make sure that your latest, best content is always available to our users."

Teoma is the same service that told the press, when it emerged from beta earlier this year: "Now with Teoma's search methodology, users can feel more confident in the accuracy and relevance of their Web searches."

Shouldn't an accurate and relevant search service already have a site owner's "latest, best content"?

What About Google?

Google alone among the major crawlers doesn't operate a paid inclusion program. The company has repeatedly said it thinks such programs are unfair and would alter the quality of search results.

Given the mixed messages above, Google's been very wise in taking this stance. The company can say it offers high quality, comprehensive, and fresh search without fear that marketing material from its paid inclusion program will be used to dispute this, as in the above examples.

Despite this, Google's stand leads to a different conflict. One of its businesses is making company Web sites searchable. For instance, if you go to Cisco, you can search all of Cisco's content using Google.

Google should have as complete as record as one can get of the Cisco site, as with other sites for which it provides site search services. But this complete record is not made available to those searching the Web from Google's own site. Google prevents this, so as not to be accused of running a form of paid inclusion.

This means that Google, whose "mission is to deliver the best search experience on the Internet by making the world's information universally accessible and useful," finds itself in the odd situation of making some information inaccessible on its Web site.

Clean Up the Message

Those who understand how crawlers work know they do include many of the Web's best pages. Good content will tend to be included for free.

This doesn't deflect from paid inclusion's advantages. If you have a product catalog locked inside of a database, paid inclusion offers the ability to get that content into crawlers, which typically forgo that type of material.

You might have pages you believe are important that the crawlers haven't picked up, perhaps because their link analysis systems don't deem them important. Paid inclusion allows site owners the ability to include these.

Those offering paid inclusion don't need to sell their programs by sending conflicting messages about what they do. Explain the value without playing on fear. It's more honest with potential customers and less troublesome to potential searchers and search partners.

Search Engine Marketers: Include your company on before the list launches June 24th.

ClickZ Live Chicago Join the Industry's Leading eCommerce & Direct Marketing Experts in Chicago
ClickZ Live Chicago (Nov 3-6) will deliver over 50 sessions across 4 days and 10 individual tracks, including Data-Driven Marketing, Social, Mobile, Display, Search and Email. Check out the full agenda and register by Friday, Oct 3 to take advantage of Early Bird Rates!


Danny Sullivan

Danny Sullivan left Search Engine Watch as of Dec. 1, 2006.

COMMENTSCommenting policy

comments powered by Disqus

Get the ClickZ Search newsletter delivered to you. Subscribe today!



Featured White Papers

IBM: Social Analytics - The Science Behind Social Media Marketing

IBM Social Analytics: The Science Behind Social Media Marketing
80% of internet users say they prefer to connect with brands via Facebook. 65% of social media users say they use it to learn more about brands, products and services. Learn about how to find more about customers' attitudes, preferences and buying habits from what they say on social media channels.

An Introduction to Marketing Attribution: Selecting the Right Model for Search, Display & Social Advertising

An Introduction to Marketing Attribution: Selecting the Right Model for Search, Display & Social Advertising
If you're considering implementing a marketing attribution model to measure and optimize your programs, this paper is a great introduction. It also includes real-life tips from marketers who have successfully implemented attribution in their organizations.



    • Web Writer
      Web Writer (Money Map Press) - BaltimoreDo you have a passion for the markets and investing, and writing? Do you want to spend your days providing...
    • Web Production Specialist
      Web Production Specialist (Money Map Press) - BaltimoreMoney Map Press is looking for a self-starter to perform and oversee the production of daily...
    • Internet Marketing Campaign Manager
      Internet Marketing Campaign Manager (Straight North, LLC) - Downers GroveWe are looking for a talented Internet Marketing Campaign Manager...