I remain a defender of paid inclusion. There’s an advantage to paid programs that allow site owners to expedite controlled entry into a search engine’s database. Getting the home page of a new site listed right away can benefit search engines, their users, and site owners. Similarly, the ability to feed search engines content they might typically be unable to access can be helpful to all.
Paid inclusion can be a difficult concept for search engines to “sell” to site owners, especially for crawler-based search engines. After all, aren’t they supposed to pick up pages across the Web naturally? Why should a site owner pay to make this happen? You should, say the search engines on their paid inclusion pages, because we don’t get all the pages, we don’t index them quickly, and we don’t refresh them on a regular basis.
This message might convince site owners they’d better sign up for the various paid inclusion programs, but it also works against a completely different message search engines want potential portal partners and the public to believe: They have fresh, relevant, and comprehensive listings of what’s on the Web.
Let’s examine what crawler-based search engines are saying and the mixed message paid inclusion produces. We’ll start with the pioneer of paid inclusion, Inktomi.
Inktomi pitches its “Search Submit” program by saying “traditional search engines” (including Inktomi):
- Refresh their indexes less than once a month
- Crawl only a portion of your site (usually between 30 and 50 percent)
- May not fully leverage your metadata
On the other hand, Inktomi tells potential partners it provides “highly relevant search results,” “large, regionally specific databases chosen from a master database of two billion URLs,” and “rapid content refresh cycles.”
So which is it? An out-of-date index that gets only parts of Web sites or quality search?
At FAST we’re told we should consider the paid inclusion program resold by its partners because “typical search engines take as long as 45 days to update their listings. And, most listings include only the home page and maybe additional section pages… With FAST PartnerSite, all of your site’s key public pages are available.”
On the page targeting potential search partners, FAST says, “We are committed to providing our users with the largest and freshest catalog on the Net.”
FAST tells potential search partners it has the largest and freshest catalog, meanwhile it tells site owners most of their pages will be ignored or not frequently refreshed.
AltaVista explains its paid inclusion program helps end “the need to continually submit URLs to AltaVista.”
To potential users it says, “AltaVista offers the world’s fastest, most comprehensive search service.”
Why does the most comprehensive service require site owners to continually submit their URLs?
Even new player Teoma has inconsistencies. Here’s its paid inclusion pitch: “With Site Submit, we guarantee your most important web pages will be included in the database powering Ask.com and Teoma.com. As a member of the Site Submit program, we’ll make sure that your latest, best content is always available to our users.”
Teoma is the same service that told the press, when it emerged from beta earlier this year: “Now with Teoma’s search methodology, users can feel more confident in the accuracy and relevance of their Web searches.”
Shouldn’t an accurate and relevant search service already have a site owner’s “latest, best content”?
What About Google?
Google alone among the major crawlers doesn’t operate a paid inclusion program. The company has repeatedly said it thinks such programs are unfair and would alter the quality of search results.
Given the mixed messages above, Google’s been very wise in taking this stance. The company can say it offers high quality, comprehensive, and fresh search without fear that marketing material from its paid inclusion program will be used to dispute this, as in the above examples.
Despite this, Google’s stand leads to a different conflict. One of its businesses is making company Web sites searchable. For instance, if you go to Cisco, you can search all of Cisco’s content using Google.
Google should have as complete as record as one can get of the Cisco site, as with other sites for which it provides site search services. But this complete record is not made available to those searching the Web from Google’s own site. Google prevents this, so as not to be accused of running a form of paid inclusion.
This means that Google, whose “mission is to deliver the best search experience on the Internet by making the world’s information universally accessible and useful,” finds itself in the odd situation of making some information inaccessible on its Web site.
Clean Up the Message
Those who understand how crawlers work know they do include many of the Web’s best pages. Good content will tend to be included for free.
This doesn’t deflect from paid inclusion’s advantages. If you have a product catalog locked inside of a database, paid inclusion offers the ability to get that content into crawlers, which typically forgo that type of material.
You might have pages you believe are important that the crawlers haven’t picked up, perhaps because their link analysis systems don’t deem them important. Paid inclusion allows site owners the ability to include these.
Those offering paid inclusion don’t need to sell their programs by sending conflicting messages about what they do. Explain the value without playing on fear. It’s more honest with potential customers and less troublesome to potential searchers and search partners.
Search Engine Marketers: Include your company on SEMList.com before the list launches June 24th.
We don’t generally think of paid search as a great channel for personalisation, but increasingly, it's becoming one.
Site search matters, yet many ecommerce sites are actually deterring customers through poor experiences. Indeed, a fifth of UK shoppers are not ... read more
How can you create content marketing which works for search, right from the start? Many of us probably think of SEO as ... read more
As you’re no doubt aware, Google finally rolled out its Google 4.0 algorithm update at the end of last week. Penguin is ... read more