The Evolution of Paid Inclusion

Paid inclusion has always been a tricky concept to explain. Understanding it is important both to Webmasters and to searchers, as recent changes have made paid inclusion commonplace with search engines. Do these new programs mean that Webmasters won’t get listed unless they pay? Do these programs present a threat to the quality of searchers’ results?

Paid inclusion means that in exchange for payment a search engine will guarantee to list pages from a Web site. These programs typically do not guarantee that the pages will rank well for particular queries, however. Whether a page ranks well still depends on the search engine’s underlying relevancy algorithms.

In contrast, paid placement does guarantee top listings. If you pay enough, you’ll be top ranked for the terms of your choice. So why would anyone bother with paid inclusion when there’s paid placement?

For one thing, paid placement can be expensive. Getting and keeping a top ranking at paid-placement search engine such as GoTo.com involves an ongoing expense because you pay by the click for any traffic you receive. In contrast, basic paid-inclusion programs at crawler-based search engines such as Inktomi operate on a flat-fee basis. You pay once, and then you get all the traffic you can handle (maybe).

Of course, with paid inclusion there’s no guarantee you’ll get any traffic at all. That’s why I’ve likened these programs to playing the “search engine lottery.” You are essentially buying additional listings that you might not ordinarily receive. In doing so, just as you would with a lottery, you may have more chances of coming up for various search terms. If this happens, you may get quality traffic for less expense than through paid-placement programs.

Paid Inclusion at the Crawlers

Inktomi is the crawler-based search engine to have offered paid inclusion the longest, having made it available to the general public since November. In June, it was joined by AltaVista. FAST announced its beta program in July, with public availability likely in September. That means three of the five major crawlers now have these programs — leaving Excite and Google as the odd ones out.

Of those two, Excite says it expects to launch its own paid-inclusion program perhaps later this year, once its parent company @Home Corp. has determined how to handle processing of subscription services that might be offered by its portal. That would leave Google standing alone, a position the company is happy to occupy.

“We have no plans for a paid-inclusion program. As we’ve stated in the past, our search results represent our editorial integrity, and we have no plans to alter our automated process, which works very well in gathering information and delivering highly relevant results,” said spokesperson Cindy McCaffrey.

That stance gives Google a PR advantage over its rivals, but Google can also afford to do this in part because the company has an alternative stream of revenue: paid placements that it sells on its site. In contrast, competitor Inktomi doesn’t operate its own search engine. Instead, it powers portals that are looking to make money, and paid inclusion allows Inktomi the ability to keep them happy with its services by sharing revenue.

To further convince Webmasters of the value of paid inclusion, Inktomi plans to roll out new reporting tools later this month to its paid-inclusion customers. These will show you exactly how many clicks Inktomi listings brought to your Web site, the search terms associated with those clicks, and in the future perhaps even the exact search engines that generated the traffic. FAST says its program will offer similar reporting.

Such reporting is going to be a powerful weapon to convince Webmasters that the money they’ve spent on paid inclusion is worthwhile. In fact, some Webmasters would pay just to have traffic reporting itself, independent of being promised that any particular page would be included in an index.

LookSmart’s Paid-Placement Transformation

Like Inktomi, LookSmart has also faced pressure from advertisers wanting to see the value in paid inclusion. LookSmart’s response was to roll out a new version of its program that includes a paid-placement option.

Despite the change, it is still possible that you may find sites at LookSmart that are not paid-inclusion partners. Plenty of sites that use LookSmart’s less-expensive paid-submission programs (which charge a one-time fee) do get listed and ranked well. Even nonprofit sites that wouldn’t have paid anything to be listed do come up, such as the official Red Cross site on a search for “red cross.”

The key thing to watch is whether a taste of paid-placement revenue will leave LookSmart wanting more. If so, the company could move to a more-aggressive paid-inclusion model.

Paid-Placement Crawlers?

Similarly, crawler-based search engines could decide that paid inclusion is no longer an option but instead a requirement for Webmasters to get listed. It’s a fear I often hear expressed by Webmasters: Are the crawlers going to drop free listings in order to promote their paid-inclusion programs?

For example, most Web sites probably already have many pages listed in AltaVista. What incentive do they have to enroll in its new paid-inclusion program, particularly when most sites probably don’t need the fresher update that the program offers? None, really — especially when you consider the cost of AltaVista’s program, over $1,000 per year to get 25 pages listed.

What Does the Future Hold?

Hopefully, we won’t see a pullback on crawling — or worse, a transformation into carrying only paid-inclusion listings. If this happens, then your favorite search engine might suddenly become a Yellow Pages-like guide for products and services. That’s fine when you want products and services, but users also turn to search engines for other needs. Allowing listings to be dominated by advertisers’ entries doesn’t always serve users best.

Turning back the clock and eliminating paid-inclusion programs isn’t the solution. No one wants to return to a situation where it took months to get in after submitting to a directory, or where large Web sites could only get a single listing to cover their broad range of products and services. Similarly, paid inclusion has helped ensure that high-quality pages that sometimes get missed by search engines can be represented. All of these changes benefit Webmasters as well as searchers.

The answer will probably be for the search engines to be more creative in how they develop new revenue sources. New services such as click-through reporting add value to a paid-inclusion program, giving Webmasters greater incentive to sign up. In turn, the search engines are relieved of the pressure from feeling that they need to scale back on indexing in order to force Webmasters to pay for traffic.