Surviving in Today's Search World

Search engines, like other dot-coms, have encountered choppy seas since ad revenues went south. Many are struggling for survival. All are looking for new ways to monetize.

Yahoo is cutting operating expenses with layoffs and paid subscriptions and services. Most astonishing was a deal with Overture to display paid listings, along with plans to launch a paid placement program in the spring. Fans may not like it, but the firm waited too long to change its business strategy and lost its dominant position. Nielsen//NetRatings shows Yahoo trailing AOL in the top 25 Web properties for November.

Search engines surviving the storm were not unscathed. Still, many endured. Those struggling include LookSmart, AltaVista, Infoseek, and Excite (ExciteAtHome is in bankruptcy). Those running a tight ship include Google, Overture, FAST, AOL, MSN, and Mamma.com (to name a few).

Navigating in Troubled Waters

What does it take to remain profitable in hard times? Lesser-known, middle-tier search engines such as Mamma.com have remained profitable by sticking to the knitting and capitalizing on core assets. When asked what makes Mamma.com successful, Marketing Director Davide Molino replied, “Middle-tier engines have survived by maintaining small staffs with reasonable salaries, keeping their sites simple, listening to employees who are in the trenches every day, running with new ideas, and, most of all, being honest with users.”

Mamma.com went live in 1996 and grew slowly through word of mouth. It currently reaches 8 million unique users per month through its network of sites. At the height of the dot-com frenzy in 1999, it employed a banner sales force and business development team, and advertised heavily with offline billboards and TV spots. This search engine recognized its niche and didn’t become a me-too portal.

Recognizing a trend, Mamma.com started selling search result rankings with its “Premium Search Results” program, shuffling paid listings in with meta-search results. It later labeled them “Mamma Recommends,” when search engines were criticized for not fully disclosing advertising. They were subsequently renamed “Mamma Classifieds,” to clarify that they are paid listings. Further diversification includes a proprietary banner delivery system to replace DoubleClick’s DART (Advertising Delivery System, a.k.a. ADS) and an automated registration system slated for launch in the spring.

Mamma.com’s strategy of capitalizing on core assets seems to be paying off. It will provide search functionality for InternetNews.com and EarthWeb as internet.com (ClickZ’s parent) provides a “Search the Web” option on those sites. An internet.com user selecting “Search the Web” will be linked to a Mamma.com search result page with the option to continue searching or return to internet.com.

Meta-Search Explained

In case you don’t know, a meta-search engine queries multiple sites to provide results, and does not maintain its own database. When you type in a query, it simultaneously searches a series of other sites and compiles and displays those results. Because they don’t maintain a database, meta-search engines do not accept URL submissions.

In using a meta-search engine, you get a snapshot of the top results from a variety of search engines. Proponents say these tools return fewer results with a greater degree of relevance. Critics say the searches are quick and dirty, because they don’t query all the best sources (such as Google and Northern Light) and don’t allow refined searches. Mamma.com claims to return results 96 percent of the time, displaying data from five different sources (on average). Besides Mamma.com, other meta-search engines include Copernic, Dogpile, Ixquick, MetaCrawler, ProFusion, and Viv&iacutesimo, to name a few.

Monetizing the Search

Most search engines and directories have adopted some means of making money from presenting search results. Paid links were considered a threat to editorial integrity, so the first monetizing effort by major search portals was the paid submission/inclusion program started by Yahoo These programs require payment to expedite entry of Web sites/pages into a search database.

Paying for timely consideration allows the editorial product to remained untainted while search engines gain revenues. Examples of paid inclusion programs are Yahoo’s Express, LookSmart’s Express Submit, Inktomi’s Search Submit or Index Connect, AltaVista’s Trusted Feed, and FAST’s PartnerSite. The engines that don’t use paid inclusion are Google, Open Directory Project (ODP), and Northern Light, although Google provides AdWords as paid placement.

When it launched, no one would have predicted Overture’s phenomenal success. Despite the fact that a year ago paid listings were widely condemned, they’ve gained a foothold and now seem to be accepted as long as links are clearly labeled as advertising.

Staying Competitive

Based on information from the recent Search Engine Strategies 2001 conference, the current industry focus is on freshness and indexing more data of various file types from the deep Web to entice users.

Many search engines are improving database freshness, selectively updating pages that change frequently by crawling them more often. FAST completely refreshes its database every 9 to 12 days, claiming to be “the Internet’s freshest and most up-to-date search engine.” Inktomi presently re-indexes its “Best of the Web” every 9 days (about 20 percent of its database). Google’s spider crawls the entire Web (about a billion pages) every 28 days, crawling news sites and such more often.

Search engines are eager to index beyond HTML and text page results. They’re seeking better ways to integrate relevant information from Web pages, images, video files, MP3 files, Usenet messages, maps, news, and FTP files all in one search, showing multiple results on the same page. FAST started doing this in August; others, such as Google and Lycos, are not far behind.

Survival Guide

You don’t have to be a top banana or household name to succeed in the search engine game. It helps to have a good business plan, to identify your niche and industry trends, to be decisive, and to move quickly.