FAST and Fresh

How would you like a snapshot of the past year’s trends and a glimpse into the search engine future? To gather the information needed to take such a look, I’ve interviewed some of the major engines: AllTheWeb, AltaVista, Google, LookSmart, Lycos, and Yahoo I’ll be sharing what I’ve learned, beginning with AllTheWeb, a site owned by Fast Search & Transfer (FAST).

By offering AllTheWeb as a public search engine, FAST aims to provide users with comprehensive results while using the engine as a demonstration piece for its original equipment manufacturer (OEM) partners. The public search engine enables the company’s corporate partners to evaluate the technology before rolling it out on their own sites.

“FAST attacks the four pillars of search: relevancy, freshness, size, and speed,” said Rob Rubin, FAST general manager, Internet business unit. “This year, FAST introduced its linguistics tool kit to automatically detect phrases and provide accurate search results.” (Check this out by entering “Who is Little Richard?” or “What is the weather in New York?”)

Fast and Fresh, But Is It Profitable?

FAST’s business model is different than most search engines, Rubin explained. “The same search technology developed by FAST to power Internet portals on the Web (such as Lycos and T-Online) is used to provide ASP-based hosted search solutions for major corporations such as IBM,” he said. “We distribute our technology in shrink-wrapped packages and OEM for companies such as Reuters and TIBCO. In addition, we supply our technology to major e-commerce vendors, such as eBay and BroadVision.” This sounds more like a technology company than a search engine.

“Because AllTheWeb is a site where FAST delivers and, quite frankly, experiments with its technology on a significant number of sophisticated Web users, we have modest goals in its monetization,” said Rubin. “With AllTheWeb, we are looking to cover costs — not to drive the site itself to profits. That is for our OEM partners to do with their destination sites.”

But FAST did, in fact, make money. Its third-quarter revenue of $9.6 million, compared to the second quarter’s $8.7 million, was an increase of over 10 percent — not shabby considering last year’s meltdown. Rubin said he expects the same growth in the fourth quarter. When FAST released its third-quarter numbers, year-to-date revenue came in at $25.6 million, over five times the total revenue for the same period of the previous year.

In 2001, FAST trimmed its overhead and sharpened its focus toward revenue-producing product suites, such as FAST Web Search and FAST Data Search. By the end of 2001, the workforce headcount went from 230 to 170 and management resources became concentrated in Oslo, Norway, and Boston, due to some office closings in the U.S., Norway, and the U.K.

Creative Monetizing

FAST has two business units: the Internet business unit and the enterprise business unit. The Internet unit, which includes AllTheWeb, competes with the likes of Google, Northern Light, Inktomi, and AltaVista. The enterprise unit, which powers Internet portals and provides OEM solutions to major corporations, competes with companies such as Google, Inktomi, Autonomy, and Verity. FAST’s major income comes from the enterprise unit, which accounted for 60 percent of its revenue in the third quarter.

But efforts are also underway to monetize the Internet unit, namely AllTheWeb’s paid-inclusion PartnerSite service. That service guarantees insertion in the company’s databases (without influencing positioning or ranking) for a per-URL fee.

PartnerSite includes a 24-hour refresh and is not recommended for small sites. It can be a good thing for those who take the time to monitor and rewrite each page every 24 hours to achieve high rankings, but most in-house search engine optimization (SEO) operations lack the staffing for such redundant monitoring and modification.

With the PartnerSite paid-inclusion service available through FAST’s partner Lycos, customers are able to guarantee fresh inclusion in AllTheWeb’s index, have their site searched by the same search technology that powers major corporations such as IBM, and get distribution through all of the FAST customers.

Indexing the Deep Web

What has FAST done to index more of the “deep Web”? It completely rewrote and relaunched its crawler technology in November 2001. FAST can index many document types and can handle all kinds of dynamic content. For example, documents in PDF format are indexed at Scirus.

FAST’s PartnerSite inclusion program allows companies that want different content types indexed to be included on the Web. For example, through an inclusion program, IBM has the option to decide which of the documents on its public Web site it would like available via the index. (Actually, many of these different document types are currently available via an FTP search on AllTheWeb.)

To support news crawling, FAST has created a real-time cluster located in its data center. This cluster is based on FAST’s DataSearch product, which is the same product being used to power Reuters. FAST uses this technology to continuously crawl 3,000 premier news sites, keep the index around for five days, and continually refresh it every two hours. As new documents are crawled, the index is updated in subsecond latencies.

Tips for Good Rankings

Want high rankings? FAST recommends that technicians stick with the basics and work on leading-paragraph content. Use great design, title text, clear descriptions, and linkage from good reference points inside the network. SEO technicians should focus on improving content on the site. FAST will first check for Open Directory Project (ODP) descriptions; if none, it will look at meta descriptions; if none, it will examine the first 250 characters. Linguistic analysis of keyword frequency often reveals funny patterns — too many adjectives or an overuse of keywords. The document will then be flagged as spam.

FAST doesn’t like cloaking or spam and will fight vigorously to eliminate both — including black listing offending Web sites. If freshness and comprehensiveness are important to you, FAST suggests using its PartnerSite inclusion service. FAST reports that it constantly adjusts its relevancy algorithms to bring the freshest, most relevant content to its searchers.

Has Search Behavior Changed?

We’ve all heard that 86 percent of users find Web sites through search engines (GVU WWW User Surveys). Other studies report 42 percent or lower. Which to believe? Rubin recently read some Jupiter reports indicating that sophisticated users turn to search engines and destination sites, making search activities an integral part of the user experience. According to Jupiter’s Rob Leathern (August 2001), users go to search engines more when they are experienced. Approximately 30 percent of e-commerce purchases start at a general-purpose search engine. “FAST knows from its log investigations that people are launching their searches and navigating more using the Web engine,” said Rubin.

Engines of the Future

In the future, Rubin believes we’ll see more navigation features like “FAST Topics,” which are similar to the ODP topic listings seen at the top of the page and help searchers further refine their searches within specific categories. There is a demand for access to more Web content, and many companies want to make such content available. Rubin believes we’ll soon see more specialized catalogs like the ones FAST created for science at Scirus and for soccer information at Megasoccer.com. The focus for search companies will be on better intelligence to guide users to the results they are looking for. That will include the use of categorization and phrase matching with larger phrase dictionaries. Rubin predicts a period of great innovation and says that FAST is posed to lead the way.