Microsoft Builds Itself a Search Engine

In April, news emerged that Microsoft intended to make a huge, new investment in Web search. Evidence of that is appearing. Microsoft’s MSN Search site posted a large list of jobs in May, then drew attention last month when official information about its search spider, “MSNBOT,” was posted.

What’s MSN Search’s future? Details are still being determined, the service says. Building a proprietary crawler-based solution to gather editorial listings is viewed as essential for winning the fiercely competitive search sweepstakes.

“We view it as a three-horse race between ourselves, Yahoo, and Google — with Google in the lead,” said Lisa Gurry, a MSN group product manager who works with the search team. “We’re in a good place with our search engine, but there are good opportunities to increase the relevancy.”

MSN Search is already among the most popular search sites on the Web. Brand recognition is a different story. Google is so tightly associated with search, some use the name as a synonym for search. Even MSN admits many don’t realize how they end up searching at its service.

“We have a lot of work to do to build equity in our MSN Search brand,” said Gurry. “It doesn’t have the brand recognition that Yahoo and Google have, and that’s an area we hope to address over the next year or so.”

Monopolist or Counterbalance?

MSN’s plans to fight harder in search don’t please everyone. Microsoft’s market dominance is widely known. Most PCs run some version of the Windows OS, and the company commands a large share of certain software types, such as word processing programs. Some assume Microsoft will use its deep pockets to leverage dominance in Web search.

Such fears inspired Boycott Microsoft Search, a campaign to enlist bloggers to block Microsoft’s crawler from accessing their pages. The site warns:

Microsoft is building a Web search engine, and they intend for it to become the industry standard. Given Microsoft’s track record during the browser wars, there is every reason to believe the company will again use its monopoly power to eliminate competition by building a Web search service into the next version of Windows.

Scary stuff. Let’s look back to 1997, when news first emerged that Microsoft planned to launch its “own” search engine.

According to Time magazine in August of that year:

Excite and Yahoo may sound like the happiest places on the Web, but their party is about to get crashed. For more than six months, a team at Microsoft has been working on its own search engine/-directory, code-named Yukon. The company should have a beta version up by October.

Nearly five years after that launch, Microsoft has hardly crashed Yahoo’s party. Excite lost the portal wars, but not because Microsoft had a killer search engine. Despite Microsoft’s (and Yahoo’s) money, Google came out of nowhere to dominate search.

Google’s audience share is so large, some believe it should be regulated for having a near monopoly. If anything, Microsoft’s move casts that company in the unfamiliar role of monopoly-breaker.

Built-In Search Advantage

Others detractors, including News.Com and an anonymous MSN Search job candidate cited by Scripting News, suggest Microsoft will use the Windows OS to deliver an audience to its search engine in a way other competitors cannot.

This is hardly conjecture, but a long-time reality. Most surfers use Microsoft’s Internet Explorer (IE) with built-in MSN Search integration that works in a variety of ways. A “built-in search advantage” is how MSN currently reaches most of its audience.

“The bulk of our traffic is driven to us from our IE integration, and then from the MSN home page,” confirms Gurry.

Gurry said Microsoft would like to raise that search integration profile. Many IE users aren’t aware of it. Some push IE’s search button, never realizing Microsoft’s search engine delivers results from MSN Search. Others don’t know they can query MSN Search from the browser address bar.

Integration changes may boost Microsoft’s traffic. Yet there’s every reason to expect many will continue to use Google, Yahoo, and other search engines, as they have despite years of a Microsoft built-in advantage.

Search isn’t software. It’s not installed on a desktop. Switching search engines doesn’t require relearning commands or data formats. The only way Microsoft can lock people into its search engine would be to literally prevent them from navigating to others. Public opinion would hardly allow that.

Microsoft’s built-in advantage helped it become a major search player. It’s made the company competitive, but it’s not a winning edge. To beat Google and Yahoo, Microsoft needs more.

An End to Outsourcing

What can Microsoft do? Improving search quality is obvious. MSN Search has had pretty good quality over the years, a fact that’s largely been unrecognized. Microsoft believes things can be even better.

“We’ve asked, how can we improve the experience? Across the board, about 50 to 70 percent of queries go unanswered. That indicates to us that there’s a lot of growth yet to come in the search category,” said Gurry, who explained high failure rate is based on internal Microsoft research. “We’ve felt we should really develop our own [crawler-based] search engine to try and solve this problem.”

That “close look” at search occurred in March, when the company finished a review of where it was going in search. It decided outsourcing no longer makes sense.

“We looked at all options, whether it be potential acquisition or continuing to outsource. Ultimately, the right move, we decided, is to bring it in-house,” Gurry explains. (MSN may buy some components, as I conjectured last week.)

That decision echoes Yahoo’s acquisitions of Overture last week and Inktomi in December.

Yahoo’s and MSN’s moves are a fast swing back to search engines owning their technology rather than leasing it.

In 1995, popular search sites had internal solutions powering their main search results. By the end of last year, the majority — Yahoo, MSN, and AOL — were outsourcing their main results. Of the megasearch sites, Google alone owned its technology.

The situation is in full reverse. In addition to its watershed Overture acquisition, Yahoo owns crawler technology and will almost certainly use it in the near future. When MSN’s technology is ready, the company will make a change. AOL remains the sole outsourcing standout.

Building the Crawler

When will we see MSN crawler listings? MSN will only say no immediate switchover is planned.

“It’s not going to be something that happens overnight,” said Gurry. “We’re not making any big changes in the short term.”

Despite hype about the crawler emerging in June, it’s actually been online since April, under the agent name MicrosoftPrototypeCrawler. The first sighting was posted at WebmasterWorld.com. Last month’s excitement came because general information about the crawler, rebaptized MSNBOT, was spotted as publicly offered.

What about that call to block the crawler? Live for several weeks now, fewer than 100 sites signed on. Gurry hopes those that did will reconsider.

“Being associated with Microsoft is good in some ways and can have some negatives for some people, because it is a big company,” Gurry said, adding, “A lot of people come to MSN because it is a good site and we deliver a good service to consumers… The best way we can do that [with the new search engine] is having all the Web sites out there in the process, participating with the MSN crawler.”

This column was adopted from ClickZ’s sister site, SearchEngineWatch.com. A longer, more detailed version is available to paid Search Engine Watch members.