Growing Pains in MetaVertical Search

Passions were inflamed last week when classified ad aggregator Oodle revealed Craigslist had asked it to remove the site’s ads from its index (and it complied with the request). Despite a cacophony of discussion about Craigslist’s decision, it wasn’t initially clear exactly why the company made that request. What’s obvious, however, is there are plenty of possible reasons a company like Craigslist might resist cooperation with Oodle and its ilk. Yet others in Craigslist’s position — like the New York Times Company — are taking the opposite approach, by investing in the Oodle-like jobs aggregator Indeed.com.

My curiosity piqued, I sought out Craig Newmark and Craigslist President & CEO Jim Buckmaster to get the company’s rationale for the request.

“Craigslist users and their listings are not a commodity to be traded upon for commercial purposes,” Buckmaster told me in an email exchange. “Robotically harvesting content from our site compromises our own users’ access to finite server resources, while providing little or no offsetting benefits.”

Oodle was singled out, Buckmaster said, only because it called attention to itself “by scraping way more listings, and by using our name in their press releases, homepage, and marketing materials.” Others with similar models would do well to check out Craigslist’s terms of service, he suggested. (Oodle maintains those TOS were recently changed to disallow a practice previously unmentioned.)

It’s an important issue, particularly in the “Web 2.0” era of ostensible openness. APIs, RSS feeds and mash-ups (and ad-supported businesses based upon these concepts) are the rule rather than the exception. I was surprised the issue hadn’t come up before, what with newspapers and job boards doggedly defending their classifieds businesses against myriad threats. (Ironically, one of the biggest competitors so far is Craigslist.) Any of these players — local newspapers, especially — could argue their classified ads sections comprise a community of buyers, sellers, recruiters and lonely-hearts who advertise to a targeted audience of one another.

You Scratch My Back?

It appears anyone arguing against crawling and aggregating would have plenty of legal ammunition. Back when the Internet started to make aggregation models possible, you saw plenty of pushback from various stakeholders. Intellectual property attorney Martin Schwimmer reminded me of a few cases. In a 2003 lawsuit, American Airlines successfully argued FareChase’s spiders (now owned by Yahoo) were trespassing when they crawled the AA.com site, winning a preliminary injunction to keep the travel search engine away. Similarly, eBay (which owns a 25 percent stake in Craigslist) won an injunction that kept auction aggregator Bidder’s Edge from crawling its site without its permission in 2000. (Both cases were settled out of court.)

Now, largely due to the boom in search and the idea there are benefits to cooperation, prevailing attitudes have changed. As Oodle founder and CEO Craig Donato pointed out to me, eBay now offers an API that lets applications make up to 10,000 calls to its database per month for free. (On the other hand, there aren’t any auction aggregators left standing I’m aware of.) Many travel players, meanwhile, now pay affiliates like SideStep and Kayak.com a finders fee when people make reservations after being referred from those sites.

“For us [legal precedents] are a moot issue,” Donato said. “If it’s not a win-win, I don’t think it needs to go there. I believe in my bones that it’s in everyone’s interest to be in my index.”

Indeed’s Paul Forster agrees, saying he’s had nothing but positive reactions from companies whose site the jobs aggregator crawls. “It’s been remarkable,” he told me. “It’s been very well received by everybody. We get emails every day and called every day from services that have third party job boards and recruiters wanting to have their jobs included in our index. It’s a case of very, very positive reaction from everybody.”

Who’s Got the Power?

The aggregators’ arguments stem from the idea players like Oodle don’t detract from communities like Craigslist, but rather enhance them, actually sending traffic to the Craigslist sites — and its advertisers’ listings — they wouldn’t otherwise get. Donato says Oodle sent “well over a million referrals” to Craigslist last month.

Buckmaster reckons aggregators as a whole represent between 0 and -5 percent of Craigslist’s traffic. “Harvesting robots are very resource intensive, and lessen the number of pages that would otherwise be seen by our users, but don’t refer much traffic to us (because they don’t have much to refer) such that the net impact on traffic is negative,” he explained, referring me to an Alexa chart comparing the two sites’ traffic numbers.

Even search engines aren’t that material to Craigslist, Buckmaster claims. He estimates around 1 to 5 percent of the sites’ traffic are from organic search listings from regular, non-vertical search engines.

The CEO denies competitive concerns are behind the company’s move. “Give our public service model,” he said, “we don’t really think in terms of competition.” Whatever Buckmaster’s reasons, his actions effectively serve to outflank Oodle. Craigslist is enough of a powerhouse that it doesn’t need Oodle. We’ll see whether the balance of power changes if and when the blue and red lines on that chart are a little closer together.

If You Can’t Beat ‘Em, Buy ‘Em

Taking the opposite approach from Craigslist is the New York Times Company, which recently took a 14 percent stake and a seat on the board of Indeed.

“We don’t know what’s going to happen to this metasearch model,” Martin Nisenholtz, president of the New York Times Digital, said on the company’s earnings call this week. “Part of the reason we acquired equity in the company and now sit on the board is to learn and to be a part of this — what we believe to be a very important trend…. We think it’s an exciting trend and we think it could be a game-changing trend over time.”

Nisenholtz added the company is looking at “other opportunities in metasearch.” Perhaps the publisher would expand beyond jobs to get into metasearch for real estate, automotive, etc. So long as you own a piece of the aggregator, in addition to the classifieds business, yourself, where’s the harm in participating?

On the One Hand…

How’s it going to shake out? Will other Craigslist-like publishers ask their listings to be removed?

“We have recommended to our [publisher] clients that, in many cases, they should participate,” said Peter Zollman of consulting group Classified Intelligence. “In most cases we believe they should participate with the aggregation companies — most specifically Google — because there is a lot to be gained. There is frankly also a lot to be lost, but we think the benefits outweigh the downsides.”

Some suggest the pricing model for classifieds must change in the face of aggregation, such that publisher are rewarded for bringing advertisers leads (and increasing distribution through search). An addition to that model is one in which advertisers pay extra, like on eBay, to get their listings highlighted.

“As the price point for classifieds goes down — and a lot of the pressure to push that down comes from sites like Craigslist — there’s more volume,” opines Oodle’s Donato. “The flip side of volume is there’s more clutter to break through.”

How will it all shake out? I think Zollman said it best: “With a lot of difficulty.”

It’s only just begun.