Twitter Ad Revenue Play: Now Allowing More Search Engine Indexing

Twitter has updated its robots.txt file to allow search engines to crawl more of the site. Specifically, Google, Bing, Yandex, and others will now have access to Twitter’s search results pages, which display Promoted Tweets where available.

The change will allow search results pages to show up in Google’s index, though user search is still disallowed. For example, the results page for a search on the Olympics becomes indexable and therefore could show up in Google search results to users seeking information on the event.

With the new indexing direction, searches for keywords like “Ford” could potentially direct Google/Bing searchers to Twitter’s internal search for the term. This is different from the way real-time search operated in that users were apt to see actual tweets on the topic displayed in Google’s results. They would see Google’s ads on the search results page, then click on a Twitter result and go straight to a user page, where there are no ads. With this change, the results appearing in search engines would direct the user to a Twitter search results page, which displays a Promoted Tweet embedded in the feed (when available).

Bing already has firehose access, so it can use real-time tweets in its search results. Google used to have this level of access. Then things between the two went sideways.

Twitter and Google have had a seemingly frosty relationship since their firehose deal ended last year. Twitter had provided access to real-time tweets until the deal ended in July 2011. In October, CEO Dick Costolo spoke with John Battelle at Web 2.0 Summit and hinted that the issue keeping the two from reaching an agreement was not a monetary one.

The timing begged the question, did the deal end because Google launched its own social network? Costolo said then that Google+ and Twitter were competing in different arenas; Google in features and Twitter in simplicity.

By January, Twitter had come out with guns blazing against Google’s announcement of Search Plus Your World, which incorporated its own social data into Google Search. Twitter said at the time:

“Twitter has emerged as a vital source of this real-time information. As we’ve seen time again, news breaks first on Twitter. As a result, Twitter accounts and Tweets are often the most relevant (search) results. We’re concerned that, as a result of Google’s changes, finding this information will be much harder for everyone. We think that’s bad for people, publishers, news organizations and Twitter users.”

Google responded with a statement on its Google+ Page:

“We are a bit surprised by Twitter’s comments about Search plus Your World, because they chose not to renew their agreement with us last summer, and since then we have observed their rel=nofollow instructions.”

Twitter’s instructions now tell Google it’s OK for it to crawl and index more of the site. But will Google? It’s difficult to decipher what actually happened at the table between Google and Twitter; we may never actually know the real reason they chose not to work together in that capacity any longer. It seems the deal itself was a huge missed opportunity for Twitter, in that users were directed around seeing its ads.

This change seems designed to address that. Now that Twitter has beefed up its own search a bit, is it hoping the search engines can help it direct traffic only to areas of the site it chooses, where the potential for ad revenue exists?

Twitter doesn’t have sidebar ads, so having search engines crawl and index tweets may draw in users, but doesn’t make them any money. It encourages people to look outside of Twitter for Twitter content. However, directing people only to those monetized places is an option that makes a lot of sense for Twitter. Will it work for the search engines? We’ll have to wait and see what shows up in its results.

This article was originally published on http://searchenginewatch.com/sew/news/2208943/twitter-ad-revenue-play-now-allowing-more-search-engine-indexing.