Twitter made a change to its robots.txt file that will allow search engines like Google to crawl and index Twitter search results pages.
Twitter has updated its robots.txt file to allow search engines to crawl more of the site. Specifically, Google, Bing, Yandex, and others will now have access to Twitter's search results pages, which display Promoted Tweets where available.
The change will allow search results pages to show up in Google's index, though user search is still disallowed. For example, the results page for a search on the Olympics becomes indexable and therefore could show up in Google search results to users seeking information on the event.
With the new indexing direction, searches for keywords like "Ford" could potentially direct Google/Bing searchers to Twitter's internal search for the term. This is different from the way real-time search operated in that users were apt to see actual tweets on the topic displayed in Google's results. They would see Google's ads on the search results page, then click on a Twitter result and go straight to a user page, where there are no ads. With this change, the results appearing in search engines would direct the user to a Twitter search results page, which displays a Promoted Tweet embedded in the feed (when available).
Bing already has firehose access, so it can use real-time tweets in its search results. Google used to have this level of access. Then things between the two went sideways.
Twitter and Google have had a seemingly frosty relationship since their firehose deal ended last year. Twitter had provided access to real-time tweets until the deal ended in July 2011. In October, CEO Dick Costolo spoke with John Battelle at Web 2.0 Summit and hinted that the issue keeping the two from reaching an agreement was not a monetary one.
The timing begged the question, did the deal end because Google launched its own social network? Costolo said then that Google+ and Twitter were competing in different arenas; Google in features and Twitter in simplicity.
"Twitter has emerged as a vital source of this real-time information. As we've seen time again, news breaks first on Twitter. As a result, Twitter accounts and Tweets are often the most relevant (search) results. We're concerned that, as a result of Google's changes, finding this information will be much harder for everyone. We think that's bad for people, publishers, news organizations and Twitter users."
Google responded with a statement on its Google+ Page:
"We are a bit surprised by Twitter's comments about Search plus Your World, because they chose not to renew their agreement with us last summer, and since then we have observed their rel=nofollow instructions."
Twitter's instructions now tell Google it's OK for it to crawl and index more of the site. But will Google? It's difficult to decipher what actually happened at the table between Google and Twitter; we may never actually know the real reason they chose not to work together in that capacity any longer. It seems the deal itself was a huge missed opportunity for Twitter, in that users were directed around seeing its ads.
This change seems designed to address that. Now that Twitter has beefed up its own search a bit, is it hoping the search engines can help it direct traffic only to areas of the site it chooses, where the potential for ad revenue exists?
Twitter doesn't have sidebar ads, so having search engines crawl and index tweets may draw in users, but doesn't make them any money. It encourages people to look outside of Twitter for Twitter content. However, directing people only to those monetized places is an option that makes a lot of sense for Twitter. Will it work for the search engines? We'll have to wait and see what shows up in its results.
This article was originally published on Search Engine Watch.
Want to learn more?
Attend ClickZ Live New York March 30 - April 1. With over 15 years' experience delivering industry-leading events, ClickZ Live brings together over 60 expert speakers to offer an action-packed, educationally-focused agenda covering all aspects of digital marketing. Register today!
A member of the Professional Writer's Association of Canada, Miranda has authored more than 60 e-books, 300 client projects, and thousands of articles and blog posts for clients ranging from SMBs to government agencies and Fortune 100 companies.
Miranda studied e-commerce at Athabasca University and specializes in marketing, business and educational material. She currently assists the Province of Ontario Ministries of Research & Innovation and Economic Development, Trade & Employment with their copywriting and SEO goals. She is one of a handful of Canadian consultants experienced with Ontario's new adult literacy curriculum framework and as such, is contracted by literacy agencies and publishing houses to develop new learning material.
Singapore, 5-6 March
Bangkok, 17-18 March
Hong Kong, April 2015
A Buyer's Guide to Affiliate Management Software
Manage your performance marketing with the right solution. Choose a platform that will mutually empower advertisers and media partners!
Google My Business Listings Demystified
To help brands control how they appear online, Google has developed a new offering: Google My Business Locations. This whitepaper helps marketers understand how to use this powerful new tool.
March 3, 2015
1:00pm ET/10:00am PT