Twitter made a change to its robots.txt file that will allow search engines like Google to crawl and index Twitter search results pages.
Twitter has updated its robots.txt file to allow search engines to crawl more of the site. Specifically, Google, Bing, Yandex, and others will now have access to Twitter's search results pages, which display Promoted Tweets where available.
The change will allow search results pages to show up in Google's index, though user search is still disallowed. For example, the results page for a search on the Olympics becomes indexable and therefore could show up in Google search results to users seeking information on the event.
With the new indexing direction, searches for keywords like "Ford" could potentially direct Google/Bing searchers to Twitter's internal search for the term. This is different from the way real-time search operated in that users were apt to see actual tweets on the topic displayed in Google's results. They would see Google's ads on the search results page, then click on a Twitter result and go straight to a user page, where there are no ads. With this change, the results appearing in search engines would direct the user to a Twitter search results page, which displays a Promoted Tweet embedded in the feed (when available).
Bing already has firehose access, so it can use real-time tweets in its search results. Google used to have this level of access. Then things between the two went sideways.
Twitter and Google have had a seemingly frosty relationship since their firehose deal ended last year. Twitter had provided access to real-time tweets until the deal ended in July 2011. In October, CEO Dick Costolo spoke with John Battelle at Web 2.0 Summit and hinted that the issue keeping the two from reaching an agreement was not a monetary one.
The timing begged the question, did the deal end because Google launched its own social network? Costolo said then that Google+ and Twitter were competing in different arenas; Google in features and Twitter in simplicity.
"Twitter has emerged as a vital source of this real-time information. As we've seen time again, news breaks first on Twitter. As a result, Twitter accounts and Tweets are often the most relevant (search) results. We're concerned that, as a result of Google's changes, finding this information will be much harder for everyone. We think that's bad for people, publishers, news organizations and Twitter users."
Google responded with a statement on its Google+ Page:
"We are a bit surprised by Twitter's comments about Search plus Your World, because they chose not to renew their agreement with us last summer, and since then we have observed their rel=nofollow instructions."
Twitter's instructions now tell Google it's OK for it to crawl and index more of the site. But will Google? It's difficult to decipher what actually happened at the table between Google and Twitter; we may never actually know the real reason they chose not to work together in that capacity any longer. It seems the deal itself was a huge missed opportunity for Twitter, in that users were directed around seeing its ads.
This change seems designed to address that. Now that Twitter has beefed up its own search a bit, is it hoping the search engines can help it direct traffic only to areas of the site it chooses, where the potential for ad revenue exists?
Twitter doesn't have sidebar ads, so having search engines crawl and index tweets may draw in users, but doesn't make them any money. It encourages people to look outside of Twitter for Twitter content. However, directing people only to those monetized places is an option that makes a lot of sense for Twitter. Will it work for the search engines? We'll have to wait and see what shows up in its results.
This article was originally published on Search Engine Watch.
On the heels of a fantastic event in New York City, ClickZ Live is taking the fun and learning to Toronto, June 23-25. With over 15 years' experience delivering industry-leading events, ClickZ Live offers an action-packed, educationally-focused agenda covering all aspects of digital marketing. Register today!
A member of the Professional Writer's Association of Canada, Miranda has authored more than 60 e-books, 300 client projects, and thousands of articles and blog posts for clients ranging from SMBs to government agencies and Fortune 100 companies.
Miranda studied e-commerce at Athabasca University and specializes in marketing, business and educational material. She currently assists the Province of Ontario Ministries of Research & Innovation and Economic Development, Trade & Employment with their copywriting and SEO goals. She is one of a handful of Canadian consultants experienced with Ontario's new adult literacy curriculum framework and as such, is contracted by literacy agencies and publishing houses to develop new learning material.
Hong Kong, May 5-6, 2015
Gartner Magic Quadrant for Digital Commerce
This Magic Quadrant examines leading digital commerce platforms that enable organizations to build digital commerce sites. These commerce platforms facilitate purchasing transactions over the Web, and support the creation and continuing development of an online relationship with a consumer.
Paid Search in the Mobile Era
Google reports that paid search ads are currently driving 40+ million calls per month. Cost per click is increasing, paid search budgets are growing, and mobile continues to dominate. It's time to revamp old search strategies, reimagine stale best practices, and add new layers data to your analytics.
May 6, 2015
12:00pm ET/9:00am PT