Bing’s twitter search doesn’t honor robots.txt

Twitter robots

Edited 12.08 PM:
Jeroen has kindly pointed out to me that I was looking at the wrong robots file. Instead of www.twitter.com/robots.txt, I should have looked at twitter.com/robots.txt which does show that Twitter wants their content to get indexed and doesn’t say anything about Bing’s position on robots file. Thanks!

Twitter robots file

A few minutes ago, Bing – Microsoft’s own search engine announce that it’s providing a taste of real time search by introducing ‘@’  keyword to its search term list. By typing in @ronaldwidha for example you will see my Twitter timeline. However at the time of writing this post, Microsoft has only just started to index thousands of more prominent Twitterers.

What’s interesting to me is the fact that Twitter doesn’t want its contents to get indexed. As you can see from Twitter’s robots file above, they are disallowing any crawlers to index the site. I guess that doesn’t stop Bing from doing so.

I’ll be checking Bing closely today to try out the feature as soon as the roll out is completed.

source: Bringing a bit of twitter to bing