http://www.ronaldwidha.net/2009/07/02/bing-twitter-search-doesnt-honor-robots-txt/ podcast addict, believes in occam razor, software consultant Mon, 15 Mar 2010 20:24:58 +0000 http://wordpress.org/?v=2.9.2 hourly 1 http://www.ronaldwidha.net/2009/07/02/bing-twitter-search-doesnt-honor-robots-txt/comment-page-1/#comment-344 Jeroen van Eck Thu, 02 Jul 2009 07:19:46 +0000 http://www.ronaldwidha.net/?p=250#comment-344 As you probably know twitter uses the non-www URL's as main domain. You're showing the www. version of robots.txt. Now check http://twitter.com/robots.txt and you'll see that Twitter does want it's content to get indexed. As you probably know twitter uses the non-www URL’s as main domain. You’re showing the www. version of robots.txt. Now check http://twitter.com/robots.txt and you’ll see that Twitter does want it’s content to get indexed.

]]>