Battling the pink Robots.txt


Just sorted out an interesting problem where a site I was asked to come up with a major over haul for – had a problem with its robots.txt (a robots.txt file is used to block search engines form areas of a website that they should not go into normaly admin areas and the like)

Turns out they where trying to some complicated rewriting and had mistakenly blocked all files with php in it yes they had blocked all the dynamic pages including the home page no wonder the Google advanced search result looked so odd.

 So children the Moral of the tale is know what your doing with robots.txt and test test test!

One thought on “Battling the pink Robots.txt

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s