Robot.txt

Robot.txt:

You should create robot.txt file following search engines guidelines. Here is GOOGLE DOCUMENTS on robot.txt file. It is advisable that you should go through it.