What is a robots.txt File and How to Create One
Instructions are given in a robots.txt file in an effort to prevent spiders and other web robots from accessing and indexing all or certain parts of your web site. The robots.txt file is just a plain text file, uploaded to the web root directory of your web space. The filename must be all lowercase.
Please note that not all bots will respect the directives given in the robots.txt file. It should also never be relied upon as a way to keep parts of your site private. As the robots.txt file is available to anyone, it can be abused and used to reveal the areas of your account that you wish to remain “secret”. If the file or directory should be protected from general unauthorized access, password protect it.
A free robots.txt generator can be found at:
Robot Control Code Generation Tool
Even if you don’t wish to prevent a spider or bot from accessing any part of your web site, it is still a good idea to have a robots.txt file. If all directories and files on your site are open to indexing, specifying the following in your robots.txt file will send this message:
To validate your robots.txt file run it through an online validator such as the free Robots.txt Checker.