Instructions are given in a robots.txt file in an effort to prevent spiders and other web robots from accessing and indexing all or certain parts of your web site. The robots.txt file is just a plain text file, uploaded to the web root directory of your web space. The filename must be all lowercase.

Please note that not all bots will respect the directives given in the robots.txt file. It should also never be relied upon as a way to keep parts of your site private. As the robots.txt file is available to anyone, it can be abused and used to reveal the areas of your account that you wish to remain “secret”. If the file or directory should be protected from general unauthorized access, password protect it.

Some examples of robots.txt files:
http://www.w3.org/robots.txt
http://www.nytimes.com/robots.txt
http://www.google.com/robots.txt

A free robots.txt generator can be found at:
Robot Control Code Generation Tool

For more information on what a robots.txt file is and how to write your robots.txt file, visit these links:
robots.txt File
Working with the robots.txt File

Even if you don’t wish to prevent a spider or bot from accessing any part of your web site, it is still a good idea to have a robots.txt file. If all directories and files on your site are open to indexing, specifying the following in your robots.txt file will send this message:

User-agent: *
Disallow:

To validate your robots.txt file run it through an online validator such as the free Robots.txt Checker.