Robots.txt files (dont make the mistake of calling it robot.txt, in singular) are created by webmasters to disallow files and directories from being crawled and indexed by search engine spiders. A typical robots.txt would be,

User-agent: *
Disallow: /cgi-bin/

Tip

User-Agent: * - Although commands are not case sensitive, we advise you to write exactly "User-agent", that is all lowercase except for the capitalised "U".