Robots.txt

The robots.txt file is a file that good bots will look at before spidering your site. It just tells them what not to spider and in some cases not to spider your site at all.

User-agent: *
Disallow: /*?
Disallow: /*.txt$
Disallow: /400.htm
Disallow: /401.htm
Disallow: /403.htm
Disallow: /404.htm
Disallow: /500.htm
Disallow: /banned.htm
Disallow: /cgi-bin/
Disallow: /downloader/

user-agent: Accoona-AI-Agent
Disallow: /

The first section "User-agent: *" means ALL spiders, and then is followed up by lines telling the spider which directories not to index. Then the last two lines are agent specific and tell the "Accona-AI-Agent" spider not to spider my site at all.

Just one small tip on your robots.txt file: If you have a directory that is secret, for example an control panel or something of that sort that is in no way linked to your site then DO NOT add it. One little trick is that you can often pull up a sites robots.txt and you may find listed directories that they have listed because they want to keep them private. Well as a hacker those are like road signs pointing you in the direction you need to go. So again if it is not linked to from your site then you don't need to add it to the robots.txt file.