Allow all bots to visit all files and directories User-agent: * Disallow: Keep all bots out of the /cgi-bin/ and /private/ directories User-agent: * Disallow: /cgi-bin/ Disallow: /private/ Keep all bots from accessing /stats/ User-agent: * Disallow: /stats/ Keep all bots from accessing /psar/ User-agent: * Disallow: /psar/ Limit hits from all bots to at most once every two minutes User-agent: * Crawl-delay: 120