# robots.txt to stop damn robots! # this should be active for all domains User-agent: * Disallow: /cgi-bin/ Disallow: /logs/ Crawl-delay: 3 Request-rate: 1/3 Visit-time: 2300-0700