# see # http://web.nexor.co.uk/mak/doc/robots/norobots.html # http://info.webcrawler.com/mak/projects/robots/robots.html # http://www.robotstxt.org/wc/robots.html User-agent: * #Disallow: / Disallow: /cgi-bin Disallow: /recursion Disallow: /images Disallow: /manual Disallow: /icons #Disallow: /hbvar #Disallow: /depot # # These may not work (docs say no wildcards or regexps) Disallow: */cgi-bin Disallow: */images Disallow: */usr Disallow: */bin Disallow: */lib Disallow: */src Disallow: */home Disallow: */dev/ Disallow: *hbvar* # test #User-agent: Slurp #User-agent: Arachnophilia #User-agent: webvac #Disallow: /