# Sample robots.txt file 2 User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler [c01] Disallow: / User-agent: CRAWLER Disallow: / User-agent: ia_archiver Disallow: /