User-agent: CCBot Crawl-Delay: 20 User-agent: gsa-crawler Disallow: * User-Agent: * Crawl-Delay: 2 Disallow: /books/compcopy* Disallow: /books/textadoption* Disallow: /books/downloadSubjectCsv* Disallow: /journals/e-readers/processor/* Disallow: *.ctl Disallow: *.epl