# Limit ScoutJet's crawl rate (example is to crawl # no more than 1 page every 10 seconds) User-agent: ScoutJet Crawl-delay: 10 Disallow: /~jhardin/private/ User-agent: * Disallow: /~jhardin/private/