You can control the minimum delay between accesses for Yahoo!’s Slurp crawler by adding a “Crawl-delay” line to your robots.txt.
User-agent: Slurp
Crawl-delay: 20
Thank you Yahoo! Search Blog!
Niall Kennedy's home on the web
You can control the minimum delay between accesses for Yahoo!’s Slurp crawler by adding a “Crawl-delay” line to your robots.txt.
User-agent: Slurp
Crawl-delay: 20
Thank you Yahoo! Search Blog!