You can control the minimum delay between accesses for Yahoo!’s Slurp crawler by adding a “Crawl-delay” line to your robots.txt.
User-agent: Slurp
Crawl-delay: 20
Thank you Yahoo! Search Blog!
Yahoo! Slurp crawl delay
You can control the minimum delay between accesses for Yahoo!'s Slurp crawler by adding a "Crawl-delay" line to your robots.txt. User-agent: Slurp Crawl-delay: 20 Thank you Yahoo! Search Blog!...
Feed
Niall Kennedy is a web technologist in San Francisco, California in the United States. I am very interested in the world of... MORE »