AboutBlog

Navigation

  • About
  • Blog
  • About
  • Blog
Niall Kennedy's weblog

Yahoo! Slurp crawl delay

Publication date: February 11, 2005

You can control the minimum delay between accesses for Yahoo!’s Slurp crawler by adding a “Crawl-delay” line to your robots.txt.

User-agent: Slurp
Crawl-delay: 20

Thank you Yahoo! Search Blog!

Category:
  • Yahoo!
AboutBlog

Source: https://www.niallkennedy.com/blog/2005/02/yahoo-slurp-cra.html

© 2026 Niall Kennedy. All rights reserved.