Navigation

  • About
  • Blog
  • About
  • Blog

Yahoo! Slurp crawl delay

February 11, 2005

You can control the minimum delay between accesses for Yahoo!’s Slurp crawler by adding a “Crawl-delay” line to your robots.txt.

User-agent: Slurp
Crawl-delay: 20

Thank you Yahoo! Search Blog!

  • Yahoo!
AboutBlog

© 2026 Niall Kennedy. All rights reserved.