If two customers crawl the same website and it uses crawl-delay, how does it han... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		skybrian 24 days ago \| parent \| context \| favorite \| on: Cloudflare crawl endpoint If two customers crawl the same website and it uses crawl-delay, how does it handle that? Are they independent, or does each one run half as fast?

PeterStuer 23 days ago [–]

You put a governor on the domain, and you return from the cache instead.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact