Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Yes, but not at a rate that enables them to be a risk to your hosting bill. My understanding is that the goal here isn't to prevent crawlers, it's to prevent overly aggressive ones.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: