Unburied

UnburiedBot

UnburiedBot crawls a website when its owner, or someone they authorised, asks Unburied to audit it. It reads pages the way a search engine would, to find what keeps them from being found.

How it behaves

To block it

Add this to your robots.txt:

User-agent: UnburiedBot
Disallow: /