UnburiedBot
UnburiedBot crawls a website when its owner, or someone they authorised, asks Unburied to audit it. It reads pages the way a search engine would, to find what keeps them from being found.
How it behaves
- It identifies itself as
UnburiedBot/1.0 (+https://unburied.io/bot). - It obeys robots.txt for the UnburiedBot token and for the
*group. - It waits between requests to the same host (250 ms by default) and keeps at most four requests open at once.
- It only fetches public pages. It never logs in, submits forms or buys anything.
To block it
Add this to your robots.txt:
User-agent: UnburiedBot Disallow: /