SiteSherpaBot is the web crawler for SiteSherpa, an AI chat assistant that businesses add to their own websites. It reads a site so SiteSherpa can answer visitors' questions from that site's content.
When it visits
SiteSherpaBot only crawls a website after someone adds it to their SiteSherpa account. It reads that site's pages, starting from its sitemap or home page, and checks back for changes about once a week. It is not a general web crawler and does not build a search index.
How to recognize it
Requests from SiteSherpaBot use this user agent:
SiteSherpaBot/1.0 (+https://sitesherpa.ai/bot)
It makes no more than 2 requests per second to a site.
Controlling it with robots.txt
SiteSherpaBot follows robots.txt. It uses a group that names SiteSherpaBot if you have one, and otherwise the rules for *. Pages marked noindex, with a meta tag or an X-Robots-Tag header, are not added to SiteSherpa.
To keep SiteSherpaBot out of your whole site:
User-agent: SiteSherpaBot
Disallow: /A site owner using SiteSherpa can choose to ignore robots.txt for their own site, for example to include pages they keep out of search engines.
Questions or problems
If SiteSherpaBot is causing trouble on your site or you think it is crawling a site it shouldn't, email hello@sitesherpa.ai with your domain and we will look into it.