youcanchatbot — the youcan.chat crawler

youcanchatbot reads public pages of websites that youcan.chat customers have added to their account. The customer’s AI representative uses the text of these pages to answer visitors in the customer’s chat.

How the crawler works

It identifies itself in every request as:

• It reads only public pages: it doesn’t log in, doesn’t submit forms and doesn’t run scripts.
• It follows robots.txt, including Crawl-delay (up to 10 seconds). It waits at least one second between requests to the same site.
• It rechecks a site once a week or once a day, depending on the customer’s plan. A customer can start a check manually no more than once every 3 hours.
• It downloads a page again only if it has changed: it asks the server first, using ETag and Last-Modified.
• It doesn’t read files linked from pages (PDF and others) and doesn’t download pages larger than 5 MB.

How to block the crawler

Add this to your site’s robots.txt:

User-agent: youcanchatbot
Disallow: /

The crawler will read the rule on its next check and will stop downloading the site’s pages. You can also block only part of the site — put the path in Disallow.

Complaints and questions

If someone else added your site to youcan.chat without your consent, or the crawler interferes with your site, write to us and include the site’s address. We’ll review your request and remove your site’s pages from the customer’s data.