About onWebChatCrawler

Identity How to recognise it in your logs

Every request carries this User-Agent header:

onWebChatCrawler/2.0 (+https://www.onwebchat.com/crawler)

The token to use in robots.txt rules is onWebChatCrawler.

Requests from our previous crawler version carry the shorter onWebChatCrawler without the link, so one rule written for that token covers both.

One exception, stated here so nothing about our traffic is a surprise: if your server answers 403 Forbidden to that user agent, we retry the page once with a standard desktop browser user agent and then give up on it. We never do this for a page a robots.txt rule disallows, because a disallowed page is never requested in the first place.

Why Why the crawler visited your website

Somebody with an onWebChat account entered your address as the website their AI chatbot should learn from. In almost every case that is you or a colleague, setting up the chat widget on your own site. The text we read is used for one purpose only: answering the questions that visitors of that same website ask in its chat widget. We do not build a search index, we do not resell content, and we do not train public AI models with it.

If nobody on your team asked for this, block the crawler as described below, or contact us and we will stop the crawl and delete what was read.

Behaviour What it fetches, and how often

• robots.txt first: fetched before every crawl. Disallow rules for onWebChatCrawler or for * are honoured, and a Crawl-delay is respected up to 10 seconds.
• Only HTML pages of the website itself. Images, video, stylesheets, scripts and binaries are never downloaded. A page over 3 MB is dropped, and a request that takes longer than 15 seconds is abandoned.
• Page meta rules are honoured: a page marked noindex is not stored, and links marked nofollow are not followed.
• Two requests at a time, at most, spaced 300 to 800 milliseconds apart. One crawl reads 50 to 200 pages, depending on the customer's plan.
• Cheap to repeat: we store the ETag and Last-Modified of every page and send them back on later crawls, so pages that did not change answer 304 Not Modified and transfer no body at all.
• Only when asked: once when the site owner first sets up their chatbot, and again whenever they start a crawl from their dashboard.
• No forms, no logins, no checkout. The crawler only performs GET requests, never submits a form, never signs in, and never places an order.

Block How to keep the crawler out

Add this to your robots.txt and we will not request a single page:

User-agent: onWebChatCrawler
Disallow: /

To keep only part of the site out, list the paths instead:

User-agent: onWebChatCrawler
Disallow: /checkout/
Disallow: /my-account/

Changes take effect on the next crawl, because robots.txt is read at the start of each one. You can also block the user agent at your web server or CDN, or write to us and we will add your domain to our own block list.

Data What happens to the text

Readable page text is split into short passages and stored, together with the page address and title, in the account of the customer whose website it is. It is used to answer that website's own chat visitors, it is deleted when the customer deletes the training data or their account, and pages that disappear from the site are removed on the next crawl. Our privacy policy covers the details.