Chatwoot uses Firecrawl internally to fetch and index your website content. This allows the Chatwoot Assistant to answer questions using your site’s information.
However, if your website’s robots.txt has Disallow rules, Firecrawl will respect them and skip crawling those pages.
When that happens:
-
No documents from your website will be added to Chatwoot
-
The Assistant will not receive any additional context
-
Answers may become incomplete or generic
To ensure the Assistant works properly, you need to allow Firecrawl to crawl your website.
Step 1: Check Your robots.txt
Visit:
https://yourdomain.com/robots.txt
Look for any Disallow: entries that might block user agents.
Step 2: Add an Allow Rule for Firecrawl
Ask your website admin or developer to add the following lines to the robots.txt file:
User-agent: FirecrawlAgent
Allow: /
This tells Firecrawl that it is permitted to crawl your entire site, even if other bots are restricted.
Step 3: Save and Publish
After updating robots.txt, ensure the file is deployed and publicly accessible.
You can verify by visiting:
https://yourdomain.com/robots.txt
and checking that the new rules appear.