If you added your website URL to Boei but the chatbot keeps saying it doesn't know anything about your business, there's a good chance Cloudflare is blocking Boei from reading your site.
This is one of the most common setup issues, and it's easy to fix. This guide walks you through it in about 5 minutes.
How to get there: Go to Tools → Chatbot in the top menu → click your chatbot → Website Crawl tab (under the Training section in the sidebar).
When you connect your website to Boei, Boei visits your pages and reads the content — just like Google does when it indexes your site. This is how your chatbot learns what products you sell, what questions you answer, and how to represent your business.
Cloudflare is a popular security service used by millions of websites. Its job is to keep out bots and automated visitors. The problem is that Boei's crawler looks like a bot to Cloudflare — because technically, it is one. A friendly one that you invited, but Cloudflare doesn't know that yet.
The result: Cloudflare blocks Boei before it can read a single page. Your chatbot ends up knowing nothing about your business.
The fix is simple: you tell Cloudflare to let Boei through.
Not sure if you're using Cloudflare? Here's a quick way to check:
You can also just try the fix below. If your site isn't on Cloudflare, it won't cause any harm.
Go to dash.cloudflare.com and log in. Click on your website domain from the list.
In the left sidebar, click Security, then click WAF.
Click the Custom rules tab, then click the blue Create rule button.
Give the rule a name, something like: Allow Boei chatbot crawler
Under If incoming requests match, set up the condition like this:
| Field | Operator | Value |
|---|---|---|
| User Agent | contains | BoeiBot |
What is a User Agent? It's how a visitor identifies itself to your website. Boei's crawler introduces itself as "BoeiBot" so that you (and Cloudflare) can recognize it.
Under Then take action, select Skip from the dropdown. Then check all the boxes that appear underneath, including:
This tells Cloudflare: when you see BoeiBot, don't apply any security checks, just let it through.
Click Deploy (or Save) in the top right corner. The rule is active immediately.
Once the rule is saved in Cloudflare:
Boei will now be able to read your pages. Within a few minutes, your chatbot will know your business.
A few things to check if it still fails after following the steps above:
You have Super Bot Fight Mode enabled. This is a more aggressive setting on paid Cloudflare plans. If you have it turned on, you may need to create the WAF rule slightly differently. Contact Boei support and we'll walk you through it.
Your hosting provider has its own firewall. Some website hosts (like WP Engine or Kinsta) add their own security layer on top of Cloudflare. If you're on one of these, you may need to whitelist BoeiBot there as well. Your hosting provider's support team can help.
Your site is password-protected or behind a login. Boei can only read pages that are publicly accessible. If your content is behind a login, you'll need to add it to Boei manually using the text or file upload option instead.
Some other security tools and hosting providers can also block crawlers. Common ones include:
If you use one of these, look for a "whitelist IP" or "allow bot" setting and add BoeiBot as an allowed user agent. If you're not sure how, reach out to Boei support and we'll help you figure it out.
Questions? Contact us via the Boei chat widget on boei.help or email us directly.