ChatGPT, Claude, Perplexity and Google have long been reading your website, not just classic Google Search. Many site owners have therefore found a switch that Cloudflare has been offering for a while: “Block AI crawlers”. One click and the bots are out. Sounds sensible. For most company websites it is still the wrong decision. We explain why, and what to do instead, using our own domain.
Not every AI bot wants the same thing
The umbrella term “AI crawler” covers very different visitors:
- Answer bots fetch a page the moment someone asks ChatGPT, Claude or Perplexity a question. The answer usually names your site as a source, with a link. That is visibility, often even a visitor.
- Search bots build the index these answers draw from, much like Googlebot does for classic search.
- Training bots collect content to train future models. That brings no direct visitor, but in the long run a model may “know” your brand and your offer.
- Aggressive scrapers download a lot, never cite you and only put load on your server.
Lump them all together and block them, and you disappear from the answers of AI assistants. That is exactly where more and more people ask before they ever visit a website.
What we did on mignuti.com
We worked through the question for our own domain and made a deliberate choice: answer, search and training bots are allowed, a few aggressive bots with no visible benefit stay out. Our content is public marketing copy without personal data, and we want to be found, by AI too.
Along the way we noticed three things worth knowing beforehand:
- Cloudflare can override your robots.txt. The “Managed robots.txt” setting puts its own rules in front of your file. On the free plan this meant a fixed training ban we could not choose, plus conflicting rules. We switched it off. Since then the robots.txt we maintain ourselves applies.
- “Failed requests” are not always real AI bots. Cloudflare’s analytics showed unsuccessful requests under well-known bot names. A good share of them were scanners merely pretending to be known bots. Deriving new blocks from that blocks the wrong ones.
- Small gaps only show up in the analytics. AI assistants, for example, asked for a sitemap at the classic address, which did not exist in that form on our site. A single redirect fixed it.
What about Pay per Crawl?
Cloudflare now offers to let AI crawlers pay per request. Those who do not pay get “402 Payment Required” with a price instead of the page. That is interesting for sites with a lot of valuable content, such as specialist portals, publishers or large product databases. For a typical company website the amounts are small, and visibility in AI answers is usually worth more. The scheme is also still new. We would only use it where the numbers add up.
Our recommendation in four steps
- Look first, then block: which bots really come, which succeed, which do not?
- Set your goal: do you want to appear in AI answers? Is there content you want to protect or sell?
- One source for the rules: either your robots.txt or Cloudflare’s managed file, not both against each other.
- Think of your own services: a website chatbot, monitoring or an SEO tool crawls your site too. A blanket block takes them down as well. For our own chatbot crawler we openly explain how to allow or block it.
If you would rather not sort this out yourself: we check your website, recommend a line and set it up. More on our page AI Crawlers & AI Visibility.