How to block or allow AI crawlers, and the trade-off nobody states

The distinction that decides everything

Most vendors run more than one crawler and they do different jobs. OpenAI runs GPTBot for training, OAI-SearchBot to build the index behind answers, and ChatGPT-User to fetch a page because a person just asked about it. Anthropic and Perplexity have the same split. Google separates Google-Extended, which governs training use, from Googlebot, which governs search.

This matters because the usual instinct — write one rule, block the lot — produces a result almost nobody actually wants. Keeping your content out of a training corpus is one decision. Removing yourself from the answers people receive is a completely different one, and a single wildcard rule does both.

The syntax, written out

robots.txt lives at the root of your domain and is a plain text file. A rule to keep content out of OpenAI training while remaining eligible for answers looks like this: a User-agent line naming GPTBot, followed by a Disallow line with a single forward slash. Then a second block with a User-agent line naming OAI-SearchBot, followed by an Allow line with a single forward slash.

The equivalent for Google is a block naming Google-Extended with Disallow, which affects training use, while leaving Googlebot untouched so ordinary search continues.

Two things to know about the file. Rules are matched by user agent token, so casing and exact spelling matter. And robots.txt is a request rather than a control — well-behaved crawlers honour it and badly behaved ones ignore it entirely, which is why it is a policy statement rather than a security measure.

Enforcing it when a request ignores the file

If you need something stronger than a request, the enforcement point is your edge. In Cloudflare that means a firewall rule matching the user agent and returning a block, or the managed bot controls if your plan includes them.

At the origin, nginx and Apache can both match a user agent and return 403. This is more brittle than it looks: the match is on a string the client controls, so it stops anything honest and nothing determined.

For real enforcement the check has to be on the source address against the vendor's published ranges, not on the name the request announces. That is more work and it is the only version that actually holds.

The trade-off, stated plainly

Blocking training crawlers protects your content from being absorbed into a model you do not control, and for a publisher whose archive is the business that is a coherent position with real value behind it.

Blocking search crawlers removes you from the answers. If a buyer asks an assistant for options in your category and your site is not in the index, you are not among the options. There is no partial version of this outcome.

Blocking user-request crawlers is the one people regret fastest, because there is a person on the other end of that request asking about you specifically. A block returns them nothing.

The costs are asymmetric and they arrive at different times. Training exposure is a slow, diffuse cost. Absence from answers is an immediate, specific one, and it is invisible in your analytics because traffic that never happens leaves no trace.

A defensible default

For most businesses that sell something, allowing search and user-request crawlers while deciding separately about training is the position that survives scrutiny. It keeps you eligible to be named and leaves the training question as its own decision.

For publishers whose archive is the product, the calculus genuinely differs, and blocking training crawlers while remaining searchable is a reasonable place to land.

Whatever you choose, record the date and the reasoning in a comment in the file itself. Crawler policy is set once in a hurry and then inherited by people who have no idea why, and that is how a site quietly disappears from a channel nobody was watching.

Reviewing a policy you inherited

Most robots.txt files in production were written by someone who has left, for reasons nobody recorded. Before changing anything, read what is actually there and work out which of the three crawler types each rule affects.

Look specifically for a wildcard disallow aimed at AI, because that is the pattern most likely to be blocking the search crawler along with the training one. It is also the pattern most likely to have been added quickly during a period of concern about training data.

Check the effect rather than the intent. Your server logs show which crawlers still fetch pages and which stopped, and a rule that has been silently failing for a year looks identical to one working perfectly until you look.

Then write the decision down in a comment in the file, with the date. The next person to inherit this deserves better than what you inherited, and the cost is one line.

A position that is easy to defend later

Allow the crawlers that decide whether you appear in answers, and treat the training question separately and explicitly. That combination is defensible to a board, to a legal team and to yourself in a year.

Document the decision beside the rules with a date and one sentence of reasoning. Policy without recorded reasoning is policy that gets reversed by whoever is least informed.

And schedule a review. Vendors add crawlers, split existing ones and rename them, and a file that was correct when written drifts out of date without any visible symptom.

The one thing not to do is decide by default. A file nobody has read in two years is still a policy, and it is being enforced whether or not anyone intended it.