GPTBot user agent: what it is and how to find it in your logs

The three OpenAI crawlers and what each is for

GPTBot fetches pages for model training. Its user agent contains the token GPTBot and a URL pointing at OpenAI's documentation, in a string of the form Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot.

OAI-SearchBot fetches pages to build the search index that answers use. If you care about appearing in ChatGPT's answers rather than about training data, this is the one that matters to you and blocking it has direct consequences.

ChatGPT-User fetches a page because a person asked ChatGPT about it in the moment. It is closer to a browser than to a crawler: there is a human waiting on the other end of that request.

Confusing the three is the most common mistake in this area, and it is expensive. A robots.txt rule written to keep your content out of training can block the crawler that decides whether you appear in answers at all.

Finding them in an access log

In a standard combined-format access log the user agent is the last quoted field. A single line looks like this: 203.0.113.10 - - [06/Aug/2026:10:12:44 +0000] "GET /pricing HTTP/1.1" 200 8123 "-" "Mozilla/5.0 compatible; GPTBot/1.2; +https://openai.com/gptbot".

To count hits by crawler, a single pass over the file is enough. Filter lines containing GPTBot and count them; repeat for OAI-SearchBot and ChatGPT-User. Match case-insensitively, because the surrounding boilerplate changes more often than the token does and casing has varied.

To see which pages are being fetched, extract the request path from the same lines and count by path. That distribution is the useful artefact: it tells you which of your pages the crawler considers worth returning to, which is a signal you cannot get any other way.

Verifying a request is genuinely OpenAI

A user agent is a claim, not proof. Anyone can send a request announcing itself as GPTBot, and scrapers do it precisely because sites treat the name as trusted.

OpenAI publishes IP ranges for its crawlers. Verification means checking the source address of a request against the published ranges, and the ranges change, so a check written once and never updated slowly stops working.

For most sites this matters less than it sounds. If you are counting crawler activity to understand which pages get attention, a forged request adds noise but changes little. If you are using the user agent to grant access to something you would otherwise protect, verify properly or do not use it as a gate at all.

What crawler activity actually tells you

Crawl frequency is a rough proxy for how much a page matters to the model behind it. A page fetched weekly is being kept current; a page fetched once and never revisited is not.

Crawl activity is also the only AI signal that survives referrer stripping. When an assistant sends visitors without a referring page, your analytics goes quiet while your logs keep showing the crawler reading you. If you want any measurement of AI attention for those assistants, the logs are where it lives.

What crawl activity does not tell you is whether anyone read the answer that resulted. A heavily crawled page can produce no visits at all. Treat it as attention rather than as outcome.

Before you decide to block anything

Blocking is a real option and it is not obviously wrong. If your content is your product and you do not want it in a training corpus, disallowing GPTBot is a coherent choice.

The trade is specific. Blocking GPTBot affects training. Blocking OAI-SearchBot affects whether you can appear in answers. Blocking ChatGPT-User affects a person who is asking about you right now. These are three different decisions and they deserve three different lines in your robots.txt.

Whatever you decide, record the date and the reasoning somewhere your successor will find it. Crawler policy set in a hurry and never revisited is how a site quietly disappears from a channel nobody was watching.

Turning a one-off grep into something useful

Counting crawler hits once answers a question and then goes stale. If crawl attention matters to you, the count needs to run on a schedule and land somewhere you will look at again.

The table worth keeping is four columns: date, crawler, path, hits. That is enough for every question anyone asks of this data, it contains nothing personal, and it stays small enough to keep for years without a retention conversation.

Watch for a new crawler token appearing rather than for the total moving. A vendor starting to read you is a real event with consequences; a twenty per cent move in monthly hits usually is not.

Also watch status codes. A crawler receiving 404 or 403 repeatedly on pages you care about is a problem you can fix in an afternoon, and it is invisible in every other report you have.

The question this data cannot answer

Crawl frequency tells you a model is paying attention to a page. It does not tell you whether that page is ever named in an answer, and the correlation between the two is weaker than intuition suggests.

A page can be crawled weekly and cited never. Another can be crawled rarely and named constantly, because the model learned it long ago and does not need to return.

If the question you care about is whether you appear in answers, crawl data is a leading indicator at best. The direct measurement is asking the questions and recording what comes back.

None of this requires special tooling. A text editor, a log file and twenty minutes will tell you which OpenAI crawlers read your site and which pages they return to, and that is genuinely most of the value available here.