AI crawler user agents
Every AI crawler we track, what each one is for, and an example log line you can match against your own access log. This page renders from the same file our engine matches against, so it does not go stale independently of the product.
Most operators run more than one crawler and they do different jobs. One builds training corpora, one builds the search index behind answers, and one fetches a page because a person just asked about it. Blocking them is three separate decisions, not one — see how to block or allow AI crawlers.
Dataset version 2026-08-06, updated 2026-08-06. 10 records.
| User agent token | Operator | Purpose | Added | Source |
|---|---|---|---|---|
GPTBot | GPTBot | Training | 2026-08-06 | Vendor docs |
OAI-SearchBot | OAI-SearchBot | Search index | 2026-08-06 | Vendor docs |
ChatGPT-User | ChatGPT-User | Fetched for a person | 2026-08-06 | Vendor docs |
ClaudeBot | ClaudeBot | Training | 2026-08-06 | Vendor docs |
Claude-User | Claude-User | Fetched for a person | 2026-08-06 | Vendor docs |
PerplexityBot | PerplexityBot | Search index | 2026-08-06 | Vendor docs |
Perplexity-User | Perplexity-User | Fetched for a person | 2026-08-06 | Vendor docs |
Google-Extended | Google-Extended | Training | 2026-08-06 | Vendor docs |
Bingbot | Bingbot | Search index | 2026-08-06 | Vendor docs |
meta-externalagent | Meta External Agent | Training | 2026-08-06 | Vendor docs |
Example log lines
Each line below is in the combined log format that nginx and Apache emit by default. The user agent is the last quoted field. Match on the token rather than the whole string: operators rewrite the surrounding text far more often than they change the token.
- GPTBot
203.0.113.10 - - [06/Aug/2026:10:12:44 +0000] "GET /pricing HTTP/1.1" 200 8123 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot"- OAI-SearchBot
203.0.113.11 - - [06/Aug/2026:10:13:02 +0000] "GET /docs HTTP/1.1" 200 4410 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot"- ChatGPT-User
203.0.113.12 - - [06/Aug/2026:10:14:19 +0000] "GET /blog/post HTTP/1.1" 200 9902 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"- ClaudeBot
203.0.113.20 - - [06/Aug/2026:11:01:07 +0000] "GET /features HTTP/1.1" 200 7331 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com"- Claude-User
203.0.113.21 - - [06/Aug/2026:11:02:55 +0000] "GET /changelog HTTP/1.1" 200 3120 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Claude-User/1.0; +Claude-User@anthropic.com"- PerplexityBot
203.0.113.30 - - [06/Aug/2026:12:20:11 +0000] "GET / HTTP/1.1" 200 15220 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot"- Perplexity-User
203.0.113.31 - - [06/Aug/2026:12:21:48 +0000] "GET /pricing HTTP/1.1" 200 8123 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user"- Google-Extended
203.0.113.40 - - [06/Aug/2026:13:05:00 +0000] "GET /guide HTTP/1.1" 200 6600 "-" "Mozilla/5.0 (compatible; Google-Extended/1.0; +http://www.google.com/bot.html)"- Bingbot
203.0.113.50 - - [06/Aug/2026:14:00:31 +0000] "GET /sitemap.xml HTTP/1.1" 200 1042 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"- Meta External Agent
203.0.113.60 - - [06/Aug/2026:15:40:12 +0000] "GET /about HTTP/1.1" 200 2210 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
Changelog
- 2026-08-06 — Initial registry: ten AI crawler user agents across seven engines.
A user agent is a claim rather than proof: anything can announce itself as any crawler. Where a vendor publishes address ranges, the verification link in the table above points at them. For counting attention a forged request adds noise you can live with; for granting access it does not.