How to read server logs for AI crawler activity

What a log line contains

The default format for both nginx and Apache is the combined log format, and it has a fixed shape: client address, two fields almost nobody uses, a bracketed timestamp, the quoted request, the status code, the response size, the referring page in quotes, and the user agent in quotes.

The two fields you care about for this exercise are the request path and the user agent. The client address matters only long enough to distinguish automated traffic from human traffic, and it should not survive into anything you keep.

CDNs emit JSON instead. Cloudflare Logpush names the same things ClientRequestURI, ClientRequestUserAgent, ClientRequestReferer and EdgeStartTimestamp; Vercel uses path, userAgent, referer and timestamp. The field names differ, the content does not, and a single alias table handles both rather than a parser per vendor.

Which user agents to look for

The tokens worth matching are short and stable. For OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User. For Anthropic: ClaudeBot and Claude-User. For Perplexity: PerplexityBot and Perplexity-User. Google's training crawler is Google-Extended. Microsoft's is Bingbot, which feeds Copilot. Meta's is meta-externalagent.

Match on the token alone rather than on the full string, and match case-insensitively. Vendors rewrite the surrounding boilerplate regularly and the token almost never changes, so a match on the full string breaks quietly on the day a version number moves.

Note the pattern in the names. Several vendors run one crawler for training and a separate one that fetches because a person asked. Counting them together loses the distinction between a corpus being built and a human waiting for an answer.

Counting the two things worth counting

The first is hits per crawler per page. This is the crawl attention distribution and it is the most useful thing in the file: it shows which of your pages the models keep returning to.

The second is human visits that carried an AI referring page. Filter to lines whose referring page is one of the assistant hosts and whose user agent is an ordinary browser. Keep these strictly separate from crawler hits — a request from a known bot is a crawl regardless of what referring page it carries, and mixing them inflates both figures.

A useful third pass is status codes for crawler requests. A crawler repeatedly receiving 404 or 403 on pages you care about is a problem you can fix in an afternoon, and it is invisible in every other report you have.

Strip before you keep, not after

Access logs contain personal information inside URLs — password reset tokens, invitation links carrying email addresses, session identifiers in query strings. That is fine while the file sits on the server under its normal retention policy. It stops being fine the moment you copy it into a spreadsheet, a dashboard or a third-party tool.

Redact at the point of extraction rather than afterwards. Drop every query parameter except an allowlist of campaign and source tags — the utm_ family, and little else. Use the client address to separate bots from humans and then discard it rather than carrying it into the aggregate.

And set a rule for what you do when a URL cannot be made safe, such as a path containing an email address. Rejecting the file and saying so is the defensible answer. Silently dropping the line understates your figures without telling you, which is the worse failure of the two.

Turning it into something you will actually look at

A one-off analysis answers a question once and then rots. If crawler activity matters to you, the extraction needs to run on a schedule and write somewhere durable.

The shape worth keeping is small: date, crawler, page, hits. That is enough for every question anyone asks of this data, and it contains nothing personal, which means the retention conversation is short.

Keep a handful of redacted sample lines alongside the totals. When somebody asks where a number came from, being able to show three real lines is the difference between a figure that is believed and one that is argued about.

When logs are the only signal you have

For assistants that strip the referring page, your logs are not one source among several — they are the only place any evidence exists. Analytics shows nothing, because nothing was sent.

That changes how much weight the crawl distribution deserves. On a site where a stripping assistant is a meaningful reader, the pages that crawler returns to are the closest thing to a visibility measurement available, and no amount of analytics configuration substitutes.

It also means log retention becomes a measurement decision rather than an operations one. A thirty-day rotation is fine for debugging and useless for seeing a trend, and by the time you want the trend the data is gone.

Extract early and keep the aggregate rather than extending raw retention. Four columns of counts carries the signal, contains nothing personal, and can be kept indefinitely without anyone needing to think about it again.

A minimal implementation that works

Read the file line by line, match the request path and user agent, drop everything else. Count hits by crawler and path into a map, and write the totals with the date attached.

Redact as you parse rather than afterwards. Strip query parameters that are not campaign tags, use the client address to classify the request and then discard it, and refuse the file entirely if a path contains something you cannot make safe.

Keep a handful of redacted sample lines beside the totals. When somebody questions a number, three real lines settle it faster than any explanation.

Everything above works with tools already on the server, and none of it requires a purchase to get started.