Cloudflare bot analytics for AI crawlers: a practical walkthrough

The setting that decides whether any of this works

Cloudflare only sees requests for a hostname that is proxied through it. In your DNS panel each record shows either an orange cloud, meaning proxied, or a grey cloud, meaning DNS only. A DNS-only record sends visitors straight to your origin and Cloudflare never sees the request at all.

Check this before anything else. A site whose records are grey will show an empty bot analytics view forever, and it looks identical to a site nobody visits. The number of hours lost to that confusion across the industry is considerable.

If your records are grey and you want this data, the alternative is your origin's own access logs, which contain exactly the same information in a less convenient shape.

Where the crawler data lives

In the dashboard, open your zone, then Security, then Bots. The overview separates likely automated traffic from likely human traffic and names the verified bots it recognises, which includes the major AI crawlers.

For anything beyond the summary, the Analytics section under your zone lets you filter traffic by user agent and by path. That combination — which crawler, which page — is the useful view, and it is the same distribution you would compute from a log file.

The retention window depends on your plan and it is shorter than most people assume. If you want a series longer than the window, you have to export before the data ages out, which is an argument for automating the pull rather than opening the dashboard monthly.

Pulling it out with the API instead

Cloudflare exposes analytics through a GraphQL endpoint at api.cloudflare.com/client/v4/graphql. The dataset for HTTP traffic is httpRequestsAdaptiveGroups, and it accepts a zone tag and a datetime window and returns grouped counts with the dimensions you request.

The two dimensions that matter here are userAgent and clientRequestPath, with count as the metric. That single query gives you the whole crawl attention distribution for a period in one call.

The token needs Zone then Analytics then Read. This is a different permission from the DNS token most people already have, and a token that manages your DNS perfectly well will return a permission error here. That mismatch is the second most common way this exercise stalls.

What the dashboard will not tell you

Bot verification tells you a request came from an address the vendor publishes. It does not tell you why the crawler came, what it did with the page, or whether any human ever saw the result.

Crawler activity is attention, not outcome. A page crawled daily by three assistants may send you nobody, and the dashboard has no way to show that because it cannot see your conversions.

It also cannot see the visits themselves when the assistant strips the referring page. Cloudflare records the request; it records no referring page because none was sent. So the same blind spot that affects your analytics affects this view too, for the same reason.

Turning it into a habit rather than a one-off

Pull the same query on a schedule and store date, crawler, path and count. That table is small, contains nothing personal, and answers every question anyone will ask of this data.

Watch for two changes rather than for the total. The first is a new crawler appearing, which tells you a new assistant has started reading you. The second is a shift in which pages get crawled, which tells you something changed in how the models see your site — usually before it shows up anywhere else.

Both are more informative than the headline count, and both are invisible if you only ever look at a monthly total.

Reconciling Cloudflare with your origin logs

The two will not match, and the difference is informative rather than alarming. Cloudflare counts requests at the edge, including those it served from cache and those it blocked. Your origin log counts what reached the server.

A large gap usually means caching is doing its job, which is good. It also means origin logs undercount crawler attention, sometimes severely, so a site behind a CDN that measures only at the origin will conclude assistants read it far less than they do.

If both sources are available, prefer the edge for crawl counts and the origin for anything requiring the full request. Use one consistently for any series you intend to compare over time.

And record which one a given figure came from. Two people measuring the same month from different layers will produce different numbers and both will be right, which is a difficult meeting to have without the label.

What to check before you trust the numbers

Confirm the hostname you are analysing is the one receiving traffic. A zone with several hostnames, some proxied and some not, produces partial data that looks complete.

Check the retention window against the period you want to compare. Discovering the data aged out is a common and entirely avoidable disappointment.

And confirm the user agent dimension is available on your plan. Some analytics dimensions are plan-gated, and a query that returns empty because a dimension is unavailable looks identical to a query that returns empty because nothing matched.

One final caution about comparing this with anything published. Different tools count crawler activity at different layers and with different definitions of a bot, so two figures for the same month can differ by a wide margin without either being wrong. If you cite a number, cite where in the stack it was measured, because that context is what makes it comparable to anything else.

And if the numbers still do not make sense after all of that, compare a single day at both layers before assuming either source is broken. One day of overlap usually explains a discrepancy that a month of aggregates only obscures.