AI crawlers: what the WebRankInfo list changes for your traffic

WebRankInfo published on 3 November 2026 a list of the main crawlers used by AI engines, naming ChatGPT, Gemini, Perplexity, Claude, Mistral and others. The report gives site owners a reference point for either blocking those agents or making sure they can reach their pages. It does not specify how each crawler behaves, what user-agent strings or IP ranges it uses, or how the engines treat a page once it has been fetched.

Why the list matters before the strategy

Until now, most teams treated AI crawling as a single switch: allow everything or block everything in the robots file. A named list changes the granularity of the question. Each engine runs its own retrieval pipeline, and a page that is fetchable by one agent is not automatically fetchable by another. The practical consequence is that "we are visible to AI" is not a status you can assume, it is a set of per-agent checks.

The announcement does not say which engines rely on live crawling versus previously indexed data, nor whether blocking an agent has any effect on an existing index. Those gaps matter: a decision taken today may only show its effect after some delay, and the report offers no timeline.

The trade-off for acquisition and monetisation

Allowing AI crawlers is an acquisition decision. If an engine can read your content, it can cite it in an answer, and that citation is a referral channel that did not exist a few years ago. Blocking them protects content and, in some models, removes a source of unpaid reuse, but it also removes you from answers where a competitor may appear. Neither side of that trade-off comes with guarantees, and the list alone does not quantify either outcome.

For monetised sites, the question is sharper. Traffic arriving from an AI answer often lands on a single page, with little navigation and no ad exposure before the user leaves. A team that allows crawling without measuring those sessions is optimising for a number it cannot yet price.

What a team can check this week

• Identify which AI agents currently reach your site, using server logs filtered on the crawler names from the list.
• Compare the robots rules you have in place with the agents you actually want to admit, and write down the reason for each exception.
• Tag AI-referred sessions separately in your analytics so their conversion and retention can be compared with organic search.

These three checks take less than a week and produce a baseline. What the report leaves open, notably how each engine attributes a source and how often it re-crawls, is exactly what the logs will tell you over the following weeks.