What Server Logs Reveal About AI Crawlers

01Analytics cannot see any of this

Start with the fact that makes log analysis unavoidable rather than optional. AI bots do not execute JavaScript. Client-side analytics works by firing a JavaScript tag. Therefore Google Analytics, Plausible and every similar tool record essentially none of your AI crawler traffic.

That is not a reporting gap, it is a blind spot covering the entire channel. A practitioner running self-hosted log analytics put the scale at roughly 40% of traffic invisible to client-side tools, and a founder who pulled logs for a side project found 38% of all traffic was bots and crawlers, with 30 distinct crawlers hitting the site in a twelve hour window.

So before buying any AI visibility tracker, there is a cheaper question worth answering: can the crawlers physically reach and read your pages at all. A tracker will report that you are invisible. Logs tell you why.

Two public datasets underpin this article: a 48-day analysis of server logs shared in r/TechSEO, and a controlled 41-day link architecture experiment shared in r/OrganicTrafficLab. Both are single-site studies by individual practitioners, not peer-reviewed research, and we flag the limits of that in the final section.

02The JavaScript experiment, and why it is the important one

The 41-day experiment is the most useful piece of public evidence we have found on AI crawlability, because it is properly controlled rather than observational.

The setup: a directory site of roughly 2,400 pages, 21 sections split in half. One half had every internal link hard-coded in HTML. The other half had links injected by JavaScript after load. Sitemaps were disabled and breadcrumbs removed, so rendering the parent page was the only possible route to a JavaScript-linked page. Every bot request was logged, with Googlebot and Bingbot verified by reverse DNS to filter spoofed agents.

That design matters. Because every alternative discovery path was closed, a hit on a JavaScript-linked page is only possible if the bot actually executed the script. The results after 27 days:

Five crawlers scored a flat zero. Bingbot recrawled the ten JavaScript section pages on 18 of its 20 active days and never once followed an injected link, with the three pages it did reach looking like side-door discovery rather than rendering.

The line worth carrying out of this: the pages that never get past discovery are not ranking slowly. They are not there at all.

03What fixing it later actually costs

On day 27 the experimenter converted every JavaScript link to plain HTML. Same URLs, same content, same server, and kept logging. The recovery pattern is arguably more useful than the original finding.

Two asymmetries stand out. AI crawlers bounce back fast when link architecture is fixed, with GPTBot sweeping 250 pages in under three hours. Google’s indexing crawler does not, having spent its first-crawl budget on a fresh domain. Getting discovered late is expensive in a way that is hard to walk back.

ClaudeBot is the odd one out and worth watching: it kept hitting the site with hundreds of requests but only pages it already knew, suggesting whatever map it drew on the first pass stayed sticky for weeks. If that generalises, a bad first impression with Claude costs more than with OpenAI.

One nuance the experimenter flags honestly: the JavaScript penalty is entirely front-loaded in discovery. Once GoogleOther found a JavaScript page, it recrawled it as fast as comparable HTML pages, at two of three depth levels 24 to 27% faster. Discovery is the wall, not what happens afterwards.

04robots.txt is a request, not enforcement

The 48-day log study reports behaviour that should change how you think about blocking. GPTBot and Meta-WebIndexer never checked robots.txt at all during the measurement period.

The inverse case is equally strange. Bytespider and CCBot checked robots.txt repeatedly and then never crawled anything, consuming your directives without following through and generating compliance overhead with zero content indexing.

The practical conclusion is unglamorous. If you want to allow crawlers, robots.txt is a reasonable signal. If you genuinely need to block one, do it at the firewall or edge, because a directive that two of the most active crawlers never read is not a control.

Whether you should block at all is a separate and contested question. A r/webdev thread that drew over two thousand upvotes captured the frustration behind it, from a developer who found ClaudeBot had hit their site roughly 881,000 times in a single day:

That is a real cost and a legitimate grievance. It also cuts against AI visibility: blocking a retrieval crawler removes you from the answers you are paying a tracker to measure. The resolution most teams land on is to block training crawlers where bandwidth genuinely hurts, and keep retrieval and user-session agents open, which requires knowing which is which.

05Three bots per company, and most robots.txt files know about one

OpenAI and Anthropic each operate three separate bots: one for training and indexing, one for search, and one for live user sessions. Blocking one does not block the others.

This is the most common configuration error we see discussed. A team decides to block AI crawlers, adds a GPTBot directive, and leaves both the search retrieval bot and the live user agent untouched, achieving neither the bandwidth saving they wanted nor the visibility they assumed they kept.

The bot landscape is also wider than the familiar names. Alongside the majors, one log study names PromptingBot, LinkupBot, Brightbot and Observer as actively crawling, and a founder’s log pull listed 30 distinct crawlers in twelve hours including DeepSeekBot, xAI-SearchBot, Amzn-SearchBot and PetalBot. Any allowlist built from memory will be incomplete.

06ChatGPT-User is the most valuable line in your logs

Of every agent in this article, one deserves its own alert. ChatGPT-User requests represent real people, in live conversations, pulling your page into ChatGPT.

That makes it the closest thing to observed AI citation data available without buying anything. It is not a crawler building an index. It is measurable word of mouth.

The supporting detail is convincing. In one dataset ChatGPT-User showed a near one-to-one IP-to-request ratio consistent with individual user sessions, spread across 15 countries and 584 unique IPs, while GPTBot operated from just 2 IPs consistent with centralised infrastructure. IP patterns distinguish genuine user-triggered fetches from automated crawling.

Two behavioural notes matter for what you publish. ChatGPT-User extracts text only, fetching zero images, CSS or JavaScript, which means your HTML content is what reaches the conversation and visual design contributes nothing. And the top ChatGPT-User pages in that dataset were all implementation guides and technical explainers, suggesting deep specific content earns these fetches more than broad overview content does.

If you do one thing after reading this, log ChatGPT-User separately and track which URLs it pulls. That list is a live signal of which pages are actually entering AI conversations, and it complements the simulated data every tracker sells, which we cover in our ChatGPT tracker analysis.

07The llms.txt finding, stated plainly

Across 48 days of server logs, zero AI bots requested /llms.txt.

That is one site over seven weeks, so it is not proof of universal behaviour. It is, however, direct observational evidence against a practice currently being recommended widely, and it is the kind of evidence nobody selling llms.txt optimisation seems to publish.

The reasonable position is that llms.txt may become a standard and costs almost nothing to publish, so there is no strong reason to remove one. There is also no measured basis for prioritising it over link architecture, which the JavaScript experiment shows has an enormous and immediate effect. If you are choosing between an afternoon on llms.txt and an afternoon converting JavaScript navigation to HTML, the logs say the second one is not close.

We looked at the same question from the content side in does llms.txt actually work, and the log evidence here is consistent with that analysis.

One related finding cuts the other way and is worth acting on. GPTBot and ClaudeBot both began consuming sitemaps in March 2026 for the first time in that dataset. If your sitemap is stale, incomplete or missing language variants, AI crawlers will miss content, which makes sitemap hygiene a higher-return task than it was a year ago.

08Bursts, timing and what they do to your server

AI crawlers do not behave like Googlebot’s polite steady trickle. They arrive in bursts, and that has operational consequences.

One measurement recorded GPTBot hitting 114 requests per minute inside a three minute window. If your server cannot absorb burst traffic, AI crawlers hit errors during exactly the indexing runs you want to succeed, and a 503 served to a retrieval crawler is a page that does not exist as far as the answer is concerned.

Timing differs by bot as well. In that dataset GPTBot concentrated around 04:00 UTC, Claude-SearchBot peaked overnight, and PerplexityBot burst at 23:00, 05:00 and 09:00. If you deploy during off-peak US hours, AI bots may be the first visitors to see a change, including a broken one.

Volume can also step-change without warning. GPTBot went from 0 to 187 requests inside a single week in the same dataset, and coordinated events occur across bot families, with GPTBot and OAI-SearchBot firing simultaneously from the same Microsoft infrastructure on one recorded day. Capacity planning that assumes gradual growth will be wrong.

09The volume problem, and the uncomfortable trade

The bandwidth cost is not hypothetical. The r/webdev report of roughly 881,000 ClaudeBot requests in a day drew 258 comments precisely because it is a common experience, and Meta-WebIndexer was described as the most aggressive crawler by volume in the 48-day dataset, with zero robots.txt checks.

The trade is genuinely uncomfortable and worth naming rather than side-stepping. Serving these crawlers costs you money and returns nothing directly. Blocking them removes you from AI answers, which is the visibility most marketing teams are now actively trying to buy.

The defensible middle is agent-level discrimination, which is only possible once you have read your logs. Keep search retrieval and user-session agents open because those directly feed the answers buyers see. Rate-limit or block training crawlers at the edge when volume is genuinely hurting. Do not make that decision from a blog post, including this one, because the right answer depends on your traffic profile and hosting costs.

There is also a multilingual angle worth knowing if you publish translations. In the 48-day dataset, Meta-WebIndexer spent 80% of its crawl budget on language variants, GPTBot 62% and Bingbot 60%. Translated content is being indexed aggressively, which makes it a higher-leverage asset for AI visibility than its organic traffic alone would suggest.

10How deep research modes read a site

A common question is whether the extended research modes in ChatGPT and similar products behave differently from ordinary retrieval. The log evidence points at a specific pattern rather than a different technology.

What logs show is many sequential fetches originating from user-session agents rather than a single page pull, consistent with a system decomposing a question and gathering several sources before answering. That matches the fan-out behaviour engines describe publicly, and we cover the mechanic in our query fan-out guide.

What follows practically is the same conclusion as everywhere else in this article, arrived at from a different direction. A research mode fetching ten sources on a topic will reach the pages that are linked in HTML, readable without rendering, and structured clearly enough to extract a specific claim. The rendering wall and the extraction wall are the two filters, and passing both is the entire technical half of AI visibility.

We should be precise about what we did not verify. We have not tested deep research modes ourselves, and neither public dataset isolates them as a separate agent. Treat this section as the reasonable reading of the available evidence rather than a measured finding.

11Running the audit yourself

The whole diagnostic costs an afternoon and no software budget.

Step one, the thirty second JavaScript check. Fetch a category page without rendering and count the internal links present in the raw HTML. If your navigation only exists after hydration, the count comes back at or near zero, and every non-Google crawler in the experiment above found nothing.

Step two, pull the last 30 days of access logs and filter by user agent. Separate them into three buckets rather than one: training crawlers, search retrieval bots, and user-session agents. The three answer different questions and mixing them produces a number that means nothing.

Step three, separate every Google agent. Googlebot and GoogleOther are not interchangeable, and only one of them feeds Search.

Step four, check status codes by agent. A crawler receiving 403s or 503s during its burst window is being silently excluded. This is the single most common fixable finding and it never appears in any AI visibility dashboard.

Step five, list the URLs ChatGPT-User fetched. That is your live citation signal, and it is free.

If you want tooling, options range from Screaming Frog’s Log File Analyser through hosted crawler-log products to open-source self-hosted readers that parse edge logs from Cloudflare or Nginx locally. Any of them beats not looking.

12What this evidence does not prove

Both datasets in this article are single-site studies by individual practitioners, published to communities rather than peer reviewed. That is worth stating clearly because the findings are strong enough to be tempting to over-read.

The JavaScript experimenter flags his own limits: one site, one niche, 41 days, a fresh domain with no external links pointing at the buried pages. Established sites may see Googlebot invest more in rendering. His counter, which we find reasonable, is that the non-Google zeros leave little room for interpretation, since eight crawlers found zero JavaScript pages under conditions where rendering was the only possible route.

The 48-day study is observational, so its findings describe what those crawlers did on that site during that window. Crawler behaviour changes quickly. GPTBot and ClaudeBot only started consuming sitemaps in March 2026, by that study’s own account, which is a reminder that any specific behaviour reported here may already have shifted.

What we would treat as durable: analytics cannot see AI bots, agent-level differences are real and large, and discovery through plain HTML links is the highest-leverage technical variable available. What we would re-test before relying on it: any specific percentage, peak hour, or per-bot behaviour above. Read your own logs. They are the only dataset that describes your site.

13Frequently asked questions