We Tracked AI Bot Fetches for 30 Days. The Pages With the Most Hits Weren’t the Ones Getting Read.

Only 29% of 13,319 robots.txt requests on our site verified as coming from the AI agents they claimed to be. For 30 days, the other 71% looked like ChatGPT, Claude or Perplexity in the user-agent field but did not pass our IP check.

I used to tell clients to watch AI bot traffic as a proxy for visibility. I was wrong. Even verified traffic is only the start of the story: our category pages took more than 1,100 hits each and produced no observed live-answer reads, while one dated changelog post received 576 verified fetches, including 355 user-triggered reads. The useful question is not whether a bot found your site. It is which page a bot read when someone asked a question.

Three counts that should never share one label

A hit is a log line with a familiar user-agent. It does not establish who made the request. For the robots.txt count, we checked source IPs against the vendors’ published information. OpenAI publishes crawler IP information, Anthropic publishes bot information, and Perplexity documents its crawlers and IP ranges. A claimed identity that fails that check does not belong in the verified total. I would not use the user-agent field alone to call it genuine traffic.

Then there is a second distinction. A verified request for robots.txt is not a read of a page that could answer a buyer’s question. A verified, user-triggered page read gets closer, but it still does not prove the page appeared in a response. These are different measurements:

The first three rows come from our edge-log analysis over those 30 days. The fourth requires checking answers. A fetch is an opportunity for a citation, not a citation. That sounds painfully literal until someone puts a climbing bot-hit chart in a deck and calls it AI visibility.

The robots.txt result shows why the distinction matters. We counted 13,319 claimed-agent requests for that file and verified 3,806: about 29%. Those figures describe one path on one domain, not the verified share of every AI-related request on the site. They also say nothing about which page helped a customer. A bot can check permission repeatedly without reading a single paragraph you wrote.

That leaves the more useful question: after the permission checks and the questionable identities are set aside, where did the full-page reads go?

The changelog drew reads; the category pages drew hits

In our page-level analysis, three kinds of URL behaved differently. The figures below come from the same 30-day edge-log window. I have kept the category-page hit count labeled as raw hits because the draft data does not give a verified-fetch denominator for those pages. Calling their result a verified-to-live rate would make the table tidier and less honest.

The homepage result is understandable. Someone asking about a brand or its pricing may prompt a read of the brand’s homepage. In our logs, 802 of its 1,140 verified fetches were ChatGPT-User reads in a live-answer context. That is a substantial share, though the logs cannot tell us what any resulting answer said.

The changelog was the more useful surprise. It was dated and narrow: what changed, when, and for whom. Of its 576 verified fetches, 355 were user-triggered page reads. A page that contains a complete answer in one place gives an answer engine something usable without asking it to follow a chain of links. The numbers are consistent with that explanation. They do not isolate the date, the page format or the wording as the cause, and they do not establish that the post earned 355 citations.

The category pages showed the opposite pattern. Each received more than 1,100 hits, yet we observed no live-answer reads for them in this window. These pages serve a navigation job: they introduce a topic and point to other URLs. A person can use that structure. A model looking for a passage it can use in response to a specific question may have little reason to stop there. We cannot tell from these logs whether every hit was a meaningful page read, much less whether the pages were considered and rejected. We can say their high hit counts did not translate into the user-triggered reads we measured.

I would not turn three rows from one site into a universal ranking of page types. I would, however, stop reporting those rows as if they described the same kind of attention. The homepage and changelog have verified-fetch denominators and observed live-context reads. The hubs have raw hits and none of those observed reads. That difference is the finding.

Why one answer on one URL is the better bet

I built hubs like these for years because they made sense for traditional SEO. Put a strong category page above a set of related articles, link cleanly, and help the visitor find the right detail. None of that makes a hub useless. It does mean its value as navigation can be mistaken for its value as an answer.

The changelog post did a different job. It addressed a bounded question on a single page. A reader did not need to open a category, scan ten links, choose a post and assemble the answer. The homepage had a similarly direct relationship to brand questions. The hub offered routes to information, but not necessarily a passage that settled the question where the bot landed.

That is my working explanation for the split, not a controlled test of page structure. A category page could contain a direct answer; a dated post could be vague. Our logs do not tell us what would happen if we rewrote the hubs and held everything else constant. They do tell us which URLs received user-triggered reads during this period, and they make an old assumption harder to defend: a page attracting bot hits is not necessarily a page supplying answers.

The practical editorial test is simple. Take a question you care about and open the URL you expect an AI assistant to use. Is the answer there, in words, without a second click? If the URL mainly says where to find the answer, its hit count is not much comfort. Give the question a page that can stand on its own, then measure what happens rather than declaring the format a winner in advance.

ChatGPT and Claude did different things in our logs

Agent-level totals can hide the same mismatch. In this window, Claude-User repeatedly requested robots.txt and verified against Anthropic’s published information, but few full-page reads followed. ChatGPT-User made fewer of those permission checks and accounted for nearly all the live-context page reads we could identify.

I do not know the internal reason. From the edge, Claude’s activity looked more like checking what it was allowed to access; ChatGPT’s looked more like fetching pages in response to user questions. That is a description of requests on our domain, not a claim about how either product always behaves. Someone else’s site, questions and traffic mix may produce a different split.

Add the two agents together, though, and you lose the distinction immediately. The combined number says AI systems showed interest. The path and agent columns tell you whether that interest stopped at robots.txt or reached a page that could answer something. Split requests by agent and path before you interpret the total.

A log report still cannot tell you who was cited

If a report offers only AI bot hits, ask for the next columns: verified hits and user-triggered page reads, broken down by agent and URL. On our site, that first check took 13,319 claimed robots.txt requests down to 3,806 verified ones. The page-level view then exposed a different distribution of reads than the raw hit counts suggested.

But logs stop at your server. They show what was requested, not what a person saw in an answer. To assess that side, use a repeatable prompt test: run the same set of buyer prompts across the assistants and record citations, mentions and share of voice. A citation is a link to your page. A mention names your brand. Share of voice compares how often you appear with competitors on the same questions. Keep the wording and schedule consistent, or changes in the test can masquerade as changes in visibility.

For a workable starting set, I would use 30 to 50 buyer questions and run them weekly across ChatGPT, Perplexity, Gemini and Claude. That is a measurement routine, not a benchmark our log data validates. The point is to look at the answers alongside the fetches. If a page gets read but never cited in the prompts you care about, you have a different problem from a page that never gets read at all.

Then map each miss back to a URL. Which page could answer that exact question without making the assistant gather fragments from a hub, another article and a pricing table? Which page actually does? The gap between those two answers is a more useful content queue than a list of URLs sorted by bot hits. Use logs to find reads; use prompt tests to find appearances. Neither measurement does the other’s job.

What I’d change on Monday

First, the limits. This is one B2B search-marketing site measured over 30 days. Most of the user-triggered reads we identified came from ChatGPT-User. I cannot tell you that your robots.txt requests will verify at 29%, that a changelog will outperform your category pages, or that our read counts predict anyone else’s citations. I also cannot turn a fetch into proof of an answer. The data runs out there.

What it does support is a change in what gets priority. Our high-hit hubs did not produce observed live-answer reads; a narrow, dated page did. So I would spend Monday on these three jobs, in order:

1. Verify before counting. Keep claimed-agent hits, verified hits and user-triggered page reads side by side, split by agent and path. Do not turn the 13,319 robots.txt requests into a visibility score when only 3,806 passed our verification check.
2. Put important answers on pages that can stand alone. Start with buyer questions whose expected URL currently points to a hub or makes the reader click again. Write the answer where the model can find it on one page. Our changelog’s 355 user-triggered reads out of 576 verified fetches make that worth testing; they do not guarantee the same result for the next page.
3. Check the answers people actually get. Run a fixed weekly prompt set and record who was cited, who was mentioned and which URL, if any, appeared. At groas, we put weekly AI visibility tracking across eight engines on the same scorecard as crawler visits. The answer check keeps the log chart from taking credit for a citation it never showed.

Crawling fills logs. A page read in a live-answer context is closer to visibility, and a citation in the answer is closer still. Our 30 days showed a clear difference between the pages collecting hits and the pages receiving those reads. Measure the stages separately, then give each important question a page that actually answers it.

If your site sees only a trickle of AI bot traffic, skip the log project for now and start with the prompt test. Logs earn their keep once you publish regularly and need to know which URLs agents re-read. Below that volume, a consistent 30-prompt check is more likely to show you where an answer is missing, and which page to write next.