LinkedIn
Our /robots.txt drew 13,294 requests claiming to be AI bots in 30 days. Only 3,730 passed IP verification. That gap is why I chose this artefact for a teardown: an edge server log export from dunedale.com, showing what reached our site before any dashboard tried to turn the traffic into an AI-visibility score.
The more useful finding sits further down the file. Three broad category hubs drew roughly 240 verified crawler fetches each and zero live-answer fetches. One dated post, google-ads-updates-2026, drew 330 verified live-answer fetches from ChatGPT and Claude. If I had judged those pages by raw bot traffic or by how much work went into their structure, I would have missed the distinction that matters.
Marketing teams trying to appear in ChatGPT and Perplexity answers often start with a third-party dashboard. Those tools run synthetic prompts, check whether a brand appears, and identify potential content gaps. That can show you what happened in those tests. It cannot show which bots actually requested your pages. For that, I start at the edge. This teardown follows the export in the order a reader encounters it: the bot claims, the homepage, the blog index, the category hubs, the specific post, and finally the pages with no live-answer traffic at all.
The artefact: 30 days at the edge
The export covers 30 consecutive days of HTTP requests to dunedale.com. It records requests declaring AI crawler or retrieval user-agents, then separates them into raw hits, verified hits, and live-answer hits. Those are three different measurements, not three names for AI citations.
Standard web analytics will miss much of this activity. As Lemniscate Growth’s crawler-log analysis explains, crawlers do not behave like visitors who execute client-side JavaScript and generate ordinary analytics sessions. A page can be fetched without appearing as a session in your usual marketing report. The access log tells you about the request; it does not tell you everything the requesting system subsequently did with the page.
Three columns, three different claims
- Raw hits count requests carrying a declared AI user-agent, such as
GPTBot. A user-agent is a header the requester supplies. It is easy to imitate. - Verified hits count requests whose connecting IP matches the vendor’s published ranges. OpenAI publishes IP prefix files for its bots; the export uses the relevant published ranges to check claimed identities.
- Live-answer hits are verified requests attributed in this export to on-demand agents such as
ChatGPT-UserandClaude-User. They show that an agent fetched a URL in a live-retrieval context. They do not, by themselves, show the final answer, a clickable citation, or a resulting sale.
That last boundary matters throughout this teardown. I can compare which pages the agents requested. I cannot read an access-log line and declare that the page appeared in an answer. Verdict on the columns: use them to narrow the question, not to skip straight to a citation claim.
Line 1: /robots.txt exposes the raw-count problem
The first entry worth stopping at is /robots.txt. Over the 30-day window, it received 13,294 requests declaring AI crawler user-agents. That is an impressive number if the dashboard stops counting there. It becomes less impressive when I check the connecting IPs against the vendors’ published ranges: 3,730 requests verified; 9,564 did not.
An unverified request is not proof of a malicious scanner. It is proof that the declared bot identity did not pass this check. Some may be scrapers or other automated traffic using a borrowed user-agent; the log count alone does not classify every requester. WebDecoy’s investigation of unverified crawler traffic describes why bot-name spoofing is a real issue, while Centinel Analytica’s ClaudeBot tracking reports requests claiming that identity which fail IP verification. Our own line makes the practical point without requiring a theory about every one of the 9,564 requests.
If an AI-monitoring report calls all 13,294 visits from authentic AI systems, its starting figure is wrong. And even the 3,730 verified requests are visits to a crawling-policy file, not evidence that a model read our content or cited groas. Verdict on line 1: authenticate the requester before you interpret the volume. A large raw count on /robots.txt is not an AI-presence strategy.
Line 2: the homepage gets fetched; the blog index barely does
The next contrast is between / and /blog. In this export, the homepage recorded 834 ChatGPT-User live-answer fetches; the main blog index recorded 4. Both also received verified background crawling. The difference appeared when the on-demand agent selected a URL to fetch.
The homepage contains the core offer, pricing structure, customer case study metrics, and service architecture. The blog index is a directory of cards and excerpts. If the retrieval task concerns groas itself, the homepage gives the agent more material to work with in one request. The index asks it to choose another link before it reaches a complete article. That is a plausible explanation for the pattern, not a record of the prompts that caused it.
OpenAI’s crawler documentation distinguishes its bot roles, which is one reason I would not roll background indexing and ChatGPT-User requests into a single ‘AI visits’ chart. A background crawl can help a system discover pages. An on-demand fetch is a different event. Neither tells me, on its own, whether the final answer linked to the page.
Verdict on line 2: the homepage is doing more work than the blog directory in this window. I would keep the homepage’s direct, substantive answers. I would not pour effort into the index merely because it is the front door to the articles for a human reader.
Line 3: one dated post beats three category hubs on live retrieval
This is the line that makes the export worth keeping. The three category hubs cover broad topics including search engine marketing, agency workflows, and Google Ads bidding. Each drew about 240 verified crawler fetches during the window. Across all three, the export records zero live-answer fetches from ChatGPT and Claude. Meanwhile, google-ads-updates-2026, a specific post about recent Google Ads changes, recorded 330 verified live-answer fetches.
I would not translate that into ‘330 citations.’ The log cannot support it. But I would not dismiss the contrast, either. The hubs were visited by verified crawlers, so a simple ‘the bots never found them’ explanation does not fit these lines. The dated post was fetched repeatedly in the live-answer category; the hubs were not.
What the specific post offers that the hubs do not
The category pages organise links and summarise subjects. That is useful site architecture. It is less useful as a single document answering a narrow question. The updates post contains dates, named changes, bidding details, and cost-per-acquisition discussion. A request for recent Google Ads changes has a more direct match there than on a hub that points readers toward several other pages.
DataDome’s description of Claude-User is a useful reminder that a user-initiated request can target a particular URL. It does not promise that an agent will follow every link on a category page to assemble a better answer. Our export cannot expose the agent’s selection process, but it shows the result at our edge: the specific document received live fetches, and the broad hubs did not.
This is not an argument for deleting category pages. They still organise the site for people and crawlers. It is an argument against treating a well-structured hub as a substitute for a document that answers something. Ten Point Labs’ review of Ahrefs search data makes a related point about AI Overview citations not being confined to the top ten organic results. I would not use that separate dataset to explain our individual requests. The log gives me a narrower, sturdier observation: verified discovery did not translate into live retrieval for these hubs.
Verdict on line 3: keep the hubs as navigation; put the answer on the page you want fetched. In this window, the dated, specific post gave live agents a reason to request it that the broad category pages did not.
Line 4: a missing fetch is not a diagnosis
Further down the export, dozens of thought-leadership articles show zero ChatGPT-User or Claude-User live-answer hits. This is where a dashboard can make a routine absence sound like an emergency. Zero does not tell me the technical setup is broken. It does not tell me the page was penalised, either.
Perhaps no relevant live-retrieval request reached that article during these 30 days. Perhaps an answer did not require a web fetch. Perhaps another URL was selected. The access log records requests that arrived; it does not record all the questions users asked or every reason a system chose not to request a page. That limit is especially important when the sample is one site over one month.
Commercial Generative Engine Optimization tools answer a different question. As Scalenut’s evaluation of AI visibility platforms describes, platforms such as Profound, Peec AI, Writesonic, and Otterly use synthetic prompts to check brand appearances. A missing appearance can be useful to investigate. It cannot, by itself, tell you whether a bot met a firewall challenge, whether the relevant page was fetched, or whether that particular answer used the live web at all.
Verdict on line 4: investigate a zero; do not invent its cause. A prompt test and a server log can inform the same decision, but neither is a transcript of the other.
What I would keep from this teardown
I would keep the distinction the export forces on us: claimed bot traffic, authenticated crawling, and live-answer fetches belong in separate columns. Then I would use those columns in order, rather than buying a content-gap list and working backward to an explanation.
A practical log review starts with the fields this export needed:
- Capture the incoming IP, user-agent, HTTP status code, and requested path in your edge access logs.
- Check claimed OpenAI and Anthropic identities against their published IP ranges. Set unverified requests aside before comparing pages.
- Separate background crawler requests from
ChatGPT-UserandClaude-Userrequests. Compare URLs within each group rather than combining them into a sitewide bot total. - Read the requested pages. If a hub receives background crawls but no live fetches while a specific document receives many, examine what each page actually answers. If relevant bots receive an error response, examine that before rewriting the copy.
For the content itself, I would keep the updates post’s specificity: a clear subject, a date, and material a reader can use without touring the rest of the site. I would keep direct commercial answers on pages such as the homepage, too. Someone looking for a price, a service boundary, or a technical detail should not have to extract it from three paragraphs of corporate philosophy. That is good writing for humans before it is any kind of retrieval tactic.
I would not keep the draft conclusion that a live-answer fetch proves a citation, or that zero such fetches proves a passage problem. If a verified agent fetches a page, I know it reached that page. If it does not, I need more evidence before choosing between content, technical access, and simple lack of demand. The log makes the next question sharper. It does not answer every question for me.
That distinction is also why I prefer execution records to an impressive-looking activity total. At groas, the operating model pairs autonomous work across paid and organic search with a named human strategist who sets direction and owns accountability. Campaign and content actions are logged with their reasoning, with attention on attributable pipeline, closed-won revenue, and cost per acquisition rather than a chart of impressive but unqualified activity. The service runs on a flat monthly fee, with no setup fees and month-to-month cancellation; the groas MCP connection for ChatGPT and Claude lets clients query campaign, citation, and crawler activity directly.
Overall verdict: this edge export is far more useful than a raw ‘AI bot visits’ total, and less conclusive than a citation report. Out of 13,294 claimed AI requests to /robots.txt, only 3,730 verified. Further down, three crawled category hubs drew no live-answer fetches while one specific, dated post drew 330. I would keep that post’s answer-first approach and keep checking the logs. I would stop calling a request a citation.