Your page ranks third on Google. ChatGPT answers the same question and cites two competitors instead. Before you rewrite the page, check whether the answer ever made it past your robots.txt file.

On our own site, Claude-User fetched robots.txt 1,218 times for live answers in 30 days. That is the fact I would put ahead of another recycled Google audit. An assistant looking for an answer right now has to get permission, receive a usable page and find a passage it can quote. A good ranking does not carry the page through those steps.

I’ll follow one illustrative question: “What should garage door replacement cost in Dallas?” You have a page that answers it, and it ranks. Say you are spending $20k/month on search. The assistant still cites two competitors and skips you. Here is how I would find where the trip broke.

Before any bot requests your page, the assistant has to decide whether to search the web. Only about 46% of ChatGPT queries trigger live web search, according to an 80M-query Semrush analysis. For queries that do trigger it, the described path is a sequence: rewrite the person’s question as a search query, pull metadata through Bing, evaluate snippets, fetch a few full pages, then write an answer.

That split sets the limit of a technical fix. You cannot use a change to today’s HTML to put a page into an answer that never searches the live web. You can work on the path a live fetch takes. For the Dallas question, that means asking whether your page appeared in retrieval, whether its snippet earned a full fetch and whether the fetched page supplied a usable answer.

I used to talk about ranking as though it meant a page was “in the model.” I was wrong. A Google ranking and a live AI citation are different outcomes. Ranking tells you Google found reason to show the page. A citation requires the assistant to search, reach the page and choose something on it. First establish which path you are trying to improve.

Step 2: Check permission for the bot doing the fetching

If the assistant goes live, the fetcher checks /robots.txt before requesting the Dallas pricing page. This is where the 1,218 Claude-User requests matter: the rules get read before the answer does.

The bot names matter, too. OpenAI separates three jobs: GPTBot is for training, OAI-SearchBot is for ChatGPT search, and ChatGPT-User fetches pages in response to a user’s request. Anthropic separates them as well: ClaudeBot for training, Claude-SearchBot for search indexing and Claude-User for user-directed fetches. Each has its own robots.txt control.

That is why “we allow AI bots” tells me very little. A rule aimed at training crawlers is not the same as a rule for live answers. Block OAI-SearchBot and you shut out ChatGPT search discovery; block Claude-User and you block its user-directed retrieval. On our own site, we counted 1,218 robots.txt fetches by Claude-User for live answers in 30 days. Those checks were not a request for a report next month. They were part of getting an answer to someone now.

The self-inflicted version is familiar: a blanket Disallow copied from a 2023 post about blocking ChatGPT, or a rule that names the training bot while nobody checks the search and user-directed ones. For the Dallas page, read the rules as each relevant agent would. Audit permission by bot name, not by the label “AI.”

Step 3: See what the edge actually sends back

Permission in robots.txt does not mean the fetcher receives your page. It still has to make an HTTP request, and the CDN or web application firewall can stop it before your server serves the Dallas answer. Your browser may pass a Cloudflare check without much fuss. A fetcher presented with a human challenge cannot do the same. As this account of Cloudflare bot blocking puts it, robots.txt is the note on the door; the firewall is the guard. That analogy ends there: unlike a person at a door, a fetcher needs an actual usable response, not a polite explanation of why it was stopped.

With Bot Fight Mode on, the response may be a challenge page rather than the pricing page. A 403 with cf-mitigated: challenge in the header tells you the request hit a test meant for humans. Cloudflare’s Search, Agent and Training controls also separate search indexing, user-directed fetching and training. Block Agent while Search remains open, and a live answer can still lose access to the page.

Other edge rules can have the same practical effect. Slow time to first byte, a geo rule that blocks the fetcher’s region or a cookie requirement can leave the assistant waiting or looking at the wrong response. In the Dallas example, the page’s Google ranking does not rescue a request that got a challenge instead of a price. The assistant has other pages to use.

Inspect the response the fetcher gets, not the one your office browser gets. If the logs show a challenge or 403 for these agents, rewriting the pricing paragraph is work on the wrong side of the door.

A bouncer blocks a robot messenger at a server door while a human visitor passes through

Step 4: Check whether the answer exists without JavaScript

Suppose the Dallas page clears the edge. It can still arrive almost empty. Your browser runs JavaScript and shows the price range, service-area list and FAQ. The AI fetcher may only receive the initial HTML. A Vercel and MERJ analysis from December 2024 found that the major AI crawlers it examined, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot, did not render JavaScript. They downloaded JS files but did not execute them.

That difference is easy to miss in a normal audit. In a 50-site test of AI-builder homepages, 20 of 32 judgable sites sent crawlers 1 to 11 words, with a median of 7. Human visitors saw a median of 631 words. Titles and meta descriptions were still there. From a distance, the pages looked present; at fetch time, their answers were missing.

For our running example, it does not matter how clear the Dallas price looks in a browser if a client-rendered component is the only place it appears. Search the raw page source for the answer text. If the price is absent there, treat the missing HTML as the problem before editing the prose a human can already see.

Step 5: Give the fetcher a passage that stands on its own

Now the fetcher has usable HTML. That does not mean it will quote your page. The answer has to be easy to identify and useful apart from the surrounding copy.

There is an earlier gate, too: title and meta information help decide which snippets earn a full fetch. A vague title can keep the assistant from reaching an otherwise helpful page. Once it does fetch the Dallas page, a heading such as “Garage Services” gives it less to work with than a heading that names the cost question. An answer buried three screens down, beginning “It depends,” asks the fetcher to assemble the point for you. A competitor’s first sentence might simply say: “A standard garage door replacement in Dallas costs $1,400 to $2,800 installed.” Here that range is an illustrative answer, not a price quote for a real job.

The passage has to survive separation from the page. The Princeton and Georgia Tech GEO study across 10,000 queries found that targeted rewrites lifted AI visibility by up to 40%, with quotations adding 37% and statistics adding 22%. That does not mean sprinkling numbers over vague copy. For the Dallas page, I would make three deliberate edits:

  1. Put the question in an H2 so the page names the answer it offers.
  2. Answer directly beneath it in roughly 40 to 60 words, using the relevant number and date.
  3. Name the place and subject plainly, rather than relying on “we” or “it” to make sense outside the page.

Write the answer so it still works when lifted out of context. If a reader needs the three paragraphs above it to understand the price, the fetcher may choose a clearer passage elsewhere.

Step 6: Make the final citation defensible

At this point, imagine the assistant has three usable passages. It still has to choose which one to cite. Specificity gives it something it can repeat: “$1,400 to $2,800 installed, updated March 2026, includes haul-away in Dallas County” says more than “competitive pricing.” Corroboration helps when other sources give a similar range. A visible date helps a commercial answer look current.

The freshness difference is not small in the citation figures collected here: 83% of citations go to pages updated in the last 12 months, and over 60% to pages refreshed in the last 6 months. A Dallas price last updated in 2021 may be correct. Beside an answer dated 2026, it looks harder to defend.

I treat the answer block a little like ad copy: put the useful claim where someone can see it, say what it covers and keep it current. The analogy breaks at the job it performs. An ad asks for a click; this passage must answer a question even if the reader never clicks. Give the assistant a quotable number, place and date, then maintain them.

Paper strips cut from a pricing page, with one answer strip lifted for citation

Test the whole path, not an AI visibility score

Those six steps explain why a score alone is not much of a diagnosis. A tool can report that ChatGPT cited you twice last week and a competitor four times. It can then hand you a familiar list: add schema, build links, improve Core Web Vitals. That still does not tell you whether the Dallas page failed at permission, the firewall, raw HTML or the passage itself.

I would ask a tool to check the live path in order:

  • Permission by agent: Do OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User each clear robots.txt and the CDN rules relevant to their job?
  • Delivery as the fetcher sees it: What status code, time to first byte and challenge headers does the request get from the fetcher’s location?
  • Raw HTML versus rendered page: How many words survive with JavaScript off? Is the price, hours or answer text among them?
  • Passage gaps: Which buyer questions lack a standalone answer, and which pages bury that answer well below a vague heading?

That is a useful standard for the questions owners ask me: which tool exposes the technical blocks, and which platform finds the content gaps behind missed citations? I want the diagnosis tied to action and to someone accountable for the outcome, not filed away as a prettier audit. That is the point of groas as a fully autonomous growth engine with a named human strategist. If the request itself is failing, I would start with this 45-minute curl and CDN audit. Fix permission and delivery before rewriting the page. A citation report cannot substitute for a successful fetch.

Try the model on a new question

Take a different query: “Emergency plumber open now near South Congress Austin.” I have not used it to build any of the six steps. Run it through them and the likely failure points become visible.

First, the assistant needs current hours and a location. A live search is the useful path here; an old general description of a plumbing business cannot establish whether it is open now. If search fires, the snippet needs clear local signals to earn a closer look.

Next comes permission, then delivery. A plumber with an old ChatGPT block may never enter the live retrieval path. Another may allow the agent in robots.txt but send it a Bot Fight Mode challenge. Its five-star reviews do not help the assistant read a page it never receives.

Suppose two pages do load. One uses a JS-heavy site builder: the human visitor sees prices, hours and a phone number, while the raw HTML offers seven words of navigation. The other serves those details in raw HTML. It puts the question in an H2, followed by a 50-word passage stating that the business is open until midnight, serves the area, charges a flat diagnostic fee and updated the information in February 2026.

I would not promise the second page a citation without seeing the answer. But the model tells me where to look and what to fix: the first page cannot supply its visible answer to this fetcher; the second gives the assistant a passage it can use. Permission, then delivery, then an answer that stands alone. If a vendor cannot walk that path with you, its score is decoration.