Your best page will not earn a citation if a live-answer bot cannot fetch it. I would check robots.txt, the page’s initial HTML, and the sitemap before rewriting a single headline.

In 60 minutes, you can find and fix the obvious access problems, then leave with a way to verify actual bot fetches. Have one key page URL, your sitemap, and server or CDN logs ready. Our own edge logs showed Claude-User fetching robots.txt over 1,200 times in 30 days for live answers. Start at the gatekeeper file.

Before you start: open the three things you will test

Set aside 60 minutes and open:

  1. yourdomain.com/robots.txt in a browser.
  2. Your key money page and yourdomain.com/sitemap.xml.
  3. Server or CDN logs that you can filter by user-agent. If you run Cloudflare, look under Analytics > Logs. On WordPress shared hosting, look for cPanel > Raw Access Logs.

You do not need a paid crawler, a new plugin, or a content rewrite. Pick two pages you actually want cited and use those same pages throughout the checks. Otherwise, it is too easy to prove the homepage works while the service page remains broken.

1. Check which bots your robots.txt lets through

Open robots.txt and read each User-agent group. Separate bots used for training from bots that help fetch or surface current answers. OpenAI assigns those jobs to different bots: GPTBot is for training, while OAI-SearchBot is for ChatGPT search. I used to tell clients to block everything with AI in the name. I was wrong. That is a neat way to hide a good page while you keep rewriting its headline.

If you intend to block training while allowing search, use explicit groups rather than one broad AI-bot rule. For example:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Choose training and live-answer access separately

PerplexityBot surfaces sites in Perplexity search and respects robots.txt. Anthropic distinguishes ClaudeBot for training, Claude-SearchBot for search, and Claude-User for user-requested fetches. If you want Claude to retrieve a page for a user, do not block Claude-User by accident. A rule for one bot is not a policy for every bot in that company’s lineup.

Expected result: your published file expresses the access you intended, with the relevant search and live-fetch bots allowed and only the training bots you chose to exclude blocked. Fetch the file as a bot to confirm that you are seeing the published version:

curl -A "OAI-SearchBot" https://yourdomain.com/robots.txt

Common mistake: treating the response as proof that access works. This command shows you the file the request received; you still have to read the applicable group and check whether your firewall permits the bot through. A User-agent: * rule, a bot-specific rule, and a security rule can all matter. Do not assume their order alone tells you which rule wins. If the file still shows the old version, note when you saved it: SearchBot changes can take about 24 hours to take effect.

The file says who may fetch. The firewall decides who can.

2. Look for the answer in your page’s initial HTML

Open one of your chosen pages with Chrome’s View Source command, or enter the source view in the address bar:

view-source:https://yourdomain.com/your-service-page

View Source does not switch off JavaScript in the browser. It shows the HTML sent before the page’s scripts build the visible result, which is the useful check here. Many standalone AI crawlers do not render JavaScript. If the headline, service facts, and opening copy exist only after a script runs, some fetchers receive a shell instead of an answer.

I learned this on an ecommerce account where product copy lived in a React bundle. Humans saw reviews and specs. The initial HTML offered <div id="root"></div>. No rewrite of that copy could help until we changed what the server sent.

A visible web page beside the empty initial HTML an AI bot may receive

Find the text before changing the template

Use Ctrl+F in View Source for a sentence from the visible page. Then look for its H1, opening paragraph, price or service list, and at least one internal link. The answer needs to exist in initial HTML, not just in window.__DATA__, a later fetch() response, or a JavaScript mount point.

Check both of your chosen pages from the command line as well:

curl -s https://yourdomain.com/page/ | grep -i "<h1"

Expected result: the command returns an H1 from the HTML response, and View Source contains the facts a bot would need to quote.

Common mistake: finding the copy in the rendered browser page and calling the check done. If View Source lacks it, ask your developer to render the H1, first 300 words, price, address, and FAQ answers in the initial HTML. Server-side rendering or static generation can do that without discarding the interactive page. If a full template rebuild is out of scope this week, put a static HTML summary above the widget. Give the fetcher the answer before the app loads.

3. Make the sitemap a shortlist of answer pages

Open yourdomain.com/sitemap.xml and inspect its <loc> entries. Keep the canonical, indexable pages that return 200 and that you would want someone to encounter as an answer: service, location, pricing, and useful detailed posts. Remove tag and category archives, ?sort= variants, redirects, 404 pages, and pages marked noindex. Listing non-indexable or redirected URLs sends conflicting sitemap signals.

Most small sites have a familiar mismatch: 40 URLs listed, perhaps 12 doing the real work. The exact count is not the point. A sitemap is a shortlist, not an inventory.

A cluttered sitemap narrowed to the pages worth citing

Expected result: the sitemap contains your two chosen answer pages and excludes the dead or unwanted entries. Submit the cleaned file through Search Console > Sitemaps. Check that the sitemap itself responds, then check the listed URLs separately:

curl -I https://yourdomain.com/sitemap.xml
curl -I https://yourdomain.com/your-service-page

The first command cannot tell you whether every <loc> works. Test those URLs, especially the pages you want cited.

Common mistake: leaving the selection entirely to Yoast or Shopify defaults, then checking only for a successful sitemap submission. A submitted sitemap can still point at redirects, thin archives, or pages you meant to hide. At this step, your job is mostly to delete.

4. Put consistent business facts in the page and schema

Add an Organization JSON-LD block to the homepage through your SEO plugin’s Custom Schema field or your theme’s header.php. Use your real details in place of the example. The schema guidance for AI search is useful here, but do not treat markup as a substitute for visible facts.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "@id": "https://yourdomain.com/#org",
  "name": "Acme Plumbing",
  "url": "https://yourdomain.com/",
  "logo": { "@type": "ImageObject", "url": "https://yourdomain.com/logo.png" },
  "description": "Licensed plumber in Austin serving homes since 2011.",
  "address": { "@type": "PostalAddress", "addressLocality": "Austin", "addressRegion": "TX" },
  "contactPoint": { "@type": "ContactPoint", "telephone": "+1-512-555-0100", "contactType": "customer service" },
  "sameAs": ["https://www.linkedin.com/company/acme", "https://www.yelp.com/biz/acme"]
}
</script>

Expected result: Google’s Rich Results Test detects the markup without errors. More importantly, the name, phone, address, and description match what a visitor can read on your homepage, footer, and contact page. Use one business name consistently rather than alternating among Acme, Acme Inc., and acme.com.

Common mistake: polishing the schema while leaving the useful answer inside an image or booking widget. Put service and pricing facts in plain HTML too. An example of the kind of sentence a reader can use immediately:

Emergency visit $149 weekdays, $199 after hours. Service area: Austin, Round Rock, Cedar Park.

That is example copy, not a price list to paste into a live business page. Replace it with your actual facts. The page must say what the markup says. A bot cannot quote a price that exists only after someone clicks through a widget.

5. Check the response before you worry about speed

Run a header check on each answer page:

curl -I https://yourdomain.com/your-service-page

Look for a successful 2xx response and content-type: text/html. Google’s indexing guidance explains how HTTP errors affect crawling; the practical issue for any fetcher is simpler. A blocked or broken request does not deliver the page you wanted it to read.

Expected result: each chosen URL returns its HTML without an error or a chain of redirects. If you are checking speed too, time an actual request rather than guessing from a header response. The draft target here is a cold fetch under 1.5 seconds; treat that as a working target, not a promise of citation.

Common mistake: celebrating a 200 that serves “page not found” text, or checking the browser while a security plugin gives bot requests 403. A pricing page that redirects three times and a location page that times out after eight seconds also deserve attention. Fix status first, speed second. Correct dead URLs to a real 404 or an appropriate 301, check firewall rules for verified bots, then cut render-blocking work on the two templates. Do not spend the hour tuning scripts on a page the fetcher cannot access.

6. Use logs to confirm real bots reached the right pages

Filter the last seven days of server or CDN logs by these user-agents:

GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Claude-SearchBot
Claude-User
PerplexityBot
Perplexity-User

Look for successful requests to /robots.txt and to your two answer pages. Do not require them to appear seconds apart: logs show what happened, not a tidy demonstration sequence. ChatGPT-User is user-triggered, and robots.txt may not apply to it. Perplexity-User also fetches on user request and generally ignores robots.txt. For your search-access check, focus on OAI-SearchBot, Claude-SearchBot, and PerplexityBot reaching the intended URLs with 200 responses.

Verify the bot before counting the hit

Anyone can send User-agent: GPTBot. Anthropic publishes information for verifying its crawlers, and Perplexity publishes IP ranges for its bots. Compare the logged IP with the relevant published information before treating a hit as evidence or changing a firewall rule.

Expected result: verified bot requests reach the answer pages and receive 200, with readable HTML behind that response. You may not see a visit from every bot during this check; absence in a short window is not proof that the page is blocked.

Common mistake: counting hundreds of requests from an unverified IP as AI visibility, or trusting an Allow: / line while a firewall drops the actual crawler. Raw hits lie. Verified fetches tell the truth. If verified requests are blocked, correct that rule and check again after changes have had time to take effect.

Verify the fix, then move to the next citation gap

Wait 48 hours and re-filter the logs for OAI-SearchBot, Claude-SearchBot, and PerplexityBot requests to your two pages. Check their status codes and the HTML those URLs return. Then ask each engine the question the page answers and look for a citation. A fetch is evidence that the plumbing works; it is not a guarantee that your page deserves the answer.

If verified fetches arrive but citations do not, stop babysitting robots.txt. Improve the depth and proof on the page, then check what other sites link to. That is the first thing to change once access works. The ongoing watch over fetches, fixes, and citations is work I would hand to groas rather than pay someone to rediscover the same blocked file every month.