A marketer rewrites a landing page on Tuesday, spots it in a Google AI Overview on Friday, and calls the rewrite a win. I watched PPC teams make the same mistake with ad copy for years: change a headline during an algorithm shift or seasonal spike, then take credit when conversions rise. Paid search learned to keep a control. AI Overview optimization needs one too.

This is the test I would run: match query-page pairs, edit only half, and leave the other half untouched. Set the observation window and the result that would change your mind before publishing. Otherwise, the last edit gets credit for whatever Google happened to do next.

Question: Did the edit cause the citation?

Why one page before and after is not a test

Google’s AI Overview citations move without your help. Authoritas analyzed 11,203 keywords and found that 70% of citation URLs changed within two to three months. Its eight-week volatility score was 0.68 for AI Overviews, compared with 0.49 for classic organic search. In Ahrefs tracking across 43,000 queries, citation sets shifted roughly 70% of the time; consecutive runs shared only 54.5% of URLs.

So a new citation two weeks after your rewrite leaves the important question unanswered. Did the copy help, or did Google rotate its sources?

The background noise is not confined to AI Overviews. Split-testing benchmarks compiled by Kirro describe untreated organic control groups swinging by plus or minus 10% during an average test window, while roughly 75% of isolated on-page interventions show no statistically detectable impact. A single URL cannot tell you which force moved it. The holdout gives you a measure of what happened without your edit.

Setup: Build two comparable groups

Select queries that actually render AI Overviews

Start with 10 to 20 distinct commercial or mid-funnel searches your buyers use. Check them in an incognito desktop session and keep queries where an AI Overview appears consistently, with citation cards you can inspect. If the surface never appears, you cannot test whether an edit helped your page earn a citation there.

Do not limit the list to terms where you already rank near the top. Ahrefs tracking reported by DesignRush put the overlap between organic top-10 results and AI Overview citations at 38%, down from 76%; BrightEdge reported top-10 citation overlap as low as 17%. Google can assemble an answer from sub-queries rather than simply citing the first few blue links. Choose searches that matter to buyers, then record the page on your site intended to answer each one.

Match pages, then assign treatment and holdout

For each query, identify its exact target URL: perhaps a product page, category overview, or comparison page. Aim for 10 to 20 query-page pairs in total. Match pairs as closely as you can on page type and baseline organic impressions from Google Search Console. Within each matched set, randomly assign one to Treatment and the other to Holdout.

Do not put your favorite high-traffic pages in Treatment and leave the scraps in Holdout. As SearchPilot explains in its split-testing methodology, assigning comparable URLs across groups helps separate an editorial change from sitewide and market shifts. Check the resulting groups before you proceed. If one arm contains the busy product pages and the other contains quiet comparison pages, rematch them.

Matched landing pages divided into an edited treatment group and an untouched holdout group.

Put a firewall around the holdout

Write the query, target URL, page type, baseline impressions, and assigned group in a spreadsheet. Then make the control rule plain: nobody edits the holdout URLs during the test. Lock editing permissions on those paths if your CMS allows it.

Avoid sitewide template rewrites, canonical changes, and internal-linking projects during the window. They can affect both groups and make the result harder to read. In a PPC split test, I would not quietly change the control campaign halfway through and call the numbers clean. The same rule applies here. Holdout pages show what happened while you left them alone.

Baseline week: Record what Google does before you edit

Check every query on a fixed schedule

Observe the full roster for seven consecutive days before changing Treatment. Run each query twice daily, at 09:00 and 17:00 local time, using the same desktop browser profile and location setup. Keep that environment for the post-edit window too. One Monday screenshot followed by another a week later tells you remarkably little.

An SE Ranking study of 10,000 queries cited by Search Engine Land illustrates why repeated checks matter. When the same keywords were tested three times on the same day in Google’s AI interface, only 9.2% of citation URLs overlapped across all three runs; 21.2% showed no overlap. You are sampling a moving surface, not reading a permanent ranking.

Use the browser-rendered result, not a raw Gemini API response, as your observation. ZipTie’s analysis of AI search tracking tools reports that raw LLM API completions can differ from browser-rendered AI Overviews by up to 40%. The test concerns what a searcher sees in Google Search.

Log cited, mentioned, and absent separately

For every rendered AI Overview, record one state for your target page:

  • Cited (2): The exact target URL appears as a clickable citation, card, or source link in or beneath the overview.
  • Mentioned (1): The answer names your brand, domain, or product, but provides no clickable link to your domain.
  • Absent (0): The answer neither mentions your brand nor links to the target page.

Keep mentioned separate from cited. A brand name in generated prose is not a referral link. If no AI Overview renders on a check, mark it non-rendering rather than scoring it absent. Record that event, but leave it out of the citation-rate denominator.

At the end of the week, calculate each group’s baseline citation rate: cited results divided by rendered AI Overviews. With 10 queries in Treatment checked twice a day for seven days, you would attempt 140 checks. If all 140 render an overview and 14 cite your target pages, Treatment starts at 10%. If Holdout records 17 citations in 140 rendered checks, it starts at about 12%. Those are illustrative numbers, not a result from a run. Your own baseline is the one you write down.

Set the decision rule now

Before touching the copy, choose the post-edit window and the minimum net lift you would treat as worth investigating. For this protocol, use six weeks and a five-percentage-point net-lift threshold. A smaller difference is inconclusive; a larger one is a signal to repeat or extend the test, not a claim of statistical proof. Also decide that an apparent lift will not count as a clean result if the groups receive other meaningful changes or their AI Overviews stop rendering often enough to undermine the comparison.

Writing the rule now prevents the familiar maneuver of seeing a four-point lift and deciding, after the fact, that four points was the goal all along.

Intervention: Change one thing on Treatment pages

Pick one structural change type and apply it consistently across Treatment. Leave Holdout alone and keep the other elements of the treatment pages stable. Your options include:

  • Answer-first block: Place a 45-to-60-word direct answer immediately below the H1, without an introductory windup.
  • Clean comparison table: Present service tiers, pricing variables, or product specifications in a semantic HTML table rather than nested JavaScript tabs.
  • Buyer sub-query FAQ: Add three concise questions drawn from secondary searches, each answered in two factual, declarative sentences.

Do not add all three and then guess which helped. An ad test that changes the headline, description, audience, and landing page at once may move the number, but it teaches you little about the cause.

Structured data offers a useful warning against tidy before-and-after stories. A reported Ahrefs analysis of 1,885 pages that added JSON-LD schema and 4,000 matched controls found a -4.6% relative change in Google AI Overview citations. Both groups declined over the observation window; treated pages declined slightly faster. Without controls, it would have been easy to blame the entire fall on the schema change. Isolate the edit, then compare its movement with the holdout’s.

Diagram of the treatment-versus-holdout calculation for AI Overview citations.

Keep the schedule fixed for six weeks

A new Google Ads asset can enter an auction soon after review. A landing-page edit has to be crawled, rendered, indexed, and reflected in the search system before it can plausibly affect a citation. For a non-news page, that may take weeks. A Friday citation is not evidence that Tuesday’s copy did the work.

For the six weeks you chose before editing, run the same queries at 09:00 and 17:00 with the same browser and location setup. Keep a log with these columns:

  1. Date and time
  2. Target query and URL
  3. Cohort: Treatment or Holdout
  4. Visibility state: cited, mentioned, absent, or non-rendering
  5. Notes on the results-page layout and any relevant page changes

Calculate citation rates from checks where an AI Overview rendered, and keep non-rendering frequency visible alongside them. Otherwise, a change in how often the overview appears can masquerade as a change in how often your page gets cited. Do not change the schedule or end the window early because the first week looks promising.

Result: Subtract holdout drift from treatment lift

Work through the two deltas

At the end of the window, calculate the change in citation rate for each group. Then subtract the Holdout change from the Treatment change:

Net lift = (Treatment post-edit rate − Treatment baseline rate) − (Holdout post-edit rate − Holdout baseline rate).

Suppose Treatment moves from 10% at baseline to 24% afterward: a gain of 14 percentage points. If Holdout remains at 12%, the net difference is 14 points. If Holdout instead moves from 12% to 22%, its 10-point gain leaves a net difference of four points. Same treatment result, very different interpretation. These figures are hypothetical; the calculation is the point.

Under the rule set before editing, that four-point net difference is inconclusive. A result above five points is worth a repeat or a longer observation, not a license to declare the layout proven. With only 10 to 20 query-page pairs, individual pages and rotating citation sets can still move the rate sharply. Research described by Search Engine Land found that 65% of cited sources in AI-interface tracking shifted daily across 45 days.

Read the direction of both groups as well as the final subtraction. If both fall or both rise substantially, something broader than your edit may be moving the result. The holdout does not tell you precisely what changed in Google or on competing pages. It stops you from assigning all of that movement to your copy.

Vendor test: Put recommendations in Treatment, not in a slide deck

If an optimization vendor recommends a landing-page change, ask it to name the change type and the result it expects. Then apply that one recommendation type to Treatment while keeping Holdout locked. Use the same baseline, six-week window, and decision rule. A visibility score or a list of prompts is not a substitute for showing that the recommended work outperformed pages you did not touch.

This is where the distinction between advice and execution matters. Advisory tools can hand your team a backlog of definitions to write, schema to add, and comparison pages to build. Legacy agencies can turn the same backlog into a monthly review and a retainer. Neither arrangement earns credit for a citation merely because a citation appeared after the work.

That is the operating model groas is built to replace: continuous execution by specialized models, with a dedicated human strategist setting direction and guardrails and owning the outcome. Actions are logged with plain-language reasoning rather than disappearing into an unexplained line chart. The test still matters. Always-on execution should make controlled measurement easier to act on, not make it optional.

The next time an agency pitches an AI search retainer, or a tool promises inclusion in Google’s answers, give its recommendation a holdout. Match the pages. Freeze half. Set the window and threshold before publishing. If Treatment beats Holdout clearly, repeat the test before rolling the edit across the site. If the difference stays small, do not buy a wider rollout on the strength of a Friday screenshot. And if the vendor cannot tell you what would count as failure, you have learned something before paying for another dashboard.