Every few months a new “AI SEO” shortcut does the rounds. Self-promotional listicles, llms.txt files, Markdown copies of every page, bulk “mentions” on forums. And every few months, store owners ask us whether technical SEO for AI search is enough, or whether they need a separate GEO or AEO strategy to appear in ChatGPT, Perplexity or Google’s AI Overviews.
Google has now answered that question again. In a recent interview with David Iwanow, reported by Search Engine Journal on 2 November 2026, Google’s John Mueller said AI systems build on SEO, and that technical SEO in particular will “remain very relevant”.
Our audit work on Magento 2, Adobe Commerce and WooCommerce stores points the same way. The stores AI assistants cite are rarely the ones chasing hacks. They are the ones crawlers can reach, read and trust. In this guide we break down what Google said, how AI search actually finds your pages, and the technical checklist we use with our own clients.
The Short Version
Google says there are no extra technical requirements for AI Overviews or AI Mode. A page must be indexed and eligible to show with a snippet.
Most AI crawlers outside Google do not run JavaScript. Content that only appears after scripts load is invisible to them.
Your CDN, firewall or security plugin can block AI crawlers without anyone on your team deciding to.
llms.txt files, “chunking” and AI-only rewrites do nothing for Google Search, according to Google’s own guidance.
Fix crawl access, rendering, indexability, site structure and speed first. Then invest in content competitors cannot copy.
What Google's John Mueller Actually Said
Mueller made three points that matter to any business investing in search. Here they are in plain terms.
1. Short-term AI tactics come with an expiry date.
Asked about listicles being used to steer chatbot recommendations, Mueller said a listicle is not spam by default. But a strategy built entirely on one tactic collapses once search and AI systems learn to discount it. A stable, high-quality site is what lasts.
2. Question what your SEO vendors are doing.
Low-quality tactics are not new, he noted. Keyword stuffing and hidden text worked for a while too. His advice to small businesses was to keep asking providers exactly what they are doing and why.
3. Technical SEO is not going away.
AI systems are built on top of search foundations. AI crawlers do behave a little differently, for example crawling for model training rather than for the freshest content. Mueller described that as another flavour of technical SEO, not a replacement for it.
Google's documentation says the same
This is not a one-off comment. Google’s AI features and your website guide states that a page only needs to be indexed and eligible to appear with a snippet to be shown in AI Overviews or AI Mode. There are no additional technical requirements.
Google’s newer guide to optimising for generative AI search goes further. From Google Search’s point of view, optimising for AI search is still SEO. The same guide adds one new check: your site must not be excluded from Search generative AI features in Search Console.
Our take
For eCommerce businesses, this is reassuring and demanding at the same time. Reassuring, because you do not need a parallel “AI strategy”. Demanding, because AI search is less forgiving of technical debt than classic search ever was. The next section explains why.
How AI Search Finds, Reads and Cites Your Pages
An AI answer is the last step in a chain. If any link breaks, your store drops out of the answer, however good your products are.
Access. The crawler must get past your robots.txt, your CDN and your hosting firewall.
Read. Your content must be in the HTML the crawler downloads. Googlebot renders JavaScript; most other AI crawlers do not.
Index. The page must be indexable, canonical and, for Google, eligible to show a snippet.
Retrieve. When someone asks a question, AI systems run several related searches behind the scenes (Google calls this “query fan-out”) and pull supporting pages from their index.
Cite. The answer links to pages that clearly and credibly answer each part of the question.
Google’s AI Overviews and AI Mode draw on Google’s own Search index. ChatGPT, Claude and Perplexity run their own crawlers, with their own rules and capabilities.
The AI crawlers that matter for eCommerce
| Crawler | Operator | What it does | Runs JavaScript? | How you control it |
|---|---|---|---|---|
| Googlebot | Crawls for Google Search, including AI Overviews and AI Mode | Yes | robots.txt; noindex and nosnippet for what Search shows | |
| Google-Extended | Not a separate crawler: a robots.txt token covering Gemini training and grounding | Uses Googlebot's fetches | robots.txt token; does not affect Search inclusion or ranking | |
| OAI-SearchBot | OpenAI | Builds the index behind ChatGPT search | No | robots.txt |
| GPTBot | OpenAI | Collects content for model training | No | robots.txt, independent of OAI-SearchBot |
| ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user asks for it | No | robots.txt may not apply |
| ClaudeBot | Anthropic | Collects content for model training | No | robots.txt |
| PerplexityBot | Perplexity | Builds Perplexity's search index | Not documented; treat as no | robots.txt |
Why JavaScript is the hidden risk
Vercel and MERJ analysed live crawler traffic and found that none of the major AI crawlers, including OpenAI’s and Anthropic’s, executed JavaScript. They download the raw HTML and move on. The same study found AI crawlers requesting a high share of URLs that returned 404 errors.
In practice, a product page that loads its description, specifications or reviews with JavaScript can rank well in Google and still be blank to ChatGPT. A messy redirect history wastes the limited visits these crawlers make.
The Technical SEO Checklist for AI Search Visibility
These are the ten areas we check first when a client asks why their store is missing from AI answers. None of them is new. All of them matter more now.
| # | Area | What to check | Why it matters for AI search |
|---|---|---|---|
| 1 | Crawl access | robots.txt rules per crawler; CDN bot settings; WAF and security plugin rules | Google lists crawl access through robots.txt and any CDN or hosting layer as a best practice. Cloudflare has blocked known AI crawlers by default on new domains since July 2025. |
| 2 | Rendering | Product name, price, stock, description, specs and FAQs present in the raw HTML | Non-Google AI crawlers only see what the server sends. |
| 3 | Indexability | Stray noindex, nosnippet or max-snippet tags; X-Robots-Tag headers; the Search generative AI setting in Search Console | Google's AI features need an indexed, snippet-eligible page on a site that has not opted out. |
| 4 | Canonicals and duplicates | Filter, sort and pagination parameters; http/https and trailing-slash variants | Duplicates waste crawl visits. Google's AI guide explicitly recommends reducing them. |
| 5 | Internal linking | Category-to-product paths, breadcrumbs, related products as real HTML links, orphan pages | Crawlers find what you link to. Google names internal links as an AI features best practice. |
| 6 | Speed and page experience | Server response time, full-page cache, Core Web Vitals, mobile layout | Faster responses help crawlers and shoppers alike. Page experience is on Google's list. |
| 7 | Structured data | Product, Offer, AggregateRating, BreadcrumbList and Organization markup that matches the visible page | Not required for AI features, but mismatched markup undermines trust and rich results. |
| 8 | Product feeds | Merchant Center feed accuracy; price and stock parity with the site; Google Business Profile | Google says AI responses can include product listings and business details from these sources. |
| 9 | Status codes and sitemaps | 404s, redirect chains, soft 404s on discontinued products, sitemaps listing only live canonical URLs | AI crawlers already waste many requests on dead URLs. Do not add to it. |
| 10 | Measurement | Search Console Generative AI performance report; server logs for AI bots; GA4 referrals from AI assistants | You cannot improve visibility you are not measuring. |
A two-minute test you can run today: open a product page in Chrome, disable JavaScript in DevTools and reload. Whatever disappears is invisible to most AI crawlers. Your developer can confirm the same with a command-line request using a crawler’s user agent.
Magento 2 and WooCommerce: Where AI Visibility Breaks
Magento 2 and Adobe Commerce
Layered navigation URL sprawl. Every filter combination can produce a crawlable URL. Without canonical rules and parameter controls, crawlers spend their visits on near-duplicates instead of your best products.
Reviews loaded after the page. On the default Luma theme, the review list on a product page arrives through an AJAX request after load. The star summary is in the HTML, but the review text is not. Googlebot can render it; crawlers that skip JavaScript never see what customers said.
Headless and PWA storefronts. A React or Vue front end rendered in the browser can look perfect to shoppers and Googlebot, yet hand other crawlers an almost empty page. Server-side rendering or pre-rendering is essential.
Staging settings in production. A NOINDEX default under Content > Design > Configuration, or a restrictive robots.txt, copied across during a release. We check this after every deployment.
Performance debt. Heavy themes and a long list of extensions push up response times. Full-page caching with Varnish and a lighter front end such as Hyvä improve both Core Web Vitals and crawl efficiency.
WooCommerce
Security plugins and host-level bot rules. Firewall plugins and managed-hosting bot protection can block unfamiliar user agents, including OAI-SearchBot and PerplexityBot. Review the allow list rather than assuming.
Content injected by page builders. Tabs, accordions and sliders that load product details with JavaScript. Collapsed text is fine, as long as it is in the HTML.
Filter and cart parameters. Filter plugins and add-to-cart links can generate thousands of crawlable URLs. Keep cart and checkout out of the crawl and canonicalise filtered views.
Competing schema. WooCommerce core, your SEO plugin and your theme can each output Product markup, producing duplicate or contradictory data. Choose one source and validate it.
Headless builds. If you have moved to a headless front end, as covered in our headless WooCommerce guide, make sure it renders on the server so crawlers receive full HTML.
A typical scenario
Picture a Magento store selling industrial safety equipment. It ranks on page one of Google for its main categories, yet ChatGPT never mentions it. The server logs show OAI-SearchBot receiving 403 errors from the CDN, and the technical specifications sit inside a JavaScript tab.
Two fixes follow: an allow rule for the search crawler and server-rendered specifications. No new content, no AI-specific files. The store simply becomes readable to every major AI crawler, which is the precondition for being cited at all.
AI SEO Myths You Can Safely Ignore
Google’s own generative AI optimisation guide lists several popular “AEO/GEO hacks” that do nothing for Google Search. Here is our view on each, plus the ones that carry real risk.
- llms.txt files. Google Search ignores them. Mueller has said no AI system currently uses llms.txt and has compared the idea to the old meta keywords tag. Creating one does no harm, but it is not a visibility strategy.
- Markdown copies of pages for bots. More content to maintain, with no documented benefit in Google.
- “Chunking” pages into tiny pieces. Google says its systems understand multiple topics on one page. Structure content for readers.
- Rewriting content for AI. AI systems understand synonyms and intent. Write for your customers.
- Self-promotional listicles at scale. “Top 10 suppliers” pages that rank yourself first may work briefly. As Mueller warned, systems catch up and the traffic disappears.
- A page for every fan-out query. Google says mass-producing pages to target related queries can breach its scaled content abuse policy.
- Showing AI crawlers different content. Serving one version to bots and another to people is cloaking, a long-standing spam violation.
- Buying mentions, directory listings and reciprocal links. Old tactics with new labels. Google says inauthentic mentions help less than they seem, and its spam systems feed the AI features too.
Five questions to ask any agency selling "AI SEO" (including us)
Mueller’s advice to question your vendors is worth acting on. These five questions separate real technical work from rebranded shortcuts.
- Which AI crawlers can access our site today, and how did you verify it?
- What does our product page look like with JavaScript switched off?
- Which of your recommendations are documented by Google or the AI platform itself?
- How will you measure AI visibility: which reports, logs and referral data?
- What happens to our traffic if this tactic stops working next year?
CzarGroup's 30-Day AI Search Readiness Plan
You do not need a rebuild. Most stores can close their biggest gaps in four weeks, in this order.
| Week | Focus | Key tasks | Outcome |
|---|---|---|---|
| 1 | Access | Audit robots.txt, CDN bot settings, WAF and security plugin rules. Review 30 days of server logs for Googlebot, OAI-SearchBot, GPTBot, ClaudeBot and PerplexityBot. Agree a policy for training crawlers. | A written crawler policy and no accidental blocks |
| 2 | Rendering and indexing | Compare raw HTML with the rendered page for home, category, product and content templates. Remove stray noindex and nosnippet tags. Tighten canonicals and filter URLs. | Key content visible to every crawler |
| 3 | Structure and data | Validate Product, BreadcrumbList and Organization schema. Fix internal links and orphan pages. Align the Merchant Center feed with on-site prices and stock. Clean XML sitemaps. | Consistent, machine-readable product data |
| 4 | Measurement and content | Set up the Search Console Generative AI report, a GA4 channel group for AI assistant referrals and log alerts. Plan non-commodity content such as buying guides, comparisons and first-hand expertise. | A baseline to track and a content roadmap |
After that, technical SEO becomes routine maintenance. Re-check access and rendering after every release, extension install and hosting change, because that is when most AI visibility quietly breaks.
FAQ's
Frequently Asked Questions
Yes. Google says AI Overviews and AI Mode have no extra technical requirements, so a page must simply be crawlable, indexed and eligible to show a snippet. Other AI assistants rely on their own crawlers being able to access and read your HTML.
Not for Google. Google's own guidance treats optimising for generative AI search as SEO. A strong technical foundation and genuinely useful, original content serve classic search and AI answers at the same time.
No. Google Search ignores llms.txt, and Google's John Mueller has said no AI system currently uses it. You can add one if a specific service asks for it, but it will not improve your visibility.
It depends on your goals. GPTBot and ClaudeBot collect training data, while crawlers such as OAI-SearchBot and PerplexityBot power AI search results. Many stores block training crawlers and allow search crawlers; just make sure your CDN or firewall is not blocking the ones you want.
Googlebot can, but most other AI crawlers cannot. Research by Vercel and MERJ found that major AI crawlers, including OpenAI's and Anthropic's, do not execute JavaScript. Keep product names, prices, descriptions, specifications and reviews in the server-rendered HTML.
Structured data is not required for AI Overviews or AI Mode, and there is no special AI schema. It is still worth implementing accurately, because it supports rich results and gives every system a consistent view of your products, as long as it matches the visible page.
No. Google-Extended controls whether your content is used for Gemini training and grounding, and Google states it does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode follow your normal Googlebot and snippet controls and the Search generative AI setting in Search Console.
In Google, use the Generative AI performance report in Search Console; AI Overviews and AI Mode clicks also count towards the standard Performance report. For other assistants, check server logs for their crawlers and GA4 referrals from domains such as chatgpt.com and perplexity.ai.
Final Thoughts
AI search has not rewritten the rules of SEO. It has raised the cost of ignoring them. A blocked crawler, a JavaScript-only product description or a leftover staging noindex used to cost you some rankings. Now it can remove you from the answer entirely, in Google and in every AI assistant your customers use.
The good news is that these are solvable engineering problems, and fixing them helps every channel at once. Get the foundations right, then put your energy into content only your business can write: real product expertise, honest comparisons and first-hand experience.
If you are tightening bot protection at the same time, our guide to bot traffic on eCommerce websites explains how to stop bad bots without locking out the good ones.
Want to know how your store looks to Google, ChatGPT and Perplexity? Our Magento and WooCommerce specialists can run an AI search readiness audit covering crawler access, rendering, indexability and structured data, with a prioritised fix list. Talk to our eCommerce and SEO team or explore our digital marketing services.
