The useful question is not “should I block AI bots?” It is “which ones, and for what?”

Three kinds of AI bots

AI companies run different bots for different jobs:

• Training bots collect pages to train AI models.
• Search bots read pages so an AI assistant can find and cite them in answers.
• User-triggered bots visit a page because a person asked an assistant to open it.

Google works differently. Google-Extended is not a crawler. It is a token you can use in robots.txt to control whether Gemini can use your pages for training and grounding. It does not change whether your pages appear or rank in Google Search.

What robots.txt can and cannot control

robots.txt is a request, and each company decides how its bots treat it. For training and search bots, the companies above say they follow it. For user-triggered visits it is mixed: Claude-User honors it, OpenAI says it may not apply to ChatGPT-User, and Perplexity-User generally ignores it.

So robots.txt does not control every AI visit. It controls the bots that choose to follow it.

An example: allow search, block training

If you want AI assistants to find and cite your pages but do not want them used for training, a robots.txt like this does that for the bots that follow it:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /

OAI-SearchBot, Claude-SearchBot and PerplexityBot fall under the last rule and stay allowed.

Think twice before adding Google-Extended to that list. It also covers grounding, so blocking it can keep Gemini from using your pages in its answers, not just in training.

What I do on this site

This site’s robots.txt allows every crawler and points to the sitemap. I want AI assistants to find and describe my work correctly, so I do not block any of them. A store might decide differently about training, and that is a fair choice. Just do not block the search bots by accident.

To see what AI tools can and cannot read on your own store, see AI discoverability or why Shopify products do not show up in ChatGPT.

I built an AI agent, product_geo_agent, that checks these things on a Shopify product and scores it from 0 to 100. Here is what it found on three products, and how you can check your own store.

What an AI assistant needs from a product page

Before anything else, the page has to be reachable. If a store’s robots.txt blocks OAI-SearchBot, the crawler OpenAI uses for ChatGPT search, that crawler will not read those pages. My post on GPTBot and robots.txt covers which bots do what.

After that, my agent looks at seven things. The weights are my own choice, not a published standard:

The AI model judges the writing. Plain Ruby code does the math, so the score is the same sum every time for the same judgments.

Three real audits: 70, 33 and 10 out of 100

These are single runs on a Shopify development store with Shopify’s sample snowboard products, recorded on 2 October 2026.

Two honest notes. First, single runs move a little: across five evaluation runs, the complete snowboard scored between 65 and 70, because the model rated its specs “fair” three times and “good” twice. Second, the recommendation check is simple. The agent picks a buyer-style category, asks a Gemini model one general question, and checks whether the product name appears in the answer. In a run on 29 September 2026 the question was “What’s a good all mountain snowboard you’d recommend?”, and the answer named well-known brands instead. Sample products on a test store were never going to be named, so all three lost those 15 points.

The fixes that matter most

Going by where these three products lost points:

1. Put the buyer’s questions into the description. Sizing, materials, compatibility, what is included. This was the biggest single loss for the middle product.
2. Add FAQ content. A store FAQ page, or FAQs on the product itself.
3. Make the structured data complete. The Product block needs a real description and offer details, not just a name.
4. Write alt text for product images.
5. Check robots.txt so the search crawlers you want can reach your pages.

What nobody can promise

Fixing these does not guarantee that ChatGPT, Perplexity or Google AI will recommend a product. Each assistant decides on its own. The fixes remove what stops them from reading and understanding the product.

How to check your own store

1. Open yourstore.com/robots.txt and look for AI bot names such as OAI-SearchBot, GPTBot or PerplexityBot, and whether they are disallowed.
2. Read one product page as a buyer. Write down five questions a buyer would ask, and check whether the page answers each one in text.
3. Run the product URL through Google’s Rich Results Test and check that the Product data has a description and an offer.
4. Ask ChatGPT or Perplexity the question your buyer would ask, and note whether your store appears and how it is described.

If you want this done for you, see AI discoverability and a sample report.