In short: before you block GPTBot or any other AI crawler, know that AI companies run two kinds of bots: some collect pages that may be used to train their models (GPTBot, ClaudeBot), others read pages so their assistant can answer questions with links (OAI-SearchBot, Claude-SearchBot, PerplexityBot). OpenAI says each setting is independent, so you can keep GPTBot out and still let ChatGPT search show your site. A business that wants customers to find it in AI assistants usually allows the answer crawlers and decides about training separately.
Many WordPress owners first hear about AI crawlers from a security plugin or a forum post that says “block GPTBot now”. That can be the right choice for a site that sells its articles. For a shop in Accra or a guesthouse in Musanze, a careless rule can quietly remove the business from the answers its customers read.
Why does this choice matter for a business website?
When someone asks an assistant “where can I buy solar panels in Kisumu?”, it often searches the web and quotes the pages it could read. A page its bot may not read is much less likely to be part of that answer. So a robots.txt rule, written once and forgotten, can decide whether your pages are even in the running.
The opposite worry is also real: you may not want your writing used to train AI models. The main AI companies now separate the two uses, so you rarely have to choose between “everything” and “nothing”.
Two kinds of AI crawlers: training and answering
- Training crawlers collect public pages that may be used to build future models. Blocking them signals that you do not want that use.
- Answer and search crawlers read pages so the assistant can quote and link them. Blocking them can keep you out of answers.
- User-triggered fetchers visit a page because a person asked about it. Some of them do not follow robots.txt for that reason.
The main AI bots, one by one
What each company says in its own documentation:
| Bot | Company | What it does | If you block it |
|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI’s foundation models. | Signals that your content should not be used for training. OpenAI says this is independent of search. |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT’s search features. | OpenAI says opted-out sites are not shown in ChatGPT search answers (they can still appear as navigational links). |
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT user’s request needs it; not automatic crawling. | OpenAI says robots.txt rules may not apply, and this bot does not decide whether you appear in search. |
| ClaudeBot | Anthropic | Collects web content that could contribute to model training. | Signals that your future content should be excluded from Anthropic’s training data. |
| Claude-SearchBot | Anthropic | Reads content to make Claude’s search answers more relevant and accurate. | Anthropic says this may reduce your visibility in user search results. |
| Claude-User | Anthropic | Fetches pages when people ask Claude questions. | Anthropic says this may reduce your visibility for user-directed web search. |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity results; Perplexity says it is not used to crawl content for AI foundation models. | Perplexity recommends allowing it if you want your site to appear in its results. |
| Perplexity-User | Perplexity | Visits a page when a user’s question needs it. | Perplexity says this fetcher generally ignores robots.txt, because a person requested the visit. |
| Google-Extended | Not a separate crawler: a robots.txt token that controls use of your content for training Gemini models and for grounding Gemini Apps answers. | Google says it does not affect your inclusion or ranking in Google Search. |
So what should a small business do?
You want customers to find you in AI answers
This is the case for most shops, hotels, clinics, schools and consultants. Allow OAI-SearchBot, Claude-SearchBot, Claude-User and PerplexityBot. Then decide about GPTBot, ClaudeBot and Google-Extended on their own merits: according to OpenAI and Google, blocking them does not remove you from ChatGPT search or Google Search.
Your content is what you sell
If you publish paid reports or courses, you may block the training crawlers and keep the answer crawlers, so assistants can still point readers to you.
Some pages should stay private
Accounts, carts and checkout have nothing to offer an assistant. Block those paths for every bot instead of blocking a bot everywhere. Truly private content belongs behind a login: robots.txt is a public request, not a lock.
How do I block or allow these bots on WordPress?
Bots read a plain text file at yourdomain.com/robots.txt. WordPress creates one automatically, and SEO plugins often let you edit it. A rule that blocks OpenAI’s training crawler but leaves its search crawler alone looks like this:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /A few practical points:
- Check every subdomain. Anthropic notes that each subdomain needs its own robots.txt rules, so a shop on shop.yourdomain.com is not covered by the main site’s file.
- Do not rely on blocking IP addresses. Anthropic warns that IP blocking may not work reliably and can stop its bots from reading your robots.txt at all.
- Give it time. OpenAI says search changes can take about 24 hours after you update robots.txt.
- Look beyond robots.txt. Some hosting firewalls block bots on their own: check them if a bot never shows up.
If you use Amaso AI Free, you can set Allow or Disallow for the AI crawlers it lists under Settings, AI Engines (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others; OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User are added in version 1.2.0), without editing the file by hand. These rules go into the robots.txt that WordPress generates, not into a robots.txt file uploaded to your server. Amaso AI names each bot once, keeps your site’s own rules (such as keeping bots out of wp-admin and cart links), keeps blocked bots blocked and leaves a site that discourages search engines alone. Our Amaso AI Free setup guide walks through the first steps.
How do I know whether the bots actually visit?
Your server logs record each visitor’s user agent, so a developer can search them for GPTBot or PerplexityBot. Amaso AI Free keeps simpler local records inside WordPress: recognized AI crawler visits and visits referred by AI assistants.
A crawler visit shows that a bot fetched a page; it does not prove the page was indexed or cited. To see whether you are actually named in answers, run the questions your customers ask, as explained in how to check whether ChatGPT cites your WordPress site. A file such as llms.txt is a separate, optional topic; our llms.txt explainer covers what it can and cannot do.
Frequently asked questions
If I block GPTBot, will my site disappear from ChatGPT?
Not according to OpenAI. GPTBot is about training, while OAI-SearchBot decides whether your site can be shown in ChatGPT search answers, and OpenAI says the two settings are independent.
Does blocking Google-Extended hurt my Google ranking?
Google says no: Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls the use of your content for Gemini training and for grounding Gemini Apps answers.
Does PerplexityBot use my pages to train AI models?
Perplexity says PerplexityBot is not used to crawl content for AI foundation models. It is the bot that surfaces and links websites in Perplexity’s results.
Why do I still see ChatGPT-User or Perplexity-User after blocking them?
Both companies describe these as user-triggered visits, and say robots.txt may not apply or is generally ignored for them. They visit because a person asked about a page.
Can I block AI bots from only part of my site?
Yes. Use a Disallow rule for the paths you want to protect, such as /my-account/ or /checkout/, and leave the rest open.
Want to see which of the main AI crawlers visit your site and set them in a click? Install Amaso AI Free free from WordPress.org, or compare the paid plans for scheduled checks and alerts.
0 Comments