Nine robots read your funeral home's website — blocking the wrong one costs you the citation

OpenAI publishes four web robots. Anthropic publishes three, Perplexity two, and Google adds a control token that isn’t a robot at all. Every one of them checks a plain-text file on your funeral home’s website — robots.txt — before deciding what it’s allowed to do. Most owners have never opened that file — and whoever built the website may have edited it without asking.

Here’s the problem: the robots don’t all do the same job, and a block aimed at one job usually lands on another. The line an owner adds to keep his content out of AI training is one setting. The line that decides whether ChatGPT can name his funeral home when a family asks who to call is a different setting. Confuse them and you keep the training and lose the citation.

The three jobs a robot can have

Every one of the four companies splits its robots the same way:

1. Training crawlers read your pages so future AI models can learn from them.
2. Search indexers read your pages so the AI can find and cite them when someone asks a question.
3. User fetchers grab one page because a real person just asked the AI to look at it.

No company uses one robot for all three jobs. That separation is the whole game: you can keep your content out of training datasets and still be fully visible in AI answers — or block everything and quietly vanish. The companies say this themselves, in their own documentation. Here’s what each one publishes.

OpenAI: four robots, one expensive mix-up

OpenAI’s robot documentation lists four agents, and states that each setting is independent of the others — a site can allow one while disallowing another.

GPTBot is the training crawler. OpenAI’s words: “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” Blocking it is a decision about training, and only training.

OAI-SearchBot is the search indexer, and this is the line that matters for visibility. From the same page: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” That’s the citation — gone — in OpenAI’s own sentence.

ChatGPT-User fetches pages when a person asks. OpenAI notes it “is not used for crawling the web in an automatic fashion,” and that because these actions are user-initiated, “robots.txt rules may not apply.”

OAI-AdsBot checks pages submitted as ads on ChatGPT, and OpenAI states its data “is not used to train generative AI foundation models.”

So a funeral home that blocked GPTBot kept itself out of training and lost nothing in ChatGPT’s answers. A funeral home that blocked everything OpenAI at once took itself out of ChatGPT search answers without knowing it.

Anthropic: the same split, different names

Anthropic documents three robots, split exactly the same way. ClaudeBot is training: restricting it “signals that the site’s future materials should be excluded from our AI model training datasets.” Claude-SearchBot is the index: disabling it “prevents our system from indexing your content for search optimization, which may reduce your site’s visibility and accuracy in user search results.” Claude-User is the person-asked fetch.

One practical note from the same page: Anthropic discourages blocking by IP address, because doing so “impedes our ability to read your robots.txt file.” If the robot can’t read the rules, it can’t follow them.

Perplexity: neither robot trains

Perplexity’s documentation is the simplest of the four: PerplexityBot “is not used to crawl content for AI foundation models,” and Perplexity recommends allowing it so your site appears in its search results. Perplexity-User fetches on request, and because a user asked for the page, it “generally ignores robots.txt rules.” There is no Perplexity training crawler to block at all — a block here only ever costs visibility.

Google is the strange one

Google-Extended appears in robots.txt files like a robot, but Google’s crawler documentation says it “doesn’t have a separate HTTP request user agent string” — crawling happens with Google’s existing agents, and the token works “in a control capacity.”

What it controls is broader than most write-ups say: whether your content trains future Gemini models and whether it’s used for grounding — Google’s word for pulling your live content into a Gemini answer at the moment someone asks. And what it doesn’t touch is stated just as plainly: “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”

Read that as a funeral home owner: blocking Google-Extended to stay out of AI training also opts you out of being quoted by Gemini when a family in your county asks it a question. Your Google Search ranking doesn’t move either way. That’s the trade, and it’s a real one — AI Overviews sit on top of the funeral queries your families already type.

The nine robots, one table

Agent	Company	Its job	Blocking it means
GPTBot	OpenAI	training	your pages don’t train future OpenAI models
OAI-SearchBot	OpenAI	search index	you’re not shown in ChatGPT search answers
ChatGPT-User	OpenAI	user fetch	little — it may not honor robots.txt
OAI-AdsBot	OpenAI	ad checks	ads-related only; it doesn’t train either way
ClaudeBot	Anthropic	training	future Claude models exclude your site
Claude-SearchBot	Anthropic	search index	reduced visibility in Claude’s search results
Claude-User	Anthropic	user fetch	a Claude user’s request may not reach your page
PerplexityBot	Perplexity	search index	you fall out of Perplexity’s results
Perplexity-User	Perplexity	user fetch	little — it generally ignores robots.txt
Google-Extended	Google	a control token, not a robot	no Gemini training — and no Gemini quoting you at answer time. Google Search: unaffected

What to actually do, this week

Open the file. Type your own domain followed by /robots.txt into a browser — yourfuneralhome.com/robots.txt. It’s public; there’s nothing to log into. Search the page for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended.

Keep the search robots on. OAI-SearchBot, Claude-SearchBot, and PerplexityBot are how AI answers find and cite your funeral home. When we measured 924 funeral home websites, we could read 827 of them, and 35.6% of those scored a D or F on the foundations AI answers are built from — and a robots.txt that blocks the search indexers caps what any other fix can do. If a “Disallow” line sits under any of those three names and nobody remembers choosing it, that’s the first thing to change.

Decide training on purpose. Whether OpenAI or Anthropic trains on your obituary pages is a values call, and reasonable owners land on both sides of it. Make it deliberately, knowing it’s independent: blocking GPTBot and ClaudeBot costs you nothing in AI answers today.

Ask whoever runs your website what they chose for you. Any website company can read this same file and flip the same lines — this isn’t proprietary to anyone. If a block-everything set of lines is in there and nobody remembers choosing it, it likely arrived with the platform, not with a decision. The question for your vendor is one sentence: “What does our robots.txt say about AI crawlers, and who decided that?”

Give changes a day. OpenAI notes it can take about 24 hours for a robots.txt update to register. The user fetchers will keep ignoring the file either way, and that’s fine — those only fire when a person asks for your page, which is the kind of attention you want.

When a family asks ChatGPT which funeral home to call, the answer is assembled from pages a robot was allowed to read. Ten lines in a text file you’ve never opened are deciding, right now, whether your funeral home is in that answer. The audit checks exactly this — but you don’t need us for it. Open the file.