Does blocking GPTBot stop me appearing in ChatGPT?

It stops OpenAI’s training crawler from ingesting your content. To appear in ChatGPT’s live answers you especially want to allow the retrieval bots, OAI-SearchBot and ChatGPT-User: which fetch pages when a user asks a question. Blocking any of them reduces the chance you are read and cited.

Is robots.txt legally binding on AI companies?

robots.txt is a voluntary standard. The major operators (OpenAI, Anthropic, Google, Perplexity) publicly state they respect it, so in practice a disallow line does keep the well-behaved crawlers out. The point for a business that wants visibility is the reverse: make sure you are not disallowing the bots you actually want.

I unblocked the crawlers. How long until I show up?

Retrieval crawlers can fetch a page within days once access is open and the content is indexable. Being consistently cited takes longer and depends on structure, authority and freshness. The first, fastest win is simply removing the block so the machines can start reading you at all.

How do I check without editing anything?

Run our free scan, it fetches your site the way an AI crawler would, checks robots.txt and the network response together, and reports exactly which bots can and cannot read you, with no signup.

Is Your Website Blocking ChatGPT? How to Check in 2 Minutes, Footnote Works

The short answer

To check whether your site blocks ChatGPT, open yourdomain.com/robots.txt and look for lines that disallow GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot, and separately confirm your CDN or firewall (Cloudflare especially) is not blocking those bots at the network level. If either layer blocks them, the assistants cannot read your pages and will never cite you. The fix is to explicitly allow the AI user-agents in robots.txt and in your CDN’s bot rules.

Key takeaways

Two layers can block AI crawlers: your robots.txt file and your CDN/firewall (e.g. Cloudflare). Check both: a clean robots.txt does not mean you are visible.
Cloudflare began blocking AI crawlers by default for new sites on 1 July 2025, so many businesses are invisible to AI without ever choosing to be.
The crawlers that matter: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, and CCBot.
If a bot is blocked, the assistant cannot read, quote or cite your site. You are handing the answer to a competitor.

Why this happens by accident

Almost no business decides to hide from ChatGPT. Yet a large share of sites do exactly that, because blocking happens in places owners never look. A theme or SEO plugin ships a restrictive robots.txt. A developer copies a blocklist from a forum thread. Or, most common in 2025 to 2026: the CDN does it automatically.

Jul 1, 2025

Cloudflare began blocking AI crawlers by default for new domains, making “allow AI bots” an opt-in choice most owners never see. Cloudflare

That policy exists for good reasons, publishers wanted leverage over AI training. But for a service business that *wants* to be recommended by AI assistants, the default quietly works against you. Blocking the crawler that feeds ChatGPT is the AI-era equivalent of a noindex tag on Google.

The AI crawlers that matter

Not all bots do the same job. Two categories matter: training crawlers (ingest content to train models) and retrieval crawlers (fetch pages live to answer a user right now). Retrieval bots are the ones that put you in an answer with a clickable citation, so those are the ones you least want to block.

User-agent	Operator	Job
GPTBot	OpenAI	Training crawler
OAI-SearchBot	OpenAI	Retrieval (ChatGPT Search)
ChatGPT-User	OpenAI	Live fetch when a user asks
ClaudeBot	Anthropic	Training + retrieval
PerplexityBot	Perplexity	Retrieval
Google-Extended	Google	Gemini / AI training opt-in
CCBot	Common Crawl	Open dataset many models use

The user-agents to allow if you want to be read by AI assistants. Retrieval bots are the highest priority.

Check it in two minutes

Read your robots.txt
Visit https://yourdomain.com/robots.txt. Search the text for GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot. A line like User-agent: GPTBot followed by Disallow: / means you are blocking that bot. A blanket User-agent: * with Disallow: / blocks everything.
Test the network layer
A clean robots.txt is not enough, your CDN or firewall can return a 403 before the bot ever sees the file. The reliable test is to fetch a page while pretending to be the bot and read the status code, which is exactly what an AI-visibility scan does for you.
Confirm you are actually being read
The ground truth lives in your server access logs. Grep for GPTBot, ClaudeBot and ChatGPT-User and you will see whether the assistants are fetching your pages, and how often. On ebikeoracle.com, a site we built and still host, ChatGPT-User is the single most active crawler, ahead of Googlebot.

The fix

At the robots.txt layer, explicitly welcome the AI user-agents instead of relying on a blanket wildcard:

Add an explicit User-agent: GPTBot block with Allow: /, and the same for OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended.
In Cloudflare (or your WAF), turn off any “block AI scrapers / bot fight mode” rule that catches these user-agents, or add an explicit allow rule for them.
Make sure the pages themselves render real HTML on the first request. Many AI crawlers do not run JavaScript, so a page that only assembles content client-side reads as blank even when it is “allowed”.

Questions

Does blocking GPTBot stop me appearing in ChatGPT?: It stops OpenAI’s training crawler from ingesting your content. To appear in ChatGPT’s live answers you especially want to allow the retrieval bots, OAI-SearchBot and ChatGPT-User: which fetch pages when a user asks a question. Blocking any of them reduces the chance you are read and cited.
Is robots.txt legally binding on AI companies?: robots.txt is a voluntary standard. The major operators (OpenAI, Anthropic, Google, Perplexity) publicly state they respect it, so in practice a disallow line does keep the well-behaved crawlers out. The point for a business that wants visibility is the reverse: make sure you are not disallowing the bots you actually want.
I unblocked the crawlers. How long until I show up?: Retrieval crawlers can fetch a page within days once access is open and the content is indexable. Being consistently cited takes longer and depends on structure, authority and freshness. The first, fastest win is simply removing the block so the machines can start reading you at all.
How do I check without editing anything?: Run our free scan, it fetches your site the way an AI crawler would, checks robots.txt and the network response together, and reports exactly which bots can and cannot read you, with no signup.

Cite or link this article

Found this useful? Link to it.

This article is free to quote and reference. A link helps more people find accurate, evidence-cited answers, and you're welcome to reuse it with attribution under CC BY 4.0. Grab a ready-made link or citation:

Link (HTML)

<a href="https://footnoteworks.com/blog/is-my-website-blocking-chatgpt">Is Your Website Blocking ChatGPT? How to Check in 2 Minutes, Footnote Works</a>

Link (Markdown)

[Is Your Website Blocking ChatGPT? How to Check in 2 Minutes, Footnote Works](https://footnoteworks.com/blog/is-my-website-blocking-chatgpt)

Citation

John Weeks. “Is Your Website Blocking ChatGPT? How to Check in 2 Minutes.” Footnote Works, 2026, https://footnoteworks.com/blog/is-my-website-blocking-chatgpt.

See where you stand

Can AI actually read your site?

Run the free scan. It fetches your site the way an AI crawler does and reports exactly what the machines can and cannot read. No signup.

Run the free scan or book the free call

John Weeks , AI agency. We build the systems firms run on, automate the work around them, and prove it with first-party data. About the studio.

Is Your Website Blocking ChatGPT? How to Check in 2 Minutes