Allow the ChatGPT crawler on Cloudflare, then check that ChatGPT names you

OpenAI's crawler documentation lists three user agents. OAI-SearchBot surfaces websites in ChatGPT's search features, and OpenAI says sites opted out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links. GPTBot crawls content that may be used to train OpenAI's foundation models. ChatGPT-User makes visits when a person asks ChatGPT or a Custom GPT a question.

OpenAI states that each setting is independent, so you can allow OAI-SearchBot and disallow GPTBot. It also notes that ChatGPT-User visits are user-initiated, so robots.txt rules may not apply to them, and that ChatGPT-User is not used to decide whether content appears in Search. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges.

It depends on when the domain joined. Cloudflare's 1 July 2026 post says that for new domains the Training and Agent categories are blocked by default on pages that display ads, while Search stays allowed. Its 15 September 2026 post gives the full table: new domains without ads get Search, Training and Agent all on Allow, and new domains monetized with ads get Search on Allow, Training on Disallow AI Training and Agent blocked on pages with ads.

Existing domains map from their old settings. Cloudflare says a domain that never used the granular controls and had the legacy Block AI setting off maps to Allow for all three, and a domain that used the legacy Block settings maps to Search Allow, Training Disallow AI Training and Agent blocked on pages with ads. Check the current setting for your own domain rather than assuming it.

Cloudflare's AI Crawl Control docs say it is available on all plans. Open the dashboard, pick your domain, go to AI Crawl Control and its Security tab. The Crawlers table lists each crawler's category, requests, robots.txt violations and action, and you can filter by name or operator. Choose Allow in the Actions column, and you can still enforce robots.txt for that crawler.

Cloudflare's verified bots documentation classifies OAI-SearchBot as an AI Search crawler, and notes that since a taxonomy change on 1 July 2026 AI search and traditional search are both treated as Search. On the free plan, Cloudflare says detection relies on user agent strings, and higher plans add Bot Management detection. Blocking a crawler creates or updates a WAF custom rule, and the block response of 403 or 402 can be configured only on paid plans.

Three reasons come up. First, another rule may still block the request: Cloudflare says an Allow applies unless another setting or a WAF rule blocks the crawler, and its AI Crawl Control docs say unsuccessful requests may come from any rule or response error, not just its block action. Second, your robots.txt may disallow the crawler, since Cloudflare's Bot Preference Sync publishes the no-training preference there. An SEO consultant on Reddit's r/CloudFlare wrote that the older Managed robots.txt feature "silently injects" Disallow lines for GPTBot and ClaudeBot at the top of robots.txt, and that 32% of 50 client sites had it on. That count is the poster's own and we did not verify it, but Cloudflare's 15 September 2026 post does say Managed Robots.txt is to be deprecated in favor of Bot Preference Sync, so view your live robots.txt. Third, OpenAI says it can take about 24 hours after a robots.txt update.

Access may also never have been the problem. The NytroSEO guide that ranks for this query ends with a checklist of HTTP 200 checks, logs, sitemap access and referral traffic from chatgpt.com, and it offers no baseline of whether AI systems cite the site. Access is necessary for being named, not sufficient. That is the part Linkeddit measures: on the test project the one open gap was a comparison question, a content gap that no setting would fix.

That is a business choice, and Cloudflare built the settings to make it granular. Its Disallow AI Training option publishes the no-training preference in robots.txt, and the mixed-use crawlers of Applebot, Bingbot and Googlebot keep crawling for search under it. Cloudflare warns that the full Block setting stops Applebot, Bingbot and Googlebot from reaching your site, search included.

If you want to be recommended by answer engines, the search category is the one to leave on Allow. Decide training separately, and record the decision, so a later change to the setting does not quietly take you out of answers.