Article · 5 min read

Overview of OpenAI Crawlers

OpenAI Free Intermediate

OpenAI's documentation of its three crawlers and what each robots.txt setting does, including the one that decides whether your site can appear in ChatGPT search answers.

  • OAI-SearchBot surfaces sites in ChatGPT search; sites that block it are not shown in ChatGPT search answers.
  • GPTBot collects content for model training and can be blocked separately from search.
  • ChatGPT-User makes user-initiated visits and does not decide whether you appear in search.
Open on developers.openai.com ↗

The question it answers

  1. How is SEO changing with AI answers and ChatGPT, and should I even care as a small founder?

Read with it

  1. This is the clearest plain-language explainer of what GEO actually means: getting your content cited when ChatGPT, Perplexity,... Backlinko Article · ~20 min read Generative Engine Optimization (GEO): How to Win in AI Search Asif Ali
  2. If the GEO article gets you excited, read this one right after to stay grounded. Heitzman, who runs an SEO agency, shows that... Search Engine Land Essay · ~10 min read AI hype vs. SEO reality: What actually drives leads and revenue Adam Heitzman
  3. A calm, forward-looking conversation with an SEO advisor who has worked with Zapier, Tinder, and Coinbase, aimed squarely at... Lenny's Podcast Podcast · ~1 hr 15 min Rethinking SEO in the age of AI (Eli Schwartz) Lenny Rachitsky with Eli Schwartz
  4. Google's own guidance for site owners on how pages appear in AI Overviews and AI Mode, and how to measure and control that. Google Article · 8 min read AI features and your website Google Search Central
eChai Partner Brands The partnership →