AI crawler

also known as: AI bot · answer-engine crawler

The automated agents that answer engines and AI assistants use to fetch, render and index pages before citing them: GPTBot and OAI-SearchBot for ChatGPT, PerplexityBot for Perplexity, ClaudeBot for Claude, Google-Extended for Google’s AI Overviews and AI Mode. Each engine runs its own crawler under its own user agent, and a page that disallows one in robots.txt has opted out of citation by that engine specifically, whatever else the page gets right stops mattering to it.

In AI Search API

This site’s own robots.ts names GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Claude-Web and Google-Extended explicitly and allows each one full-content access, a product whose entire job is reporting what these engines say would be a contradiction if it hid from the same engines’ crawlers. The allowlist sits next to the standard-crawler rule rather than replacing it, so ordinary search indexing is unaffected. Every marketing page, glossary entry and surface spoke is written to be genuinely extractable by these crawlers, not just permitted.

robots.ts, the AI crawlers this site allows

const AI_BOTS = [
  'GPTBot', 'OAI-SearchBot', 'ChatGPT-User',
  'PerplexityBot', 'ClaudeBot', 'Claude-Web',
  'Google-Extended',
];
// each gets: { userAgent, allow: '/', disallow: DISALLOW }

Related terms

See it in the product

Every definition here is enforced in the Envelope contract. Run a capture and read the fields yourself.

Start with 500 free credits All terms