Definition
The robots.txt file, served at /robots.txt, lists rules crawler by crawler: what it may read (Allow) and what it should leave aside (Disallow). It can also declare the sitemap. Reputable crawlers respect it, but it is not a protection.
Google states that this file is not a way to keep a page out of its results: a blocked page can stay indexed if other sites link to it. Keeping it out takes a noindex rule or password-protected access.
Value for a business owner
robots.txt also decides presence in assistants. OAI-SearchBot feeds ChatGPT search and GPTBot the training of OpenAI's models; Claude-SearchBot and ClaudeBot play the same roles for Anthropic; PerplexityBot builds Perplexity's index. Google-Extended concerns the Gemini apps and Vertex AI, with no effect on Google Search.
Each crawler is set separately: blocking GPTBot does not stop OAI-SearchBot from reading the site. Blocking a search crawler, however, removes the site from the matching assistant's answers.
At Stratedge Consulting
Our robots.txt names each assistant crawler and forbids private areas to all. The AI visibility audit reads yours the way Google reads it, crawler by crawler, and reports what removes you from answers.
Common mistakes
Keeping a staging site's “Disallow: /” line after going live. The whole site disappears from search engines and assistants, often for weeks before anyone notices.
Using robots.txt to hide a sensitive page. The file is public and points exactly where to look; a private page is protected by a password.