Article
November 1, 2025
Senthor Team
Why your robots.txt is no longer enough against AI
Robots.txt used to work. With AI crawlers (GPT, Perplexity, Claude), it's mostly symbolic. Here's why — and how to protect without breaking SEO.
For decades, robots.txt set a polite boundary between search engines and publishers. With generative AIs like ChatGPT, Perplexity, Claude or Gemini, that convention no longer holds.
These bots don’t index your pages — they extract them, summarize them and monetize elsewhere, often ignoring your instructions.