Article
November 1, 2025
Senthor Team
Why your robots.txt is no longer enough against AI
Robots.txt used to work. With AI crawlers (GPT, Perplexity, Claude), it's mostly symbolic. Here's why, and how to protect without breaking SEO.
For decades, robots.txt set a polite boundary between search engines and publishers. With generative AIs like ChatGPT, Perplexity, Claude or Gemini, that convention no longer holds.
These bots don’t index your pages, they extract them, summarize them and monetize elsewhere, often ignoring your instructions.