State of AI Bots 2025
Comprehensive analysis of the AI crawler traffic explosion, impact on publishers, and trends from H1 2025
Key Indicators
AI Crawler Distribution
GPTBot / OAI‑SearchBot.
ClaudeBot.
Perplexity, Amazon, Apple, Meta, Gemini, Mistral…
Q3 2025 indication (FR panel).
URL Hallucination Rate by LLM
Frequency of invented or incorrect URLs in responses
OpenAI
26%Perplexity
27%Anthropic
32%Gemini
39%Mistral
43%💡 Note: Hallucinations include URLs with typos close to real domain names. These errors can direct users to incorrect or non-existent sites.
robots.txt Adoption and Compliance
Analysis of 7,719 popular websites
Block rate by AI bot
Explicit allow rate
💡 Insight: GPTBot and CCBot are the most blocked (17% and 16%), while only 6% of sites explicitly allow them. Most sites take no action (neither blocking nor explicit authorization).
Executive Summary
Q3 2025 marks a turning point in web history: AI bots now represent 31.5% of web traffic (median), with a ratio of 46 AI requests per 100 human visits. This surge in automated crawlers is radically transforming the digital ecosystem.
OpenAI dominates AI crawler traffic with 47% of requests (median), followed by Anthropic (19% average). However, URL hallucinations remain a major issue: from 26% (OpenAI) to 43% (Mistral) of URLs recommended by LLMs contain errors or don't exist.
Meanwhile, AI referral traffic remains catastrophically low (0.005% median), 4,564 times less than Google (22.82%). Only 17% of sites block GPTBot via robots.txt, while 89% have configured rules. The open web faces an unprecedented imbalance between massive content extraction and actual traffic generation.
Market Evolution 2024 → 2025
- Bytespider -85%
- Amazonbot -35%, Applebot -26%
- "ChatGPT-User" +2825% → 1.3%
- PerplexityBot +157,490% (part faible)
- ≈14% des top domaines; GPTBot le plus bloqué (et le plus autorisé)
Sector-wide orders of magnitude. Your actual mix depends on your vertical, size, and visibility. Decisions should remain guided by your metrics above.
What This Means for Your Acquisition
Measure AI Footprint
Track AI share and AI/Human ratio by source/crawler to prioritize efforts.
Control Access
Adjust robots.txt, rate-limits and API policies to balance visibility and protection.
Optimize Discoverability
Improve quality signals (schema, performance, linking) to be recommended by LLMs.
How Senthor Analyzes the Market
Senthor relies on real-time traffic analysis from thousands of partner sites worldwide. Our platform automatically detects and identifies AI bots through advanced behavioral signatures, far beyond simple User-Agent analysis.
This unique position allows us to observe market trends in real-time: new bots, evolving crawling patterns, protection bypasses. We keep this study updated to reflect the constantly evolving AI ecosystem.
Méthodo: priorisation des métriques Senthor (médiane/moyenne par site). À défaut, benchmarks sectoriels agrégés.
Advanced Detection
Bot identification via behavioral signatures, request patterns, and HTTP header analysis. More accurate than robots.txt.
Real-time Monitoring
Analytics dashboard to track who visits your site, which AI, what frequency, which pages. Continuously updated data.
Protect and Monetize Your Content Against AI
Senthor helps you detect, control and monetize AI bot traffic on your site. Real-time analytics, selective blocking without breaking SEO, and monetization opportunities.