Cloudflare is blocking AI crawlers by default: a signal worth reading, and why a platform block is not the fix

1. The signal: sites are pushing back on the AI crawl

Here is the news, stated plainly. From September 15, 2026, Cloudflare will by default block multi-purpose crawlers on ad-supported pages, for free-tier customers and new signups, unless the owner opts out. Multi-purpose crawlers are bots that both index for search and harvest content for AI. Because Google runs a single crawler for both, the change lands hardest on Google.

Read past the headline and the useful part is the direction of travel, not the specific tool. After two years of AI systems consuming the open web for free, site owners are starting to treat the AI crawl as something they get to decide about. That instinct is correct. What matters next is how you act on it, because a switch flipped by your CDN is a long way from a strategy you own.

2. Why the pressure is real

None of this is theoretical. AI summaries have quietly rerouted the value of the web, and the numbers on publishers are hard to ignore.

The tension is simple: AI systems extract your content for free while removing the traffic, and the ad revenue, that used to pay for it. Blocking is one response. It is not the only one, and on its own it is rarely the best one.

3. Why a platform-level block falls short

A default block delivered by your CDN has three structural limits, whichever vendor ships it:

• It only blocks. Blocking earns you nothing. It removes you from the AI answers your audience increasingly reads, and it leaves the value on the table instead of collecting it.
• It only works where that vendor sits. If your traffic runs through a different stack, the switch is not yours to flip. Your control should live with your site, not with whoever happens to sit in front of it.
• It hands the decision to a platform. An on-by-default, opt-out setting is coarse. You want per-bot, per-content rules that reflect your own strategy, and you do not want your AI policy to be one more thing you are locked into with a single supplier.

In other words: the move confirms the problem is real. It does not give most sites the control, the independence, or the revenue they actually need.

4. The fuller playbook

The strategy that pays is not block-everything. It is granular, and it treats AI demand as a revenue source rather than a threat to shut out:

1. See, in real time, every AI system fetching your content, and exactly which pages they take.
2. Keep classic search indexing so you protect your rankings.
3. For each AI bot, choose to allow, charge, or block, per content type.
4. Turn AI demand into revenue with pay-per-crawl and the x402 standard, settled in euros.

Blocking says no. Charging says yes on your terms, and gets you paid. For the case against blunt blocking, read bot traffic monetization, and for the payment layer that just became an industry standard, see the x402 Foundation launch.

5. Control and monetize, on any stack

Senthor is the AI-traffic control layer that lives with your site, not your CDN. It installs in minutes on WordPress, Vercel, Nginx, Caddy and more, and it does the two things a default block does not: it lets you charge, and it works wherever you are hosted.

There is a plainer advantage too: control and monetization should not be locked behind an enterprise contract. Senthor puts every feature in every plan, with a free tier to start and transparent, volume-based pricing, so a small publisher gets the same control as a large group. You should not have to buy the most expensive tier of a platform just to decide who crawls your own site.

6. FAQ