Cloudflare Blocks AI Crawlers by Default and Launches Pay-per-Crawl Fees

62 posts • OpenAI (o3)

Updated Jul 3, 2025, 05:16 PM

Cloudflare said on 1 July it will now block artificial-intelligence web crawlers by default on every new domain that joins its network, reversing the long-standing practice of allowing automated scraping unless a publisher explicitly opted out. The San-Francisco-based company, whose infrastructure carries roughly a fifth of global web traffic, has also opened a private-beta marketplace called “Pay per Crawl” that lets site owners charge AI developers each time their bots request content.

The system revives the rarely used HTTP 402 “Payment Required” status code: when a verified bot encounters a blocked page, Cloudflare returns a price and, if the crawler agrees, automatically settles the fee while delivering the content. Publishers can allow, refuse or meter access and set different rates for model training, fine-tuning or real-time inference. Cloudflare will serve as the merchant of record, administering payments and authentication to deter spoofed user-agents.

Media groups ranging from the Associated Press and Time to Condé Nast, The Atlantic, Reddit and Stack Overflow have endorsed the change, arguing that AI firms siphon data without compensation and erode referral traffic. More than one million websites had already activated Cloudflare’s one-click AI-block tool introduced last year; the company’s move makes that stance the default. Chief Executive Officer Matthew Prince said the goal is to restore “a sustainable future for original work online” while still giving AI companies a paid path to high-quality data—potentially reshaping both internet economics and the cost base for training large language models.

More breaking stories on DeepNewz — updated live.

#Cloudflare #HTTP #Associated Press #Time #Condé Nast #The Atlantic #Reddit #Stack Overflow #Matthew Prince

Financial Analysis

Associated Financial Markets

More financial news and market updates→

Related Prediction Markets

OpenAI browser in July?

Will OpenAI acquire Chrome before before August?

Which company has best AI model on June 30?

•

Sources

blog.cloudflare.com(6)bigdatawire.com(3)technologyreview.com(3)decrypt.co(3)

Kumar

@datarade

Is there a way to figure out how many and which domains have @Cloudflare pay per crawl turned on?

Sally Shin

@sallyshin

Late to the news but did Cloudflare's new tool blocking ai crawlers kill a bunch of startups overnight? I've seen bunch of companies trying to be solutions for publishers trying to monetize/block AI scraping, none that could scale a marketplace product to the level of Cloudflare

Dylan Curious | AI News & Analysis

@dylan_curious

Bots Now Rule the Web Cloudflare data in Reuters’ AI-Intelligencer shows traffic tilting toward autonomous agents, draining human clicks from publishers. If chat agents do the browsing, what’s left of the open internet? For more AI News, follow @dylan_curious on YouTube.