via TechCrunch AI
Cloudflare’s New Policy Forces AI Companies to Pay for Publisher Content by 2026
Cloudflare has set a firm deadline for the AI industry, requiring a clear separation between web crawlers used for traditional search (e.g., Google Search) and those employed for AI agent tasks or model training. Starting September 15, 2026, Cloudflare’s default settings will block “mixed-use” crawlers—those that combine search, agentic functions, and training—from accessing any pages that host ads. This policy applies to new Cloudflare customers, new sites set up by existing customers, and all existing free-tier users, unless site owners manually adjust their settings.
This move could significantly impact how AI model providers access web content for training and to power their agentic services. Cloudflare acknowledges that most website owners want their content discoverable via both search and AI services, but they also demand protection against the free use of their intellectual property. In its announcement, Cloudflare specifically highlighted the “world’s largest search engine” (clearly referencing Google), noting that it accesses “about 2x more information” than other AI companies because its search bot makes it difficult for customers to remain discoverable without also being used for AI purposes.
Google has pushed back against this characterization in the past, pointing to its dedicated bot, Google Extended, which allows site owners to opt out of having their content used for AI training and products like Gemini Apps and Vertex API. Google emphasizes that opting out does not affect a site’s inclusion in Google Search. However, Google’s flagship Googlebot still crawls for Search, including AI-powered features like AI Overviews and AI Mode.
“Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” said Cloudflare co-founder and CEO Matthew Prince. He was referring to a recent milestone where bots surpassed human traffic online for the first time—a shift originally not expected until 2027. Prince added: “Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities, and benefit AI companies that operate bots with clear and transparent intent. We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training.”
Beyond blocking mixed-use crawlers, Cloudflare has been developing a suite of tools to empower publishers in the AI era. In recent years, the company launched tools to combat AI bots and introduced a marketplace called Pay Per Crawl, which lets websites charge AI bots for scraping. Cloudflare has now announced that Pay Per Crawl will evolve into “Pay Per Use,” enabling publishers to charge AI companies not just when content is fetched, but when it creates actual value—such as when it powers a user-facing AI service or generates insights from training data.
These changes reflect a broader trend in 2026, where publishers and infrastructure providers are demanding fair compensation for AI’s use of copyrighted content, pushing the industry toward more transparent and equitable data-access practices.