AI Giants Back New Opt-Out Standard for Web Scraping

Overview

The biggest names in AI are backing a new mechanism that lets publishers control whether their content can be scraped for AI training. The new control enables publishers to apply their AI training preferences across multiple crawlers at once.

Key Details

The opt-out standard, announced by a coalition of major AI companies and web stakeholders, introduces a unified signal that publishers can attach to their sites. Rather than configuring exclusions separately for each AI company's crawler, publishers can now broadcast a single, standardized preference that participating crawlers are expected to respect.

Why It Matters

Simplified Compliance for Publishers

Until now, opting out of AI scraping required publishers to maintain separate rules for each crawler — an increasingly impractical burden as the number of AI models and data collectors has grown. The new control centralizes that process, reducing administrative overhead and lowering the risk of misconfiguration.

Industry-Wide Support

Backing from the largest AI companies signals a shift toward more transparent data-governance practices. By adopting a common opt-out standard, these companies aim to demonstrate good-faith engagement with publishers while continuing to build and train their models.

A 2026 Context

As AI training datasets have expanded and regulatory scrutiny has intensified in the EU, the UK, and the US, 2026 has seen growing pressure for interoperable content controls. The new opt-out standard arrives amid ongoing debates over copyright, fair use, and compensation for content creators, and it is expected to be referenced in emerging compliance frameworks.

What's Next

Participating companies are expected to publish implementation guidelines and update their crawler behavior in the coming months. Publishers will be able to apply the signal via standard web protocols, and industry groups plan to monitor adoption and enforcement. Questions remain about how the standard will be verified and what recourse publishers will have if crawlers ignore it.

The Bottom Line

The new opt-out control marks a step toward a shared vocabulary for AI training preferences — one that could reduce friction between publishers and AI developers, even as broader debates over content rights continue.

via The Verge AI