AI publisher compensation framework announced by IAB Europe
The digital landscape has seen a significant shift in recent times, with AI platforms driving a staggering 770% growth in traffic to news and media sectors during the same period. This surge in AI-driven traffic has not gone unnoticed, leading to the publication of the "Crawling for Compensation" framework by IAB Europe in September 2025.
The framework, authored by IAB Europe Data Analyst, Dimitris Beis, aims to establish fair compensation for content used in AI training and inference. It consists of three core mechanisms: content access controls, discovery protocols, and monetization APIs.
The content access controls provide publishers with the ability to control how their content is accessed by AI platforms. The discovery protocols, on the other hand, ensure that AI platforms can efficiently and effectively discover and access the content they need. The monetization APIs, including Cost-per-crawl and LLM ingest content APIs, offer a means for publishers to monetize their content, with pricing based on factors such as content type, bot classification, and access frequency.
Referrals from AI platforms have also seen a substantial increase, with a 357% year-over-year rise in June 2025, resulting in 1.13 billion visits compared to 191 billion visits from organic Google search. However, this growth has not been without its challenges. Unauthorized scraping increased by 40% from Q3 to Q4 2024, with a significant decline in robots.txt compliance. To address this issue, the framework proposes blocking unauthorized scraping through robots.txt files and Web Application Firewall methods.
The framework's implementation emphasizes European collaboration, with IAB Europe's Artificial Intelligence Working Group working closely with European publishers. Notable AI platforms such as OpenAI, Google DeepMind, Microsoft Azure AI, and Anthropic have joined the IAB Tech Lab's Content Monetization Protocols working group, signalling a commitment to the framework's principles.
Adobe research suggests that AI-referred visitors stay longer on sites, view more pages, and have lower bounce rates. This indicates a potential for increased engagement and revenue generation for publishers. For instance, AI-referred visitors stayed 8% longer on sites, viewed 12% more pages, and showed 23% lower bounce rates.
Cloudflare expanded its 402 payment protocol for AI crawler communication in August 2025, following the launch of the IAB Tech Lab working group in August. This move is expected to streamline communication between AI platforms and publishers, further facilitating the implementation of the "Crawling for Compensation" framework.
The "Crawling for Compensation" framework addresses the declining publisher revenues resulting from increased AI content scraping and zero-click searches. By establishing technical specifications for AI platforms accessing publisher content and providing a means for fair compensation, the framework seeks to create a more sustainable and equitable digital ecosystem for all stakeholders.
Read also:
- Industrial robots in China are being installed at a faster rate than in both the United States and the European Union, as the global market for these robots faces a downturn.
- Hyundai N affirms transition to hybrid performance-centric models, initiating with Tucson N
- EAFO Research Uncovers Crucial Elements in Electric Vehicle Adoption within the EU
- Stock markets in India anticipated a moderate opening, influenced by mixed signals from global markets.
