Cloudflare, which currently helps power about 20% of all websites, unveiled a novel marketplace on Tuesday aimed at transforming the way website publishers and artificial intelligence companies interact—an initiative designed to give digital publishers more control over their content.
Over the past year, Cloudflare has focused on addressing the growing concerns of publishers dealing with an explosion in AI-driven web crawlers. Earlier solutions the company introduced included tools that allowed website owners to block these crawlers easily and dashboards to track AI bot activity on their sites. Cloudflare CEO Matthew Prince had previously hinted that these initial offerings were building blocks toward an innovative marketplace through which publishers could monetize their content more directly.
Now, Cloudflare has brought that idea to fruition through its experimental new marketplace dubbed “Pay per Crawl,” currently available in a private beta program. With this system, website operators can choose to charge individual AI bots a small fee—literally a micropayment—each time the bot crawls their site. Alternatively, publishers can allow certain AI bots free access, or block them entirely. To assist with these decisions, Cloudflare provides detailed insights revealing whether the bots crawling their pages are doing so for training AI models, gathering information for AI-driven search results, or another purpose altogether.
Scaling this type of marketplace potentially opens up a new revenue stream in the AI era—one that could place Cloudflare squarely in the middle of the transaction. The launch comes as online publishers grapple with uncertainty about their future role in a digital world increasingly dominated by AI-based search tools rather than traditional search engines. While some larger publishers are pursuing lawsuits against tech companies for unauthorized AI training on their content, others have opted instead to enter licensing deals meant to compensate them for such usage.
Nevertheless, these licensing arrangements have mostly been limited in reach and scale to well-known media brands, leaving a question as to whether smaller or independent publishers can leverage content monetization in a meaningful way. Cloudflare says their marketplace aims to bridge that gap by empowering publishers to set their own pricing structures and engage directly with AI companies in a more transparent, streamlined manner.
Alongside the announcement of Pay per Crawl, Cloudflare also said new websites joining its platform will automatically block AI crawlers by default. This new setting underscores Cloudflare’s stated goal of moving toward a “permission-based approach” to digital content crawling, providing websites with default control over their intellectual property. Already, several prominent publishers—including TIME, Condé Nast, The Atlantic, Fortune, ADWEEK, and The Associated Press—have embraced Cloudflare’s approach by opting to limit AI crawler access automatically.
Historically, digital publishers have benefited from Google’s crawling activity, accepting Google’s visits in exchange for traffic referrals that drove ad revenue. However, Cloudflare’s internal data now suggests this model may be eroding in the AI-centric web landscape. In June of this year, Cloudflare found that Google’s bots visited websites approximately 14 times for every user referral the search giant delivered. In contrast, OpenAI’s crawler scraped pages roughly 17,000 times for just a single referral, while Anthropic’s crawler scraped 73,000 times per referral—indicating a stark imbalance between content scraping and incoming traffic from AI sources.
Additionally, the emergence of fully automated AI agents—capable of independently scouring the web and delivering concise answers without sending users directly to publishers’ pages—only compounds publishers’ concerns. According to Cloudflare, this is precisely the scenario where Pay per Crawl could become particularly valuable, offering sites a way to effectively monetize crawls performed by AI agents seeking specific, valuable content directly for users.
Cloudflare imagines eventually creating automated “agentic” paywalls that operate programmatically without human oversight. For example, a user’s AI agent could independently pay publishers small amounts to retrieve the latest research or information, should users provide a budget directly to their preferred digital agent.
Currently, participation in this experimental marketplace requires both publishers and AI companies to maintain Cloudflare accounts, where they set their agreed-upon crawling rates. Transactions occur through Cloudflare, which manages payments and funnels proceeds to publishers. Cloudflare spokesperson Ripley Park clarified the project does not yet utilize cryptocurrencies or stablecoins, despite speculation from industry experts that digital currencies might ultimately make such micropayments easier.
Cloudflare’s marketplace vision carries potential far-reaching implications for online publishing and the broader information economy, though significant hurdles remain. For instance, encouraging AI providers accustomed to free access to embrace this paid model may prove challenging. Nonetheless, Cloudflare appears uniquely positioned to lead this potential shift, aiming to redefine publisher-AI relationships for the future of online content.
