Matthew Prince has a vision of internet utopia.
In it, anyone can read anything online for free, just like how it was in the early dotcom days. Content creators would be flush with cash thanks to strong business models, like in the pre-dotcom days, and audiences would be satisfied with the quality of content they consumed.
But the cash keeping this system running wouldn’t come from readers’ pockets, or even entirely from advertisers. In Prince’s utopia, there are no paywalls or intrusive belly fat ads required to keep media companies afloat.
Instead, the money would come from charging an army of bots to access content on their websites—something the bots, dispatched by AI companies, currently do for free as they crawl the web and “scrape” data to train their insatiable large language models (LLMs).
Prince has a unique vantage point. As the cofounder of Cloudflare, a cybersecurity company that he says more than 20% of the entire internet sits behind, he’s seen the number of crawls on sites drastically increase.
Ten years ago, when Cloudflare was an early-stage startup, Prince says Google’s bots would crawl a publisher’s pages twice for every person—or unique visitor—that Google’s search engine referred back to that site. It was, for the most part, a mutually beneficial arrangement: Those referred users resulted in website traffic, which publishers could then monetize through subscriptions or advertisements.
Today, Prince says Google crawls websites six times for every referred user and for upstart AI companies, the crawling is significantly worse: OpenAI’s crawl-to-referral ratio is 250:1, and Anthropic’s is a whopping 6,000:1.
(Cloudflare says it works with 80% of the top AI startups and its data team calculated these numbers by comparing the number of unique page requests referred by each company versus the number of requests made from their AI crawlers. It’s also worth noting that Anthropic’s product doesn’t include a web search function and is not designed to refer users to publisher sites to the same extent that OpenAI does.)
“For these new AI systems, the value of, ‘I’m going to take your data and then in exchange I’m going to send traffic back to your site’—that’s just going to break,” says Prince. “And so we have to invent some other model.”
Prince’s motivation to save the media industry from AI demise isn’t derived purely from a sense of moral obligation, although he is technically one of those tech-billionaires-turned-media-owners like the Washington Post’s Jeff Bezos and L.A. Times’ Patrick Soon-Shiong. In 2018, Prince purchased his local paper, the Park Record, which he is currently trying to turn into a nonprofit.
In Prince’s view, the collapse of content creation is an existential threat to the internet itself. If there’s no business model to support it, then no one will be incentivized to make it. Without fresh content, there is nothing for the AI models to continuously train on, and eventually, no internet at all.
And no internet means no Cloudflare, which is Prince’s primary business.
The advent of AI has seemed like the nail in the coffin for publishers, who have experienced a slow, painful decline in their businesses since the 1990s as advertising dollars and readers got siphoned off to large tech companies like Google, Amazon, and Meta instead.
Lawsuits over content usage rights, like the New York Times vs. OpenAI, have been filed to protect original work from being scraped without permission or compensation. Other publications, like Dotdash Meredith, have struck multimillion-dollar, one-time deals with AI companies to allow LLMs to train on their work.
Bill Gross, who helped create the pay-per-click online search advertising model, founded ProRata.ai, a startup that wants to compensate publishers based on how heavily referenced their pages are in AI search results (disclosure: Coins2Day has a business partnership with ProRata and Perplexity.ai). He likens the approach to Spotify, which pays artists per stream versus on a per song or per album basis, although not all artists find that model’s revenue split satisfactory.
Prince has considered all of these options and thinks none of them totally suffice. But he thinks there is an easy solution that he’s best positioned to execute.
“Senior media executives keep saying to me, ‘There’s nothing we can do! [AI companies] take all of our data, and there’s no way to fight back except with lawsuits,” Prince told me at the World Economic Forum in Switzerland last month. “My view is, Cloudflare battles the Chinese government every day. This is just a couple of nerds in Palo Alto. This is no big deal.”
Prince’s plan to save content creators has three parts, which he says he’s already begun to roll out.
First, publishers need to see precisely how often their sites are being scraped, and by whom. Cloudflare’s clients were given this transparency last fall when the company announced AI Audit, a suite of tools sites can use to control how their content is used by AI models.
Once site-owners have that information, they can figure out which—if any—of the crawlers they want to enable access to. AI Audit blocks the crawlers at a network level through a web application firewall rule, making it more impenetrable than other solutions, which require sites to stick in snippets of robots.txt code into their backends and hope bots will obey.
Right now, AI Audit is an opt-in experience for Cloudflare users. Within the first half of 2025, Prince says that will change to become opt-out, at which point all AI crawlers to the sites Cloudflare protects will become blocked by default, like malicious traffic.
Blocking every AI company’s bots at the network-level is key, Prince believes.
“When you talk to Sam Altman of OpenAI, he’ll say, ‘Of course we need to pay content creators,’” says Prince, who says he has talked to OpenAI executives directly about the problem. But, he says, they want it to be an even playing field. “If you charge OpenAI, but you don’t charge Perplexity, at some point, OpenAI will say, ‘Well, that’s not fair.’” The solution is to create scarcity across the board, Prince says.
Once original content becomes scarce, the power then flips to the content creators, who can decide how much to charge AI companies for access to their goods. That’s the final phase of Prince’s plan, which he doesn’t have fully formulated: creating a marketplace for original content with a high rate card.
“It would be great if we got to a web that was back to: Humans get content for free, and bots pay a lot for that content,” Prince says.
It’s a nice-sounding utopia. But a lot of Prince’s plan hinges on best-case scenarios and current assumptions, all of which can be dangerous in an industry that moves as fast as AI. Content creators also aren’t waiting for a knight in shining armor; many are actively determined to shape the AI future instead of being pummeled by it. Still, any creative thinking that helps solve one of the biggest challenges to the production of high-quality content—and that creates a new framework for who pays for it—is a refreshing thought exercise.
“Maybe we can actually unlock a lot more really valuable content creation and re-enable a business model for journalists and academics and others to thrive and survive,” Prince says. “If we can be a little part of that, that would be pretty cool.”
