Publishers' Last Stand

I’m Alan Chapell. I’ve been working at the intersection of privacy, competition, advertising, and music for decades, and I’m now a regulatory analyst writing for The Monopoly Report.

The latest Monopoly Report podcast!

 This week, we feature part 2 of my discussion with former FTC Chair Jon Leibowitz (part 1 is here). Jon defends Judge Mehta’s recent remedies decision in the Google search antitrust case and provides the origin story behind Google’s long-standing policy on device fingerprinting.

When all else fails, put out a scarecrow

Last week, Ari Paparo predicted that Google won the ad tech remedies trial by bringing in Elizabeth Douglas, the CEO of wikiHow as a witness. I’ll admit: Bringing in the perspective of a smaller publisher was an effective counterbalance to the one offered by the News Corps of the world.

And while I still think Judge Brinkema is likely to require a divesture of AdX, I’m about as confident in that prediction as Charlie Brown heading into Thanksgiving.

Whether Google’s strategy here was effective in convincing Judge Brinkema that long-tail publishers’ existences come to an end without Google’s tender caress is debatable. I find it ironic that whatever hassle an AdX divestment may cause for wikiHow will likely pale in comparison to the devastation being caused by AI Overviews.

But that also demonstrates one of the more frustrating limits to antitrust law: Whatever issues raised by the loss of traffic are largely irrelevant in the ad tech trial. If anything, those issues were already adjudicated by Judge Mehta in the search remedies trial.

So when I get asked about the impact of Google’s recent antitrust trials, my answer is to look beyond what’s taking place in the courtrooms.

What’s one less remedy amongst friends?

Here’s the dirty little secret — and one of the reasons that publishers (hell, most content creators) are so sullen these days. While even a child can probably see that what’s happening to content creators is flat-out wrong on many levels, the law isn’t providing much in the way of solutions.

• Copyright? Meh, we need to win the AI war against China.

• Antitrust? Kanter and Kahn left the building, and the rules remain stacked in favor of the biggest players.

• Non-U.S. competition law? The President is being mean. And anyway, Europeans are stuck with U.S. tech.

• Congress to pass new laws? Wait — are those guys still around?

• Section 230 or Privacy Law? LOL.

There’s just not much to be excited about if you’re a content creator. The closest thing to an optimistic observation I’ve heard recently was that extinction events normally take much longer than the conventional wisdom tends to believe.

Cloudflare steps into the void

The internet was built on a handshake deal between publishers and search engines. I won’t go through all of that background and history here, but I suggest you read Search Engine Land’s take. What’s important here is that incentives have changed — and that big tech is exploiting the opacity of a system that might as well have been designed by Native Americans in 1650.

If you’re reading this, you’re probably aware of Cloudflare, a CDN that powers internet requests for millions of websites and serves 78 million HTTP requests per second on average. Cloudflare manages approximately 20% of internet traffic, so that the company’s policies will automatically apply to millions of domains. This gives the company a fair amount of leverage to help publishers reshape how AI models access and use web content.

Ricky Sutton and I recently chatted with William Allen, Cloudflare’s Chief Product Officer, to discuss what Cloudflare is attempting — specifically, its Content Signals Policy and pay-per-crawl program. You can find that discussion here.

Content Signals and pay per crawl

The Content Signals Policy is designed to transform the traditional robots.txt file from a polite request into what the company describes as closer to a contractual obligation. When a website owner specifies that AI crawlers should not access their content, Cloudflare enables the pub to potentially treat those violations as a breach of contract rather than mere technical inconvenience.

Think of it as a scarecrow — or a “get off my lawn” sign.

The pay-per-crawl system takes this concept further by introducing direct monetization. Publishers will be able to set specific fees for AI companies that want to access their content for training purposes. The idea is to create a marketplace where data has explicit value rather than being treated as a free resource ripe for harvesting.

AI Labyrinth

Cloudflare has also deployed what it calls "AI Labyrinth" — a honeypot system that feeds unauthorized crawlers convincing but fabricated content. Instead of blocking suspicious bots outright, the system wastes their computational resources on AI-generated pages that appear legitimate but contain no actual value. This approach mirrors classic cybersecurity tactics, turning the tables on scrapers by making unauthorized access costly and inefficient.

But the question remains: Will these technical measures prove to be an effective scarecrow that deters unauthorized scraping from AI bots? Or will they ultimately carry all the weight of a do-not-track signal?

The power imbalance between pubs and AI companies

For publishers, these tools offer something they've desperately needed: agency. The current system essentially forces content creators to choose between allowing unrestricted AI access or risking reduced search visibility. Many publishers have watched their content fuel AI models that compete directly with their traffic and revenue streams, all while bearing the computational costs of serving bot requests.

The DOJ proposed addressing this issue as part of its antitrust remedies against Google, potentially requiring the company to separate its search indexing from AI training activities. Such a separation could have given Cloudflare's tools much more practical utility, allowing publishers to block AI training while maintaining search visibility.

Without that separation, Cloudflare’s tools seem to fall into the “well, it’s a start” category.

How will all of this work (or will it)?

The most optimistic outcome for pubs is the continued shift toward more explicit content licensing arrangements, with AI companies building direct relationships with major publishers while smaller sites rely on platform-level protections like those provided by Cloudflare. And maybe… just maybe, a market develops for smaller pubs.

The thing that intrigues me about these tools is that they make it more difficult for AI platforms to pretend that publishers are OK with having their content scraped without compensation. They might even provide a formal legal mechanism for publishers seeking to assert those rights via litigation.

And that might be the true legacy of the Google antitrust cases — events that helped pubs realize that the law ain’t coming to save them.

In that way, the trials and Cloudflare’s tools serve as a rallying call toward collective action.

_________________________________________________________________________

If there’s an area that you want to see covered on these pages, if you agree or disagree with something I’ve written, if you want to tell me you dig my music, or if you just want to yell at me, please reach out to me on LinkedIn or in the comments below.