CEOInterviews.AI
Start App
Get an email when - John gives a new interview
One email per new interview. Unsubscribe in one click.
- John
Chief Technology Officer, Cloudflare

Building an open agentic Internet: Discoverable, callable, and payable (Cloudflare Connect 2026)

📅 Oct 25, 2026 Cloudflare Developers 33 MIN 3704 VIEWS 39 SEGMENTS · 3 SPEAKERS
Agents are a new kind of visitor. They don’t render your CSS or click your ads, but there’s a paying human on the other end of every request — block them and you block your customer. Today, billions of those requests are bots re-fetching pages that haven’t changed, which is machine effort with no outcome for either side. This talk lays out the four primitives of a web reshaped around this visitor — readable, discoverable, callable, and payable — plus the identity layer underneath and why they have to be built on open standards. How agents read, discover, call, and pay will decide whether the I...

What - John said

Written from the verified transcript and checked against it. Every figure links to the moment it was said.

John, Cloudflare's CTO, argued that the majority of internet traffic is now automated, driven by AI training and agentic use cases, and that this is disrupting the business model of content producers, especially news sites. He cited Radar data showing 97% less human traffic for some sites and a poor referral ratio for Perplexity. To address this, Cloudflare is building tools to make crawling more efficient, including a /crawl endpoint that serves markdown to verified bots, an open search index for sites behind Cloudflare's proxy, and a partnership with OpenAI to discover new domains. He also discussed monetization: a pay-per-use system for content reuse and a monetization gateway in private beta for API access payments, built with partners like Coinbase. He acknowledged challenges like content leakage and the need for bot detection, and emphasized that website owners control their own choices.

Key takeaways

  1. Cloudflare is building an open search index for the ~20% of the internet behind its proxy, with opt-in/opt-out controls for site owners.
  2. A /crawl endpoint serves markdown to verified bots, reducing load on origin servers and is part of a push for a Content Crawl Control standard.
  3. Cloudflare partnered with OpenAI to help discover new domains, resulting in several million new domains being found without heavy crawling.
  4. The monetization gateway, in private beta, allows site owners to set prices for API access, with payments handled via a protocol involving Coinbase.
  5. Pay-per-use, still under development, aims to compensate content creators for reuse of crawled content, using a bidding system similar to Google AdWords.

Numbers and commitments

FigureWhat it refers toTypeAt
13% Share of traffic from traditional search engines metric 2:18
97% Reduction in human traffic for some sites metric 4:00
3,400 Agent visits per human visit for Perplexity metric 4:44
20% Share of the internet behind Cloudflare's proxy metric 12:06
10 cents Example price for API endpoint access price 15:34
1 cent Example price per API call price 17:26

Chapters

  1. 0:00Rise of automated traffic
  2. 3:09Impact on content producers
  3. 7:28New business models needed
  4. 10:19Making crawling easier
  5. 12:06Open search index
  6. 13:13Domain discovery with OpenAI
  7. 15:34Pay-per-use monetization
  8. 17:26Monetization gateway
  9. 20:56Q&A on pricing and content control

Questions asked in this interview

7
  1. 20:56On the monetization side, um, are you guys thinking about like bid-ask spread?
  2. 22:21So, since you're targeting content providers, are you looking at ways where some content provider, let's say a New York Times, right?
  3. 24:22Because isn't that you're adding hurdles or gateways for free indexing content?
  4. 25:41So, do website publishers need to structure their website content or data in any special way, whether it be meta tagging or markdowns or hierarchy, to maximize their monetization opportunities?
  5. 26:56How do you think about those two universes kind of working together and not creating different functions?
  6. 28:38Like if anything takes off basically, right?
  7. 30:09Yeah, on the crawler, is it kind of pushing out, 'Oh, hey, something just changed,' or is it very passive still?
John 0:03 ↗
Let's get started. Welcome everyone, thank you for coming to my talk today. Um, anyone here was patching PHP applications at 6:00 a.m. this morning? Just the two of us. There you go, a few more. Uh, so I'm a little tired, but we'll get through it. Uh, it's not going to take the full 45 minutes. Topic for today is a big one. If you were at the keynote yesterday, there's actually a bit of preview. This is a bit of a longer form version of that, and actually some tactical examples of what we're doing to build, of course, an open agentic internet that helps everyone — content producers, website owners, as well as, you know, the frontier model providers. Uh, which helped me a lot this morning, 'cause Cloudflare did patch a bunch of my servers.
Right. So, the premise to the whole situation. Um, you can actually check this up on radar.cloudflare.com. If you haven't seen the website, I highly advise you take a look. There's a whole bunch of really interesting filters that you can filter by region, activity, bot type, uh, purpose of crawling. The headline, of course, is the majority of the traffic we're seeing across the Cloudflare proxy — give or take, we're talking about tens of millions of websites at this point — is mostly non-human, or better, bot and automated. And we also know that the increase over the past, you know, 12 to 18 months has mostly been driven by all the new use cases for crawling, training being the prime examples. Everyone is building a bot, everyone is fetching content via their agents, and this is the output of that consequence.
It was mentioned at the keynote, um, the order magnitude difference over the next couple of years is just going to grow. So, take an example: you're running a small little website, you may be receiving a couple thousand requests per minute today. You can probably expect that over the next years that number is going to go way higher, and it's all going to be automated, 'cause everyone is hungry for data. We zoom in. Again, this is a screenshot from radar.com. And just as I said earlier, the number one use case — and we do this by actually identifying, where possible, what bot is accessing the content behind the proxy. Uh, you may have heard of Web Bot Off.
Another really cool thing on Radar: you can go and see our bot directory, and you have a list of hundreds of bots we recognize. And a bunch of them, especially the big provider ones, submit a cryptographically verified HTTP header, so we can say, 'Yes, this is OpenAI. This is Anthropic. This is Google. This is Muse from Facebook,' and similar. And based on the bot — um, most of the providers split their bots based on use case — we can tell that, you know, number one case is training or some sort of agentic behavior to power our conversations with our new bot overlords. Um, search is still there, uh, not negligible, about 13% of traffic. That is your traditional search that you might see in Google, Bing, Yandex, you name it. Um, pretty constant, uh, but definitely not more than the agentic use cases.
What does this mean? So, we started a couple — well, a year and a half ago, we had Content Independence Day when we started noticing the behavior changing on internet traffic. And the canary of the whole situation of the business model is changing. Of course, it's websites that produce original, novel content. Okay, I actually run, as a side, one of the little news agencies for my own town, local town, my hometown in Italy. And I've actually seen this myself on the server, looking at the logs. Um, human traffic is dropping a lot, considerably. And I run the website on behalf of a friend who's actually the owner of the website. And uh, just like these big brands here on the slide, his revenue from ads is just going down. His revenue from subscriptions is going down. Why?
'Cause today you open your OpenAI ChatGPT, 'what's the latest news?' Even this morning I was looking for the latest — when was the patch out for PHP? I could have probably checked php.net, given it was the front page article on the site, but I actually asked Claude, and Claude fetched the page for me and gave me the information I needed. Did I, as a human, go to php.net? No, right? And you can see how this is changing behavior for everyone, and those who make money from human visits are suffering the most. Uh, pretty impressive digital trends. This is actually published by a third-party organization that looks at this data, so not even Cloudflare data. 97% less human traffic hitting the site. This is just complete disruption of their business model.
Okay, so some would argue, you know, historically you get a crawler accessing your site, getting your site indexed, that's good. A bunch of us probably have experience with search engine optimization. Is this bringing any value? We can actually see, looking at referer headers — and again, this is public, go to radar.cloudflare.com — how much are these providers crawling, and how many humans are they sending back to their site? Sometimes you actually may see, you know, the chat agent is giving you a referral link, and you may click on it and actually visit the site, consume the ads, buy a subscription or something. But it's pretty bleak, let's be honest. Look at Perplexity: for every 3,400 visits their agent accesses any given site, they're sending one human visit back, right? And it doesn't get better throughout the board, and this means more resources, more load, more servers that you need to stand up to keep your sites online.
Um, we track this over time. If you click on Radar, you can actually see the difference over the course of the past couple of months. And these, of course, are just the top ones that we picked, hand-picked. If you go to the bot directory, you can actually see on any given bot that you might be interested in many more providers than the ones listed on the slide. So, the big question is: what do we do now, as an internet community, or people that benefit and enjoy exploring and using the internet as part of our daily lives? We need to essentially make sure that the business model stays up so that the internet doesn't devolve into something that's worse — let's say worse over time.
Um, as I said earlier, the canaries here, of course, are the news publication websites. They're suffering the most, but you can see how this trickles through, for example, companies that are publishing research, journals, news, weather companies. And eventually it will probably hit everyone, including e-commerce installations, small blog providers. I have a personal blog that I keep updated, and I'm probably going to see the consequence of that. Not that I get many visitors in the first place, but it's only going to get worse. Um, the other thing, some of you may not know this: some of the browsers are actually embedding, or some of the front-end model providers have their own AI-native browser. Someone who heard about Comet, for example, as one example. Um, when you're accessing web pages using those browsers, sometimes they're actually doing crawls in the back end, 'cause they come across as, of course, legitimate humans, and it's harder for, you know, bot detection providers such as Cloudflare to actually detect them and stop them. And, you know, as a consequence, actually making it worse for everyone.
Good news, you know, new business models. This is just the way history evolves. Companies and industries have been disrupted before. So, for us, it's just a matter of figuring out what is the next. Couple of, you know, very well-known examples here: the Napsters of the world. Anyone here used to use Napster? Yeah? They've disappeared, replaced by Spotify with a subscription model. We can debate forever if that's good for the artist or not. Some will say yes, some will say no. There's been some very interesting, you know, scams and similar going on on some of these subscription-based platforms. I still remember as a kid going to Blockbuster, the Italian version of it, long gone, replaced by Netflix. And of course, I did see someone actually with a paper-based newspaper in the tube in London the other day. But fair to say, that's the minority, not the majority. Everyone's reading news online.
Okay. Um, new business model. We're starting to see it take shape. Um, some of the large content producers, for example, have already been able to — and this actually started earlier in the year and into even the late end of last year — signing direct deals with the big frontier model providers such as OpenAI and Tropic and similar. The good news is they actually want to sign these deals with the content producers, 'cause there's a benefit for both parties, right? If I sign a deal with a big NewsCorp and I'm a frontier model provider, I can probably optimize how I receive the data from them, so that I don't have to crawl a thousand times or continuously refresh and fight against them to try and bypass their mitigations, so that I can provide a better service to my end users.
This is the start. I would say at this point the majority of the large news publishers, even if they haven't maybe publicly declared or not, like People Inc., have actually now got in place deals with the frontier model providers for them to access content, especially because freshness of content is extremely important. Right? You want to know what's going on in your local town, you ask the chat agent, it should be able to give you really up-to-date information. It's also important that they get up-to-date information from the agents because we as humans are starting to rely on them for our everyday tasks. And if the agent doesn't have fresh information, we've all heard of hallucinations, right? We may actually get misguided, misaligned, get the wrong content from the content producers. So actually it's good for everyone if both parties work and find a good way of working together. This is just an example of, of course, Google AI Overviews maybe putting people at risk on health advice because sometimes they have just out-of-date or no access to the latest best content with the correct information.
So, at Cloudflare, you've heard this at the keynote, we're trying our best to be the impartial party, figure out how can we help both content producers, website owners, or frontier model providers. Of course, at some point we talk about money, so monetization is arguably one of the end goals, but we cannot just do that, right? We need to make it easy for both sides to achieve their end goal. So these are a few tactical examples of things we've been working on. Number one: making it easier to crawl. Now, arguably lots of people here do not want to be crawled, but fair to say some of us do want to be crawled. And if you have to crawl me, I want to make sure you're as efficient as possible while doing that, so you're not taking my website down or your crawling doesn't look like a DDoS attack against my infrastructure. I've actually had a couple of sites go down because I was getting hit quite heavily by Meta Inc. crawlers a while back, and I had to block them even if I was trying to give them product catalogs and similar.
If you go to the Cloudflare dashboard and you have a website on Cloudflare — who here has a website on Cloudflare? Okay, quite a few of you. If you want to give access, there's a little crawl option. The proxy will automatically issue a new endpoint, which will only be accessible, by the way, by a signed verified bot. So, we're trying to keep things as clean as possible. And we will parse the content out of your site and actually provide a much easier-to-crawl markdown syntax to the bot, so they can access the content over and over again if they so wish to do so, without hitting your backend server or increasing load to your application. Okay? So, this is one of the ways, um, we're trying to make it better for both parties.
Second one: make it easier to search. Um, you may have noticed if you do a prompt on a frontier model provider, on Claude Opus or whatever, very often the first thing the agent will do is kick off a bunch of sub-agents. They will actually go off and perform searches on the open internet. And this is one of the initiatives, but fair to say, still very much under development, um, with a plan of building an open index for all the sites sitting behind the Cloudflare proxy — give or take, we're talking about 20% of the internet — so that the frontier model providers or whoever, if you want to build one tomorrow, again it's about having diversity in the ecosystem, can actually refer to the index and actually get content without having to search for it, uh, which is a lot more resource intensive. Even this, by the way, is not opt-in. By default, you can go to the dashboard, you can decide to turn it on and opt in to the index or opt out. It's your content, it's your application, under your control. Plan is make it easier, so you don't receive the load, and of course it's a safe path to AI adoption over the long term.
Last one, and not least — um, search is part of it, and the index definitely helps. Um, whilst we are building out this index, of course, the search engines and the crawlers have a discovery problem. I remember I used to work at a company called Netcraft before I joined Cloudflare. One of the things we were doing was an internet service survey, and discovering new domains and sites for the purpose of surveying the internet was one of our biggest challenges. And we actually used to partner with the likes of Microsoft to get a list of domain names that were signed up by registries. Um, similar sort of incentive: we actually did a research, and you can find it on our Cloudflare blog. We partnered with OpenAI in this case. We give them signals of new domains that are being added behind the Cloudflare proxy with the crawl endpoint turned on, so opting into being discovered. We did it on a very small percentage of traffic, so not the entire application set behind the Cloudflare network. And you may have heard at the keynote, this resulted in OpenAI, in this case, discovering several million new domains and applications without having to necessarily bombard all of us with their crawls. Um, we think this is the win, especially for the smaller players on the internet. So, let's say you're starting up a new site tomorrow, you will be no one on the landscape, but with something like this, the providers will be able to find you more easily, and you actually show up in their search results or within the agent. Um, so you can get access to your end users.
Okay? There's a lot more we're doing. These are, I think, three of the best concrete examples that we're actively working on. Um, so homework: go in the Cloudflare dashboard, find the options. You can make the decisions to access or not. One last thing about the /crawl endpoint, which actually powers some of the research we did with OpenAI: there's actually a standard called Content Crawl Control, which we only let bots who are at least — it's a trust-based system — wanting to access your content for a given purpose to actually give them that content, be it training, inference, or for example, requiring referral back to your site or not. And all of these options are for you to configure and decide. We're actually pushing for a standard to be done by the industry, but still some work to be done there. But over time we're confident we'll make it happen.
Okay, so frontier models, crawlers, building a new one yourselves, you're a content producer trying to connect the dots for everyone. Now we can start thinking about monetization. These are the last two things of my presentation today. Number one: pay per use. The little video as a background. Um, you as a content provider, a content producer, want to get, you know, some part of the pie when it comes to making money at the end of the day. There's two big challenges here. Number one, you could actually get paid every time a crawler tries to access your content. And you may have heard of our monetization gateway, which is my next slide. Everything is good and great. You can set a price, you can say, 'If you access this API endpoint, I want to be paid 10 cents.' And there's a cryptographic way to make it happen.
However, there's a bigger problem, which is what we call pay per use, which is still under development, and we're figuring out how to make it scale up, which is a frontier model provider only needs to crawl you once, and they may reuse that content 100,000 times. Right? So you need to think about how do we make sure that every time the content provider uses your content, you get a little portion of that value back in your site. Okay, so if you're wondering what's the difference between pay per use and monetization gateway, this is the crux of it: is reuse out of scope versus accessing your content. Um, pay per use is based on a trust system for now. So it needs to be signed agents accessing your content. You set the price in the dashboard. OpenAI comes along. We take a look. We say, 'Yes, this is a price we're willing to pay. No, we're not.' And then they will add the content to their search engine, to their agent on frontier model providers, and in theory every time they use the content, you receive a little value back. Everyone following so far?
Second part is... it goes to the next slide. Seem to be stuck. Could you help me at the back advancing the slide? Monetization gateway. And the video — there you go. Second part is monetization gateway. This is in private beta. If you go to the Cloudflare dashboard, there's a little — on the left at the account level, there's monetization. Click on that. One click to request access to the private beta. This is essentially a technology built on top of the proxy, so that you can then define at access time how much you want to be paid. Easier from a technological standpoint, 'cause we're not relying on the trust-based system. You define the prices. Um, a signed agent comes along. Of course, we'll receive a 402 payment required response from the server. And assuming it's a signed agent and they've implemented integration with our protocol, they will submit the payment to the relevant payment provider and then re-access the content with proof that they have paid. This is the token. Proof happens, and then they access the content that you're providing. And in this case, it can be an API, right? So, if you expect — you've got a service and you're not building necessarily content, but it's an API service, you may want to get paid 1 cent for every API call. Monetization gateway, which by the way is an open standard, um, is the tool for you to achieve that, so you can scale out payments to agents without having to worry about collecting money from each one. Probably very useful for the smaller players that cannot go off and have big signed agreements with the frontier model providers.
Um, take a look in the dashboard. So, sign up for the private beta. You can actually set up rules on all sorts of filters you wish. You can decide, for example, to only make portions of your API, uh, requiring payment versus, um, different filters per region. You control the rules. We implement them as part of the proxy. Under the hood, um, who here has heard of the XYO Foundation? Couple of you, not too many. Um, we actually partnered with Coinbase and a couple of other technologies to come up with a standard and the protocol that makes it happen transparently. I'm not going to go into the details of this right now. Um, again, we are one of the providers that implement the standard. Um, there are others out there. Of course, we're the best ones that made it happen. Um, up to you, you choose. You can even implement it actually on your own servers without a proxy if you wish to do so. Um, everything is cryptographically verified. The only challenge here for us has been over time is how do we scale up this concept of micropayments? You may also have heard that we issued and are building out a stablecoin called Net Dollar, which is part of one piece of the puzzle to hopefully make this happen as it scales up to everyone using the internet at large.
Um, and yes, and hopefully, uh, maybe by the time we meet again next year at a similar conference, we start seeing the new business model of internet taking shape. With that, I promised it was going to be sweet and short. So, thank you for listening to me today, and uh, yes, any questions from the group? I'm happy to answer them. Ben? I think we have a mic so everyone hears you.
Ben 20:56 ↗
On the monetization side, um, are you guys thinking about like bid-ask spread? Like how — 'cause I think most people have no idea how to price.
John 21:07 ↗
Good question. So, think about the two use cases again: subsequent usage of content that was crawled versus payment at access time. When you go into the monetization gateway, which is again fully built, you just need to access the private beta. Once the team approves, you get access to the dashboard. You just define the price for the endpoints. And that's the price you set. You can figure out what you want. And um, the bot can decide if to pay or not. For the pay per use, which is something we're still not having fully rolled out to everyone, which we're trialing with some of the big content providers, that is more of a bidding system. Um, 'cause it's a bit more complicated to give value to something that gets reused over time, especially if it's not at crawl time. But that does have a bidding system where the providers can say, 'I want to pay you X.' And then you can decide the rules of 'I accept' or 'this is the range I'm willing to go.' And we try to optimize, a little bit like Google AdWords, you can imagine, right? To get you the best value out of your content usage. Um, having said that, this is well under development, so I'm sure there's going to be lots of subtleties that need to get worked out still. Down there.
Audience Member 22:21 ↗
So, since you're targeting content providers, are you looking at ways where some content provider, let's say a New York Times, right? I can allow OpenAI to crawl to get the headlines, but if the user then says, 'Okay, now I want to read the full article,' then I go in, click, and then I do a pay per use.
John 22:44 ↗
That's a very good question. So, you could easily build that yourself today by using monetization gateway. Um, let's say you wanted to get paid even for the headline, 'cause sometimes, you know, lots of readers just read the headline and they're happy and they move on. Um, I've actually done something similar on one of my sites. You essentially would have an API endpoint that responds to the signed agents with a summary of each article, and then you can apply a monetization gateway on top of that. Um, and as part of that summary, you have the referral back to your content. And then if a user clicks more, then they would subscribe through your standard process. The challenge with that, by the way, becomes then just generic bot mitigation. 'Cause what we're seeing — okay, we're doing all the right things, and the big providers are actually willing and wanting to make this new business model work out. There's a very long tail of other providers or potential crawlers that are trying to bypass all of these mechanisms, to the point we've seen sophisticated agents even sign up to subscriptions, crawl all of the content, and then redistribute it, right? There's actually services out there, legitimate businesses in many cases, that are doing that on behalf of, you know, content providers. Um, so the complexity becomes in locking down your content in a way that you're happy it's not leaking, right? Um, one of the side effects of this whole new business model has been the Wayback Machine, for example, right? 'Cause they're essentially a very legitimate, good reason to exist, but now they're getting abused 'cause they're just echoing content that you may not want to get leaked, right? Any other — yeah, question on that?
Audience Member 24:22 ↗
Hi. Um, very interesting models, and just curious, how do you balance the monetization versus discoverability? Because isn't that you're adding hurdles or gateways for free indexing content?
John 24:38 ↗
So, our current position is you, as the website owner or content provider, make the decision. Okay? It's near impossible for us to make a judgment on your behalf. We do not want to make the judgment. Uh, we're providing — or we think we can help everyone solve their problem by providing the tools. And then it's going to be each individual or business that makes the decision: 'I want to block bots and agents' versus 'I want to allow them for free.' That's something that many companies and businesses will want to do, versus 'I want to monetize.' It gets complicated if you start thinking about it, because I agree with you, it's a hard problem. A smaller news agency may want to give access to the agents for free, 'cause ultimately they're smaller and they're trying to grow. Maybe that's a good thing. Whilst a big company that has big bills to pay wants to start monetizing from day one, and it creates very odd, you know, balances. But it's your decision at the end of the day, and we'll figure out how to help you the best.
Audience Member 25:41 ↗
Hi, how are you? So, do website publishers need to structure their website content or data in any special way, whether it be meta tagging or markdowns or hierarchy, to maximize their monetization opportunities?

9 more exchanges in this transcript

Sign in free to read the rest of this interview. No card required.

Sign in to read the full transcript

Cite this transcript

APA, MLA, BibTeX
APA

John, -. (2026, October 25). Building an open agentic Internet: Discoverable, callable, and payable (Cloudflare Connect 2026) [Interview transcript]. Cloudflare Developers. CEOInterviews.AI. https://ceointerviews.ai/interview/3311889/

MLA

- John. "Building an open agentic Internet: Discoverable, callable, and payable (Cloudflare Connect 2026)." Cloudflare Developers, 25 Oct. 2026. Transcript, CEOInterviews.AI, https://ceointerviews.ai/interview/3311889/.

BibTeX
@misc{john2026_3311889,
  author       = {- John},
  title        = {Building an open agentic Internet: Discoverable, callable, and payable (Cloudflare Connect 2026)},
  howpublished = {Interview transcript, Cloudflare Developers. CEOInterviews.AI},
  year         = {2026},
  month        = {oct},
  url          = {https://ceointerviews.ai/interview/3311889/},
  note         = {Speaker-attributed transcript with timestamps}
}