Let's get started. Welcome everyone, thank you for coming to my talk today. Um, anyone here was patching PHP applications at 6:00 a.m. this morning? Just the two of us. There you go, a few more. Uh, so I'm a little tired, but we'll get through it. Uh, it's not going to take the full 45 minutes. Topic for today is a big one. If you were at the keynote yesterday, there's actually a bit of preview. This is a bit of a longer form version of that, and actually some tactical examples of what we're doing to build, of course, an open agentic internet that helps everyone — content producers, website owners, as well as, you know, the frontier model providers. Uh, which helped me a lot this morning, 'cause Cloudflare did patch a bunch of my servers.
Right. So, the premise to the whole situation. Um, you can actually check this up on radar.cloudflare.com. If you haven't seen the website, I highly advise you take a look. There's a whole bunch of really interesting filters that you can filter by region, activity, bot type, uh, purpose of crawling. The headline, of course, is the majority of the traffic we're seeing across the Cloudflare proxy — give or take, we're talking about tens of millions of websites at this point — is mostly non-human, or better, bot and automated. And we also know that the increase over the past, you know, 12 to 18 months has mostly been driven by all the new use cases for crawling, training being the prime examples. Everyone is building a bot, everyone is fetching content via their agents, and this is the output of that consequence.
It was mentioned at the keynote, um, the order magnitude difference over the next couple of years is just going to grow. So, take an example: you're running a small little website, you may be receiving a couple thousand requests per minute today. You can probably expect that over the next years that number is going to go way higher, and it's all going to be automated, 'cause everyone is hungry for data. We zoom in. Again, this is a screenshot from radar.com. And just as I said earlier, the number one use case — and we do this by actually identifying, where possible, what bot is accessing the content behind the proxy. Uh, you may have heard of Web Bot Off.
Another really cool thing on Radar: you can go and see our bot directory, and you have a list of hundreds of bots we recognize. And a bunch of them, especially the big provider ones, submit a cryptographically verified HTTP header, so we can say, 'Yes, this is OpenAI. This is Anthropic. This is Google. This is Muse from Facebook,' and similar. And based on the bot — um, most of the providers split their bots based on use case — we can tell that, you know, number one case is training or some sort of agentic behavior to power our conversations with our new bot overlords. Um, search is still there, uh, not negligible, about 13% of traffic. That is your traditional search that you might see in Google, Bing, Yandex, you name it. Um, pretty constant, uh, but definitely not more than the agentic use cases.
What does this mean? So, we started a couple — well, a year and a half ago, we had Content Independence Day when we started noticing the behavior changing on internet traffic. And the canary of the whole situation of the business model is changing. Of course, it's websites that produce original, novel content. Okay, I actually run, as a side, one of the little news agencies for my own town, local town, my hometown in Italy. And I've actually seen this myself on the server, looking at the logs. Um, human traffic is dropping a lot, considerably. And I run the website on behalf of a friend who's actually the owner of the website. And uh, just like these big brands here on the slide, his revenue from ads is just going down. His revenue from subscriptions is going down. Why?
'Cause today you open your OpenAI ChatGPT, 'what's the latest news?' Even this morning I was looking for the latest — when was the patch out for PHP? I could have probably checked php.net, given it was the front page article on the site, but I actually asked Claude, and Claude fetched the page for me and gave me the information I needed. Did I, as a human, go to php.net? No, right? And you can see how this is changing behavior for everyone, and those who make money from human visits are suffering the most. Uh, pretty impressive digital trends. This is actually published by a third-party organization that looks at this data, so not even Cloudflare data. 97% less human traffic hitting the site. This is just complete disruption of their business model.
Okay, so some would argue, you know, historically you get a crawler accessing your site, getting your site indexed, that's good. A bunch of us probably have experience with search engine optimization. Is this bringing any value? We can actually see, looking at referer headers — and again, this is public, go to radar.cloudflare.com — how much are these providers crawling, and how many humans are they sending back to their site? Sometimes you actually may see, you know, the chat agent is giving you a referral link, and you may click on it and actually visit the site, consume the ads, buy a subscription or something. But it's pretty bleak, let's be honest. Look at Perplexity: for every 3,400 visits their agent accesses any given site, they're sending one human visit back, right? And it doesn't get better throughout the board, and this means more resources, more load, more servers that you need to stand up to keep your sites online.
Um, we track this over time. If you click on Radar, you can actually see the difference over the course of the past couple of months. And these, of course, are just the top ones that we picked, hand-picked. If you go to the bot directory, you can actually see on any given bot that you might be interested in many more providers than the ones listed on the slide. So, the big question is: what do we do now, as an internet community, or people that benefit and enjoy exploring and using the internet as part of our daily lives? We need to essentially make sure that the business model stays up so that the internet doesn't devolve into something that's worse — let's say worse over time.
Um, as I said earlier, the canaries here, of course, are the news publication websites. They're suffering the most, but you can see how this trickles through, for example, companies that are publishing research, journals, news, weather companies. And eventually it will probably hit everyone, including e-commerce installations, small blog providers. I have a personal blog that I keep updated, and I'm probably going to see the consequence of that. Not that I get many visitors in the first place, but it's only going to get worse. Um, the other thing, some of you may not know this: some of the browsers are actually embedding, or some of the front-end model providers have their own AI-native browser. Someone who heard about Comet, for example, as one example. Um, when you're accessing web pages using those browsers, sometimes they're actually doing crawls in the back end, 'cause they come across as, of course, legitimate humans, and it's harder for, you know, bot detection providers such as Cloudflare to actually detect them and stop them. And, you know, as a consequence, actually making it worse for everyone.
Good news, you know, new business models. This is just the way history evolves. Companies and industries have been disrupted before. So, for us, it's just a matter of figuring out what is the next. Couple of, you know, very well-known examples here: the Napsters of the world. Anyone here used to use Napster? Yeah? They've disappeared, replaced by Spotify with a subscription model. We can debate forever if that's good for the artist or not. Some will say yes, some will say no. There's been some very interesting, you know, scams and similar going on on some of these subscription-based platforms. I still remember as a kid going to Blockbuster, the Italian version of it, long gone, replaced by Netflix. And of course, I did see someone actually with a paper-based newspaper in the tube in London the other day. But fair to say, that's the minority, not the majority. Everyone's reading news online.
Okay. Um, new business model. We're starting to see it take shape. Um, some of the large content producers, for example, have already been able to — and this actually started earlier in the year and into even the late end of last year — signing direct deals with the big frontier model providers such as OpenAI and Tropic and similar. The good news is they actually want to sign these deals with the content producers, 'cause there's a benefit for both parties, right? If I sign a deal with a big NewsCorp and I'm a frontier model provider, I can probably optimize how I receive the data from them, so that I don't have to crawl a thousand times or continuously refresh and fight against them to try and bypass their mitigations, so that I can provide a better service to my end users.
This is the start. I would say at this point the majority of the large news publishers, even if they haven't maybe publicly declared or not, like People Inc., have actually now got in place deals with the frontier model providers for them to access content, especially because freshness of content is extremely important. Right? You want to know what's going on in your local town, you ask the chat agent, it should be able to give you really up-to-date information. It's also important that they get up-to-date information from the agents because we as humans are starting to rely on them for our everyday tasks. And if the agent doesn't have fresh information, we've all heard of hallucinations, right? We may actually get misguided, misaligned, get the wrong content from the content producers. So actually it's good for everyone if both parties work and find a good way of working together. This is just an example of, of course, Google AI Overviews maybe putting people at risk on health advice because sometimes they have just out-of-date or no access to the latest best content with the correct information.
So, at Cloudflare, you've heard this at the keynote, we're trying our best to be the impartial party, figure out how can we help both content producers, website owners, or frontier model providers. Of course, at some point we talk about money, so monetization is arguably one of the end goals, but we cannot just do that, right? We need to make it easy for both sides to achieve their end goal. So these are a few tactical examples of things we've been working on. Number one: making it easier to crawl. Now, arguably lots of people here do not want to be crawled, but fair to say some of us do want to be crawled. And if you have to crawl me, I want to make sure you're as efficient as possible while doing that, so you're not taking my website down or your crawling doesn't look like a DDoS attack against my infrastructure. I've actually had a couple of sites go down because I was getting hit quite heavily by Meta Inc. crawlers a while back, and I had to block them even if I was trying to give them product catalogs and similar.
If you go to the Cloudflare dashboard and you have a website on Cloudflare — who here has a website on Cloudflare? Okay, quite a few of you. If you want to give access, there's a little crawl option. The proxy will automatically issue a new endpoint, which will only be accessible, by the way, by a signed verified bot. So, we're trying to keep things as clean as possible. And we will parse the content out of your site and actually provide a much easier-to-crawl markdown syntax to the bot, so they can access the content over and over again if they so wish to do so, without hitting your backend server or increasing load to your application. Okay? So, this is one of the ways, um, we're trying to make it better for both parties.
Second one: make it easier to search. Um, you may have noticed if you do a prompt on a frontier model provider, on Claude Opus or whatever, very often the first thing the agent will do is kick off a bunch of sub-agents. They will actually go off and perform searches on the open internet. And this is one of the initiatives, but fair to say, still very much under development, um, with a plan of building an open index for all the sites sitting behind the Cloudflare proxy — give or take, we're talking about 20% of the internet — so that the frontier model providers or whoever, if you want to build one tomorrow, again it's about having diversity in the ecosystem, can actually refer to the index and actually get content without having to search for it, uh, which is a lot more resource intensive. Even this, by the way, is not opt-in. By default, you can go to the dashboard, you can decide to turn it on and opt in to the index or opt out. It's your content, it's your application, under your control. Plan is make it easier, so you don't receive the load, and of course it's a safe path to AI adoption over the long term.
Last one, and not least — um, search is part of it, and the index definitely helps. Um, whilst we are building out this index, of course, the search engines and the crawlers have a discovery problem. I remember I used to work at a company called Netcraft before I joined Cloudflare. One of the things we were doing was an internet service survey, and discovering new domains and sites for the purpose of surveying the internet was one of our biggest challenges. And we actually used to partner with the likes of Microsoft to get a list of domain names that were signed up by registries. Um, similar sort of incentive: we actually did a research, and you can find it on our Cloudflare blog. We partnered with OpenAI in this case. We give them signals of new domains that are being added behind the Cloudflare proxy with the crawl endpoint turned on, so opting into being discovered. We did it on a very small percentage of traffic, so not the entire application set behind the Cloudflare network. And you may have heard at the keynote, this resulted in OpenAI, in this case, discovering several million new domains and applications without having to necessarily bombard all of us with their crawls. Um, we think this is the win, especially for the smaller players on the internet. So, let's say you're starting up a new site tomorrow, you will be no one on the landscape, but with something like this, the providers will be able to find you more easily, and you actually show up in their search results or within the agent. Um, so you can get access to your end users.
Okay? There's a lot more we're doing. These are, I think, three of the best concrete examples that we're actively working on. Um, so homework: go in the Cloudflare dashboard, find the options. You can make the decisions to access or not. One last thing about the /crawl endpoint, which actually powers some of the research we did with OpenAI: there's actually a standard called Content Crawl Control, which we only let bots who are at least — it's a trust-based system — wanting to access your content for a given purpose to actually give them that content, be it training, inference, or for example, requiring referral back to your site or not. And all of these options are for you to configure and decide. We're actually pushing for a standard to be done by the industry, but still some work to be done there. But over time we're confident we'll make it happen.
Okay, so frontier models, crawlers, building a new one yourselves, you're a content producer trying to connect the dots for everyone. Now we can start thinking about monetization. These are the last two things of my presentation today. Number one: pay per use. The little video as a background. Um, you as a content provider, a content producer, want to get, you know, some part of the pie when it comes to making money at the end of the day. There's two big challenges here. Number one, you could actually get paid every time a crawler tries to access your content. And you may have heard of our monetization gateway, which is my next slide. Everything is good and great. You can set a price, you can say, 'If you access this API endpoint, I want to be paid 10 cents.' And there's a cryptographic way to make it happen.
However, there's a bigger problem, which is what we call pay per use, which is still under development, and we're figuring out how to make it scale up, which is a frontier model provider only needs to crawl you once, and they may reuse that content 100,000 times. Right? So you need to think about how do we make sure that every time the content provider uses your content, you get a little portion of that value back in your site. Okay, so if you're wondering what's the difference between pay per use and monetization gateway, this is the crux of it: is reuse out of scope versus accessing your content. Um, pay per use is based on a trust system for now. So it needs to be signed agents accessing your content. You set the price in the dashboard. OpenAI comes along. We take a look. We say, 'Yes, this is a price we're willing to pay. No, we're not.' And then they will add the content to their search engine, to their agent on frontier model providers, and in theory every time they use the content, you receive a little value back. Everyone following so far?
Second part is... it goes to the next slide. Seem to be stuck. Could you help me at the back advancing the slide? Monetization gateway. And the video — there you go. Second part is monetization gateway. This is in private beta. If you go to the Cloudflare dashboard, there's a little — on the left at the account level, there's monetization. Click on that. One click to request access to the private beta. This is essentially a technology built on top of the proxy, so that you can then define at access time how much you want to be paid. Easier from a technological standpoint, 'cause we're not relying on the trust-based system. You define the prices. Um, a signed agent comes along. Of course, we'll receive a 402 payment required response from the server. And assuming it's a signed agent and they've implemented integration with our protocol, they will submit the payment to the relevant payment provider and then re-access the content with proof that they have paid. This is the token. Proof happens, and then they access the content that you're providing. And in this case, it can be an API, right? So, if you expect — you've got a service and you're not building necessarily content, but it's an API service, you may want to get paid 1 cent for every API call. Monetization gateway, which by the way is an open standard, um, is the tool for you to achieve that, so you can scale out payments to agents without having to worry about collecting money from each one. Probably very useful for the smaller players that cannot go off and have big signed agreements with the frontier model providers.
Um, take a look in the dashboard. So, sign up for the private beta. You can actually set up rules on all sorts of filters you wish. You can decide, for example, to only make portions of your API, uh, requiring payment versus, um, different filters per region. You control the rules. We implement them as part of the proxy. Under the hood, um, who here has heard of the XYO Foundation? Couple of you, not too many. Um, we actually partnered with Coinbase and a couple of other technologies to come up with a standard and the protocol that makes it happen transparently. I'm not going to go into the details of this right now. Um, again, we are one of the providers that implement the standard. Um, there are others out there. Of course, we're the best ones that made it happen. Um, up to you, you choose. You can even implement it actually on your own servers without a proxy if you wish to do so. Um, everything is cryptographically verified. The only challenge here for us has been over time is how do we scale up this concept of micropayments? You may also have heard that we issued and are building out a stablecoin called Net Dollar, which is part of one piece of the puzzle to hopefully make this happen as it scales up to everyone using the internet at large.
Um, and yes, and hopefully, uh, maybe by the time we meet again next year at a similar conference, we start seeing the new business model of internet taking shape. With that, I promised it was going to be sweet and short. So, thank you for listening to me today, and uh, yes, any questions from the group? I'm happy to answer them. Ben? I think we have a mic so everyone hears you.