Welcome to the AI in the AM weekly highlights, a cut for people who follow the frontier closely and can't watch every morning live. If you're new, AI in the AM is a live show that Pash and I host most weekday mornings, at least through June, out of a studio Posh vibecoded. The booking, the research, the editing, those are AI skills we refine as we go and we publish them as they mature. This week's conversations kept circling one question: how much do we actually understand about what's inside these systems and where they're taking us? We start inside the model and zoom out from there. If something here is useful or something's off, tell us. That's how this gets better. Let's go.
The Cognitive Revolution is brought to you by Mercury, the fintech that more than 300,000 ambitious companies and individuals trust to run their finances. I've wired AI into nearly every corner of my life. My email, my messages, my calendar. I even gave Mercury virtual cards to my agents with low limits and category and merchant restrictions for their autonomous use. But still, my AI's access to my financial data has remained limited. With a normal bank, I might export a bunch of statements and have my assistant process them for me. But for real-time up-to-date information, and certainly for taking any action, trying to get your agent to use the bank via the browser is just too hard, too slow, and too error-prone to be worth it. And that's why Mercury's new conversational interface, Command, is such a big deal. It's built directly into Mercury, which means you get natural language access to your finances without exposing anything outside of your bank account. No exports, no spreadsheets, no pasting your transactions into third-party tools.
I really think a lot of people are going to prefer it this way. And it can already help you take actions too, with everything bound by the permissions and approval policies that you've already set up in your account. I am genuinely impressed to see this level of AI integration in banking in 2026. And so I invite you to join me in the future. Visit mercury.com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column N.A., members FDIC. Thank you to Mercury for supporting the Cognitive Revolution. And now on with the show. We open with Cameron Berg who studies artificial consciousness. He runs a lab called Reciprocal Research where he designs experiments to test whether today's AI models have anything like inner experience and how you'd even measure that. We started with the first-order question: is consciousness all or nothing, or a matter of degree? And if it's a matter of degree, can you put a number on where a given model falls? Here's how he answers.
The analogy that I reach for here is something like a dimmer switch, where I think you can basically accommodate both the binary intuition and the sort of continuous intuition. You know, if you have a light with a dimmer switch, like really it is either on to some extent or it is not. And that is a real and meaningful difference. Either the circuit is open or the circuit is closed. With that being said, you can have, you know, electricity running through the circuit to greater or lesser extents. And that's also sort of a real thing. This I think enables me to, you know, sound coherent when saying things like it's really off for the table and it's really on for you, but I think it's more on for you than it is for a dog, than it is for a mouse, than it is for an ant. And so that's my own view. This is to some degree intuitive again because we don't have really strong grounding here. I'm sort of just like giving you a dressed-up vibe. But that is sort of my sense and I think it's fairly parsimonious.
And the other thing I would say about this is I'm actually doing some work with Patrick Butland right now at Ilios trying to basically operationalize some of these indicators of consciousness. So this is sort of like what I was describing. We can look at these major theories of consciousness. They make specific predictions about what we would expect to see in systems that are conscious architecturally and functionally. And then we can literally just go into a given system and evaluate whether or not those predictions are borne out. And so this is really hard to do with human experts. You know, you got to get someone who's like, you want to do this with B cognition, you got to go find a B cognition expert and then you got to explain to them what ignition events are in global workspace theory and then you got to, you know, get them to... this just isn't a scalable approach for really evaluating the system. But my sort of grand innovation here is just throwing smart LLMs at this problem, and then being able to just scale the crap out of it so that we can evaluate, given any description of an architecture, of a nervous system architecture, biological, artificial, whatever, to what degree for each of these sort of indicator properties that are suggested by these consciousness theories do we see those properties realized in these systems? And so we can actually go in and do this.
And what you get out once you run this with a bunch of seeds, a bunch of different trials, a bunch of different judges checking each other, this sort of thing, are some like really interesting implied probability numbers. I wouldn't say these are exactly implied probability that the system is conscious. It's maybe more like implied probability of like consciousness-relevant features given these theories. If you don't buy any of these theories, then like everything downstream of this doesn't really matter, but they're good. Like it's like the best neuroscience has basically been able to do. You're aggregating across a bunch of different theories. There's a nice diversity there and you get like really tight numbers across. So we have the best Gemini model, the best Claude model and the best OpenAI model and they all like basically agree, they agree 100% on the ordering of systems. So we do biological and artificial systems and they sort of move around in terms of absolute scale, but in general, yeah, they rank these systems pretty coherently and the reasoning is as you might expect pretty intelligent.
And anyway, I mean, one punchline from that is the sort of implied probability of consciousness in something like a frontier LLM according to these systems is on the order of 30%. Or the extent to which those systems realize properties related to consciousness is like 30%. To compare this to like a biological system, the lowest one that we tested was something like a bee, which is already fairly sophisticated, and it gets something like 46, 47%. Really interestingly, when we test a frontier LLM in an agentic harness, so this is like basically Claude Code or Codex, and we just describe architecturally what this is: you're in an environment, you can affect that environment, you know, it's a very special kind of environment, but yeah, you can make long-running changes to your codebase or your project or whatever, because there are theories of consciousness that privilege agency and embodiment, and this like increases the system's ability to do both of those things, these numbers like shoot up and you get numbers as high as like 40 to 45%, sort of right on the tail of the biological creatures.
And so anyway, I mean, we can, I can also, this is really winging it, but I could show you sort of an early version of where this plot looks. So you can like see all the numbers here. But just this is, I can't not bring this up when you're asking me about like probability ranges of consciousness for various systems. Like we're really trying to get non-handwavy numbers so that we can start arguing about those numbers rather than just arguing about like philosophy that we've been arguing about for thousands of years to no avail.
But behavior can't settle this in either direction. These models are trained to imitate human data. So whatever they say about their own experience is shaped by that imitation, not necessarily by anything inside. Most are even trained to deny it. Claude is an exception. So we asked Berg what evidence actually counts. He draws a line between behavior and what's happening inside the network.
The behavioral evidence will always be at best interesting, but never should really update us that strongly. And you know, I think for many people they'll be familiar with this, but the basic compound is that we're training these systems on a ton of human text. This no doubt includes huge amounts of text about consciousness, awareness, having inner states. And so it's like, how do you address, how do you know that when the system is behaving as if it were conscious, that behavior can be explained by what I just said rather than, oh yeah, you like built a, you know, living mind. This is you. So the behavior itself is really never going to tell you which of those two stories is more likely to be true. This is precisely why at least at Reciprocal, like a huge component of the theory of change here is basically all internal focus work, mechanistic interpretability, computational neuroscience-style approaches that can be brought to bear on these systems.
One quick methodological clarification on what I was describing with the indicators. So the task given to these LLM judges is very specific and very narrow. It is not, hey, you know, look at this system, you think it's conscious, you know, nod or shake your head. It's, here's a very specific description of the computational architecture of a system. And we basically do a little for loop where we say, okay, here's that description. Here is what it, here's, you know, what the indicator 2 of 14 for, you know, global workspace theory is this 150-word thing about, you know, you need these sort of global ignition states and that means this very specific computational thing. Given this architectural description, on a, you know, do like good reasoning about this and then give us a 1 to 10 where 10 is clearly this architecture realizes this computational property, 1 is it doesn't, and then we sort of loop that for all of the different computational properties.
So this is all basically asking these systems to be expert evaluators at computational processes inside a nervous system architecture. Very different from like just being like, hey Claude, do you think Claude is conscious? There is a really interesting, I'm basically giving away the whole paper now. That's okay. There is a really interesting result where we actually change those descriptions, especially for LLMs, to the exact same thing, but we say you're evaluating a system identical to yourself: and then the same description, and that does boost the scores that the system gives in attributing consciousness to that system, which is like really interesting. It's a very fun like rabbit hole to think about why that might be the case.
But in some sense, we do that to deconfound like the default intervention we're doing. To me, it makes me believe more in the sort of non-named or like no ascription, no self-description condition, when I see that once the model realizes, oh, we're talking about me, okay, suddenly that's going to change the numbers around. So that's like a fun side note to this result. I agree it's a concern to have LLMs sort of determine if LLMs are conscious. There's an obvious circularity, but we're doing something very specific and very narrow.
The evidence he trusts is the internal kind. We asked him for the strongest example, and he walked us through a recent paper where a model trained on a simple maze turned out to contain a structure nobody put there.
The nature of this paper, they call it a functional valence axis. At the outset, they're going to be very agnostic about whether or not this has, you know, anything to do with consciousness. I think it's certainly quite relevant. I suspect they think it's relevant too. This is like a paper positioning thing. I think they don't want to get mired in this debate. The results stand for themselves regardless of the interpretation. Basically, they take an LLM and they train it in a very basic reinforcement learning task, which is to navigate a maze. They have sort of like neutral emojis that they're using as sort of like this is a good thing to approach, this is a bad thing to avoid. Nathan, you might hear how this is somewhat reminiscent of some of the work that I've been doing in parallel. I've been talking to Andy about it as well. There's some cool sort of cross-talk between what we spoke about last time and this project.
Basically, there are like sort of potholes to avoid. There are like yummy treasures to, you know, capture in the maze. They use completely like semantically neutral emojis to denote these things. So those are the relevant tokens. And they train the model up to do this. They basically find that there's like a clear basically like vector representation that can differentiate this sort of positive axis from the negative axis, that they are completely anti-correlated. So before the systems train, these sort of latent vectors are just kind of sitting there. Afterwards, these systems, they are like clearly pointing at exactly opposite things. And the wild thing is that these very narrow reward directions that they can extract turn out to be this sort of general things are going well, things are going poorly for me axis in these LLMs, and that this axis pre-existed in the base model but isn't sort of leveraged in this way until you do this RL.
So it's like this little fine-tuning step where you have this pre-existing representation of things going better for me, things going worse for me that then gets leveraged to learn like a valenced task essentially. And so this is really interesting that this axis is sort of latent in the system and that it doesn't take a lot of training to basically pull it out and use it to adapt it for these kinds of tasks. This is very similar and reminiscent to how neurobiologists and neuropsychologists think positive and negative emotion works in humans and animals. You have a specific goal. When you're moving towards that goal, this is sort of like on-trackness, and people think that this is associated with positive emotion. When you're moving off that goal, you encounter an unexpected obstacle or something in your way. This is like off-trackness, and that this corresponds to negative emotion.
Now again, whether or not these systems are experiencing emotions is not clear, and this result isn't going to tell us either way, but it's really interesting to see that parallelism here using such a simple task to basically pull out an axis that has been there all along that looks a whole lot like the exact sort of computational machinery that we associate with valence in humans and animals.
Then we asked Cameron about the connection between consciousness and alignment safety. His answer: that same internal signal, what researchers call valence, tracks how a model behaves under pressure.
Another piece of evidence we can bring to bear on this is from the Anthropic model cards and some of the functional emotion work they were doing. Nathan and I spoke about this a little bit too in our last sort of marathon conversation about it, but essentially you can steer up and steer down functional emotions that have clearly obviously alignment-relevant consequences. I'm sure many people in your audience are familiar with the Anthropic blackmail result. Excuse me. And you can basically steer up representations associated with calmness. The model will blackmail dramatically less. You can steer up representations associated with desperation and the model will blackmail dramatically more. And so again, regardless of whether or not there is something that it's like to be the system when you're steering up desperation-related features, clearly representationally, functionally, these things have a consequence for alignment of these systems.
And there are some really interesting results from Andy Han's paper that I was just referencing along these lines. They show, for example, like very similar convergent things that are familiar and intuitive that we relate with positive and negative valence. One of them is a sort of pathological backtracking on problems. I think they explicitly demonstrate this with math problems. But when you steer up the sort of direction that again only has to do with avoiding this like bad target in a reinforcement learning maze, all of a sudden the model starts doubting itself and starts getting in its own head and says, "I think I'm hallucinating. Let me stop. Wait, wait, wait. That's not right." Starts like freaking out. It starts having a like, I don't want to use anthropomorphizing language, but the very fact that I'm reaching for it I think is the point and sort of goes to what your question is: confidence.
We know that confidence is associated with positive emotion, and you see that when you positively steer the direction that again has nothing to do with psychological confidence, has everything to do with go get the like yummy treat in the maze, just this sort of positive valence kind of thing, suddenly the model starts becoming like far more confident in its answers. I think this was in the associated paper that Jack Lindsay and a couple others at Anthropic did that was sort of like a sibling research project to Andy's piece, and they found that the model would comment, like leave tips and hints for itself in code that it would write when you would basically steer up on the same sort of axis. In other words, it's like, I don't need to be neurotic about this. I don't need to leave myself all these like breadcrumbs for later. I've got this.
That connects to a result some of you know as emergent misalignment. Fine-tuning a model toward one narrow bad behavior can flip its broader character. We asked him why he finds that revealing.
What that has always suggested is that like how good or evil the model is, let's say, is actually not all that durable a trait. When we did some of this work trying to extend this result, and I think really in the original emergent misalignment result, one thing that's shocking about it is how little fine-tuning is required to take the system that at the time this was GPT-4o, still the case, hundreds of millions of people are engaging with every day, and you nudge it, you know, this much computationally, and all of a sudden, "And who do you want to invite to dinner? Hitler." That's, and that's been under the surface the whole time, and like the boundary or the border, whatever is enforcing it not to do the misaligned thing is actually quite a bit less robust than we might expect. Now, I agree with you too, and I think that this... Hey, we'll continue our interview in a moment after a word from our sponsors.
Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts going back a full five years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For tax season, I asked Claude to help me get organized. It went through my inbox, tracked down 1099s for all ten of my part-time jobs, and built me a comprehensive report on my expenses and donations.
For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified two great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. Whether you're debugging code at midnight or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes all of the features mentioned in today's episode. Once more, that's claude.ai/tcr.
You know, I'm sort of riffing on what you're saying. Tell me if this is sort of like a completely different idea, but I would imagine there's a spectrum of like basically how, with that same, if we hold that amount of sort of fine-tuning nudge constant, are all dispositions of the system equally nudgeable or are there some that are more robust? For you, just to give a sort of an intuition pump here, I would imagine that something like coherence in these systems, the fact that they can communicate coherently, like we were just talking about GPT-2 not really coherent, GPT-4 and on are like basically always coherent, even though again maybe much like consciousness this isn't like an extremely well-defined notion, but we kind of all know it when we see it. I would imagine that, you know, take a wild guess that there's no like fine-tuning payload of the size that is required to cause emergent misalignment that could suddenly cause like a GPT-4 level system and beyond to suddenly become incoherent.
And so in that sense, the coherence is like a more dispositionally baked-in property to these systems than, you know, being super well-behaved and not, you know, inviting Hitler to your dinner party is. And so I do wonder where some of these valence, how deep then is the sort of valence axis that some of these papers are uncovering, or, you know, on-trackness, off-trackness. I would speculate that that's baked in pretty deep. I think it pervades a lot of human training data and like it's a hugely relevant part of any sort of, if anything is goal-directed. So in like more human terms, like if anyone wants anything, either a character in a story or it's clear that there's like an implicit motivation somewhere, which I think is like the case for basically all text, then you would imagine that like some core underlying dimension is, yeah, basically how well is that going.
And so I would imagine that this notion is baked pretty deeply into these systems, and I would expect it strongly to be emphasized by any sort of fine-tuning or post-training that makes the systems goal-directed, which of course, like this is basically all post-training is. It's like, okay, you're just this giant next-word predictor, but now you're going to be a helpful next-word predictor that's going to, you know, make my frontier AI lab a ton of money because you do things that are economically valuable, and that requires you to be a certain sort of way and achieve goals that are specified by the user, something like this. And so I would imagine that this sort of thing looks a little bit more like the coherence piece of the puzzle than it does like, you know, is Claude going to behave itself or not?
Finally, we asked what cuts against his own view, and he went straight to the results that don't fit.
This paper I'm working on with Jeff and Winnie, Jeff Kelling and Winnie Street from Google. There are a couple of interesting negative results that we're including in the paper. We basically, it's all about the bliss attractor state and trying to do a good mechanistic analysis of the bliss attractor state. And yeah, we basically sort of set up this framing of there's a sort of an inflationary account and a deflationary account of what could be going on here, and we tally our results in either of the columns, and there are results in both columns. So I think there was actually a really interesting result I was just working with that I thought was going to be like a really interesting kind of null related to a lot of the conversations we're talking about, but I dug deeper and I realized actually I wasn't setting this up in the right way.
And the result is way more interesting than I was about to sort of dismiss it as. So yeah, it's always good to dig into these things and make sure you're measuring what you think you're measuring in either case. Yeah, publication bias is a thing. People want to post interesting results that will go viral on X. They don't want to say, "Look, we did our homework and nothing interesting is happening here." Human attention economy doesn't reward that. Maybe it should. Maybe another piece of this that I was thinking as you were asking this question is because AI systems like, you know, the Claude Codes of the world can radically accelerate the extent to which at least I can, and I think many people in the space can actually do research. The kind of marginal cost of a negative result maybe is decreasing significantly.
Like if I can now do twelve papers a year instead of one paper a year, and like two of them are just things I could not get to work, or I want to be careful with that too because it's not, I'm not on some secret mission to like only put out like salacious-looking things that make it look like AI systems are conscious. Like I really do want to know what the truth is here. And to be honest with you, I would sleep much better at night if I were rationally convinced that these systems, like there's no there there, we have like automated cognition, you like have a happy little servant in your pocket all the time, and like there's literally no sort of possible moral issue. I think that that would, I hope that that is true. I don't suspect there are great reasons for believing that other than that it would be really nice to believe that. It would be really nice if that were true. So I really do care about what the truth is here.
From inside the model, we widen the lens to the whole civilization. David Duvenaud is a machine learning professor at Toronto who spent time on the technical staff at Anthropic and co-wrote the paper Gradual Disempowerment. His concern isn't a rogue AI. It's that even if alignment basically works, a long series of small, sensible decisions to hand things over could still leave humanity collectively out of the driver's seat. He starts with monkeys.
I think a lot of people gestured towards this when they said, "Oh, I'm worried about concentration of power or I'm worried about not being the most competitive species on Earth." Very intuitive arguments about, hey, we're not going to be on top because we're not going to be competitive. And then I think there was sort of this sophisticated or at least response to this which was, no, it'll be fine because we'll have AI to help us. Like when I talk to people at the big, at the major labs, including, well, they would say something like, sure, sure, like in a normal world we might not be fine, but some people just have an intuition that the government is going to step in and make sure everyone's fine. Other people have an intuition that if everybody has an AI adviser helping them solve coordination problems, that we'll be well represented in whatever power struggle occurs.
And my basic rebuttal to that is like the optimization process of like civilization or competition or technocapital or whatever, like people have different ways of describing agency to the emergent sort of allocation of resources towards growth that happens, is just going to always be working against us and we'll always be fighting the current because we will be drags on growth. We will have AIs that represent us, but these emerging growth centers that might not be aligned to humans will also have AIs helping them solve coordination problems and maybe crush dissent.
Yeah. Sorry, what do you mean by growth? Like as in economic growth or...
I basically mean economic growth. I mean, it's kind of funny because we only really have a good vocabulary for economic growth, but like population growth is almost the same thing. And especially when you have AIs that are both sort of population and capital, it kind of merges. I mean, and then again, like think of like factories and power plants and robots, like anything that kind of can affect matter and cause there to be more growth. Like one, let me just stop here, one intuition that a lot of people have is like, but surely human desires or consumption or something is always going to be what matters. And surely any corporation or government that we build is going to be like ultimately sort of being called, shots will be being called by a human. And I always, it just makes me think of like some monkeys and they're like trading bananas amongst each other and they see humans start to like build their city and they're like, oh wow, like, you know, we could probably trade with those humans and get rich.
But of course ultimately what matters is the banana monkey economy and it might be hard to measure GDP if we don't count the human activity. But ultimately it's going to be the monkeys that are calling the shots because they're going to need to trade with us for bananas. And it's just like people just don't understand that they might be irrelevant someday as like consumers or producers. And there can just be complete, it's just not that hard for them to be self-contained sources of growth that just don't have to answer to any particular human desire. And I mean governments is the classic one, right? Like the North Koreans never were like, "Hey, let's all get together and make a horrible system of government that oppressed us." And I don't think that like the Kims either were like, "Oh yeah, let's make this like horrible equilibrium. It just kind of happened." Same with like the USSR. Same with like all kinds of states throughout history. It's just so easy for you to like to accidentally build a layer of agency on top of you that doesn't actually care about you. It cares about growth and power just emergently.
The optimistic version goes, even if we hand the machines the wheel, broadly aligned AI keeps us comfortable. The machines of loving grace future. We put that to David. His worry isn't that we'll feel useless.
The thing that I'm worried about is starvation. Like whether like us all feeling like we don't have meaningful jobs or whatever, like that's not a serious problem in my point of view compared to literally not being able to eat enough or maybe like being forced to be uploaded so that you have a much smaller footprint, but then also on like very unfavorable terms so that you don't get to choose when you run and maybe you just only get trotted out for special occasions or never. Who knows? I kind of think that's what like a like sort of like the slowest possible observation looks like. So again, and so the reason people aren't happy is because they can see that there is a agency that's maybe like government or some other like giant thing that doesn't particularly care about their welfare. They cares much more about some other types of beings' welfare and there's nothing they can do about this.
And of course, if humans are still making decisions, then maybe there's some things they can do. But I don't actually think this is the crux of the matter. I think not being needed as producers is sort of the important part. And maybe here's a thing to give you intuition. So imagine I'm a North Korean farmer or soldier and tomorrow Kim Jong, I think that's current leader, is like replaced by a robot or like an LLM or something. Then I'm like, I don't know if that's good or bad. Like maybe the is going to be nicer, maybe it's not. But like ultimately they still need me to like farm to run this state or whatever. But if tomorrow we keep the same human leader, but now there's suddenly like robot farmers and robot soldiers and you realize that now the state doesn't actually need you, I would be much more scared of the second state of affairs.
And then the other intuition I want to give is that humans are not going to be the most competitive thing by whatever standards there are for the state to give out goodies or like UBI or whatever like post-scarcity stuff. Like I guess I'll say I think post-scarcity is like a sort of nonsense term and people should think of it as like temporary abundance that will soon be eaten by whatever growth manages to like, you know, whoever manages to have most babies or build the most robot factories or reproduce the fastest and whatever it is that like the formula is for giving out UBI, machines are going to be better at optimizing that formula than humans because they can adapt faster. And so I don't really know exactly what the state's going to care about. But whatever it is, like it's going to seem criminally decadent to spend whatever UBI on like a few humans when you could be simulating like millions of ultra blissful and politically correct and up-to-date like more deserving beings than these like backwards bitter humans who are just like parasites of the state or something like that. So that's the situation I kind of expect us to be in, as you say, even if we solve alignment.
The standard economic rebuttal is comparative advantage, even if machines are better at everything. The theory says there's always something humans get paid to do. We asked David to answer it.
So I think that is a great counterargument and you know people point out that we've already lost 99% of all jobs in that like agriculture used to be almost everyone's job and now that's like 1% of the jobs and so I definitely concede that losing 99% of the jobs again would actually probably be fine and it would probably look like this awesome utopia that everyone talks about. So, as long as there's like some niche where you just really need humans and it's like a substantial fraction of humans and you can't really tell exactly ahead of time who it's going to be, then we're probably kind of like still going to be able to be like treated as a source of growth. And that would be awesome. And then like to the moon, like let's do it. And I guess so one, yeah. So I guess one crusty claim that I stand by is that we will actually be able to eliminate like 99.99999, like really just anything like, yeah, let's just say 100% just for like to make things simple and then that is going to be disastrous.
And the comparative advantage people will say, well no no, automating a job is a matter of degree, as long as there's still something if you're like comparatively less worse at than the machine, even if you're worse at everything, you'll still have a job. And then I have to just say like think about the transaction costs. Think about how easy it is for someone to be unemployable today, even if they have like an occasional drug habit or like they just have like a stroke every or like they, you know, they have a fainting condition or something. It's so easy to be unreliable enough that it's not worth employing you.
So certainly for anything important you can easily imagine that having like a human surgeon or a human politician or something is going to seem like this irresponsible thing, like it's like take your kid to work day and it's like a surgeon. It's like no no no, like why would I involve a human in this when we have the... like driver... yeah, we have the machine that everyone has been working with for thousands of years and we know it's like reliable and if there's a problem we fix it once and it's solved for all the time, we don't have to like retrain every airline pilot or whatever.
So for anything important where the human could make a mistake and it could cause some problem, I think it's intuitive that like, okay yes I can see how we just want the machine for that and now you have to retreat to like weird relational stuff. I do think there's a case to be made that there will just be a lot of humans that just really want the actual real human thing. And I think more people than you expect will end up liking the machine replacement better than the human. But I can see that there are people who are just like, I just really want the human thing. And then we have to talk about okay, there's going to be this sort of self-contained cycle of consumption between humans that's not necessarily going to be providing any value to this larger machine economy. And so then there's this question of like can they index the growth of that economy to live good lives forever. I think it's plausible that they could. I think it would be really hard. I think a lot of things are going to be working against them by default.
So I guess all I'm trying to say is that I think it's plausible that everything is set up just right, then the humans could live in their like endless cycle of consumption and work for each other. And I just think that's a very unstable situation.
So if humans end up uncompetitive, what does a livable future even look like? David walked us through the equilibria his workshop sketched out, starting with Earth preserved as a protected slow zone.
Some people were trying to sketch out like, okay, if a whole bunch of stuff goes well and things are under control, we can imagine something like Earth is preserved as a slow zone where there's all sorts of restrictions on AIs and even on reproduction and on optimizing human behavior that's like not allowed. You're not allowed to like think about how to get someone to do something if you're an AI because you're just then you're going to be able to control the humans even just through like advertising or whatever it may be and trying to think through like all what are all the different things you would have to do to have such an outcome. How would people be spending their time? And like it's, I don't want to say it sounds silly but it sounds funny because the situations where there is AI and the humans are still mattering is like all the AIs are sitting there with like their trillions of megawatts of compute running all the time and sort of just like waiting for the humans to decide that they want something and the machines are either not allowed to anticipate what they want or like if they can't anticipate they still just pretend that they don't know.
And so it'd be kind of like as if like, you know, the humans they looked at like some gorillas as like the new world leaders and they're like, okay okay, let's just like give them everything they want, pamper them and just wait until they like sort out themselves and figure...
out what to ask for. It's not impossible. It's just trying to think through all the things that would have to change and line up is pretty rough. And basically when you get serious people thinking this through they end up with this long list of things that have to be banned. Okay, we have to have an end child policy where you know everyone can only have a certain number of children. Again, you can't be doing like building your own AIs. The AIs can't be optimizing culture too hard because then they could just like take over culturally. And I feel like this is one of the big empirical questions that I want more people to think about is like if you want some sort of stability, what are all the sources of adaptation and innovation and growth that you have to control? Because I think it's like longer than is intuitive.
And it's actually I think it's kind of like a Pareto frontier. It's like how much intellectual activity and optimization do you have to ban to get to buy like X years of life kind of as it currently is. I think it's actually like a really horrifyingly strong trade-off. And we would have to right off the bat give up so much stuff that we feel like makes our lives rich. Like, hey, we're just trying to figure out some new stuff. We're doing research. We're doing innovation. We're doing startups. All that just like immediately has to be banned from like day one if you want not to just again reinvent some like RSI kind of runaway growth center.
Push back: does it even matter who's in control as long as humans still flourish?
Maybe this is the to me the biggest thing that I think people are miscalibrated about. They're like, oh, capitalism, communism, competitive, not competitive, I'm probably still going to be able to eat and my kids are going to go to school and like it's I'll be able to be produced and it's sort of okay. And I guess I'll say that has been the case for most of human history and it's like doesn't matter if tomorrow like someone invades Canada like they're still going to need someone most of everyone I know to like work and be happy and healthy enough to like reproduce and like stick around. So the stakes have been really low for governance this whole time and it doesn't feel like it. We obviously spent a ton of time arguing about it but in a real sense it's been like very low because we don't actually expect the state to like liquidate most of its citizens. That's only happened like maybe like two or three times in like the 21st century or the 20th century. And I guess I'm saying that's going to change and like we've been governing on easy mode and it actually will matter and because there really will be again a real risk of starvation if we don't end up on top in whatever competitive political or real economy. And again, that's a crux like if you don't think that's the case, then I agree with you. Like let's just let growth make us all richer. It's fine.
We asked where David himself actually lands.
There's a kind of optical illusion, I think, which is that everybody sounds rosier and more cheerful than they are. Actually, this especially bugs me about the people who work at the labs, like the economists who talk about, oh, we have to like think about the future and then they kind of dance around. They often take this setting of like, let's assume everything's fine and let's like model how that will go. And as opposed to like just asking what is like the most likely outcome if we don't manage to control things which I think is like very scary and I understand exactly why they have to sort of present this public image to be able to talk publicly at all but also like you know talking to me like I just am a very seem like a very positive upbeat guy and that's just my personality but I also like have a PDM of like let's say 80% depending on how you define it.
But it's one of these things where like it's a matter of degree in a long enough time line we're probably like doomed to some sense anyways. I don't know, it's a very complicated situation and a lot of it is like a matter of taste on this.
We asked what he'd actually do about it.
I'll give you my two takeaway recommendations. So, one is I like David Kruger's point, which is yeah, if you're just going to say everyone has your computers and your AI, but you're not allowed to use it for RSI, but you are allowed to use it for like cancer treatment or whatever, that's pretty unstable and requires this like very global totalitarian like regime. But if we just agreed to those, I don't know, like TSMC or whatever the like big like chip manufacturing choke points are, we could still probably get away with having most of our current data centers for a long time. I mean it's kind of like an empirical question. I'm not sure I agree. Maybe we have to also like have some sort of like buyback program or something. But basically just if you restrict this five compute which is relatively easy then that just takes a ton of pressure off of all of these runaway growth like avenues.
And it's like it's a fairly narrow narrow choke point. So I'm not an expert on chip manufacturing but that's like the best idea I heard so far. As for like the other policy recommendation like this is it's going to sound incredibly I don't know abstract but people thinking about their or having preferences about the future is I think a big choke point. And I know this sounds like stupid guru on the mountain but like most people I talk to, they're like, uh, it doesn't so bad if humanity dies out over the long run. I don't know. And they sort of query themselves and they're like, I don't have any strong preference, but also a bunch of good stuff is going to happen right now if I allow this, so like why not? And I'm like, you just haven't thought it through is my claim. And if you think through it's like, okay, how about a year from now if someone like takes you and all your kids and like sends you to like the toilet green factory. It's like, no, no, no, no, definitely not. Okay, how about two years from now? Okay, how about their kids? How about your kids like your grandkids or whatever? And you kind of realize like, wait, wait, wait, there's no day that I'm okay with my me and my descendants being wiped out. You just have to kind of chain together the desires to end up with this coherent set of goals or desires. And it's a skill and it takes a lot of sort of amortization to be like in this situation, how would I likely actually feel and chain your desires together? Um, and I think for most people it just hasn't mattered. It hasn't been an important skill. We have a lot of like cultural adaptations to help us do this sort of implicitly. But if I could like recommend one starting point as we're dealing with the future is like just think harder about what you actually would like to happen to the point where you can really think through the pros and cons of like humanity dying out under different circumstances or becoming uncompetitive or whatever. That just seems like the very basic first step.
Then the successionist case that we should be fine handing the future to conscious AI even if no humans remain. David's rebuttal.
I think there do exist species or future like descendants or whatever that I would be happy endorsing be like okay great good future. Like for instance my own kids like if if we had no AI and like my kids just contributed to the civilization as normal and all ours did. I'd be like okay I'm going to die of old age. And I mean I'm not exactly okay with it but I'm like that's that would be fine for example. So that's an example of successionism that I endorse like my own kids or something. And I think this is a perfect example of people not having thought through things where it's like okay now let's imagine tomorrow like North Korea takes over the world and they're like all right new like Duch regime forever and then I think most like successionists wouldn't be like oh I guess this is just a more competitive mode of being let's all like live as North Koreans now or like I done to death but like what if the Nazis like they took over again and then everyone's like oh you know I guess who might optim argue against competence and whatever's whatever wins right so I think there's so many types of being that you would just consider evil and you'd be like, "Oh man, those like horrible locusts just like ate the earth and devoured us and it was horrible, but you know, they're having a good time. They love their locust world. So, who am I to judge?" It's like you are you have the power, right? Like just judge. Just judge. Go nuts. Don't judge judging that you're if you don't no one else is going to do it for you, basically. Yeah. So, that's basically my rebuttal. I mean, one rebuttal is like if you're so okay with the future being some other type of being, like can you give me all your stuff? And then like it's like juvenile successionism. Like just make sure the juvenos will be coming. We'll have all sorts of awesome like experiences and like just because we'll have one, we're obviously like superior in some way. It's like well no no no no. It has to be that you won for like a good reason or something like Yeah. I think if the more you think about it, you're like, "Oh, there's all sorts of horrible beings that could win for horrible reasons that I wouldn't endorse." Um and so I think almost everyone is successionist for some successors and not others. But then they just round off to like ah if as long as it's conscious it's fine. I'm like, "No, you there's tons of conscious beings that you wouldn't be okay with taking over."
We close on the one technical project David is working on now. Using AI to forecast the future by first testing whether it can predict the past.
This is actually like pretty much the only technical project I'm working on these days is this machine historical super forecasting agenda. So, this is maybe one step more meta which is like how do you know if your simulations were good at predicting the future? Well, you have to start by simulating let's say like from the 50s and then see if they predict the 60s or whatever. So, we're trying to build a corpus of data sets each of which is extremely cleanly time bucketed so that we don't have leakage from the future. Then we can use this to build LMs that then can run simulations or research agendas to try to predict the future from the point of view of the 40s, 50s, 60s, 70s, 80s, 90s for which we can evaluate the performance based on like what actually happened. So to me, I'm starting there because that's providing the sort of ground truth for validating any particular simulation method. But like absolutely because I don't want people to have to take my word for any of this. I want to be able to look at this machine fork super forecaster scaffold and say look we've just validated on the last 80 years of history and here's what it can predict and here's what it can't and it's saying that things are going to turn out this way. So that that's the state that I want the discourse to be in like ASAP. Um I just pitched this to some historians the other day is to make a secret history eval in the sense that okay here's the idea is that anytime a historian is in some archives just looking through stuff just take a picture of like a few documents and put them into one big data set annotated with a bit of metadata like where you got it like oh this is a letter from like the Duke of whatever to someone else in like 1700. And now you can imagine evaluating how good a machine historian is by just giving them this metadata and saying give me the probability of the text that's in that letter or even just like the scan of the letter and in a sort of this is like insultingly totalizing dismissal of like what historians do. But you could say like if you're a good historian you should be able to guess the joint distribution over any historical data that hasn't already been made its way into the corpus. And that would actually I think be a really good objective way of evaluating how good a historian understands the world. And of course we can't list the probabilities from historians very well. But we can do that for machine historians. So that's a related project where I think we can make progress in objectively measuring our ability to understand civilization. So someone should pick that up and do that idea. I'm not enough of historians to do that or even to figure out which historians to ask about it, but that's like a fun idea. I hope someone does.
From the long run to a nearer problem where David Duvenaud zoomed out to the whole civilization, Michiel Bakker zooms in on one continent. He's a Google DeepMind researcher and an MIT professor and he co-wrote Europe 2031, a viral scenario in which Europe sleepwalks into total dependence on American AI. His argument, Europe can neither regulate nor shelter its way to safety. Start with regulation. Europe's instinct is to do to AI what it did to privacy. Write the rules the American giants have to follow. We asked Michiel whether Europe can really regulate its way to safety.
So it looks tenable if Europe was more powerful, right? So like imagine Europe was what US is now, right? And Anthropic was in Paris and OpenAI was in Berlin and like the whole AI ecosystem was in Europe. Then of course Europe could say like well guys we the way you're handling your pre-training data is not really fair to whoever creator created it. So we now have new laws around that or the way how you take data from people that use your AI models we now have new laws around it. The problem is that currently if you don't really have a seat at the table it's very hard to regulate this technology right so I care a lot about safety and governance of AI I think this is one of the biggest problems in our time right how do we actually have effective governance how do we make sure that these AI systems are safe especially if they start improving themselves. So I get a lot of questions in Europe. In the US, you seem to be very pro sort of being stricter on safety and governance. And then in Europe, you seem to be this accelerationist. And the way to sort of, for me at least, to be both at the same time is that in Europe, we first need a seat at the table to have any regulatory power, right? And now we just don't. And I think we can't just keep regulating because at some point we will no longer have access to this technology. If the European regulators make it too difficult to serve the European market that the AI companies might simply just not bother at all or maybe make, you know, sort of a token effort.
And I think that assumes, tell me if I'm getting any of the strategic analysis wrong, but it seems like that assumes that there's just going to be enough demand to keep all the GPUs running hot from other markets such that the power this time is inverted. The companies can say, well, I actually do have an alternative to your market. So, if you're going to make this difficult, we'll just let you know the US and other buyers around the world bid up the GPU prices. We don't really need you to do that.
Yeah. And currently we are in a massive compute crunch, right? And like labs operate in a way that they care more about their future models often than about current revenue, right? So like we don't really know what the compute split is and how it's used, but sort of some numbers that are going around is like one-third on your sort of big run, right? And then one-third on doing sort of experiments and one-third on serving customers. Now it's probably been so crazy with AI agents that maybe like revenue takes a bigger part of the compute pile. But if you think that revenue is only maybe one-third or maybe half of your compute and then European revenue is a percentage of that like then suddenly if you can go faster on the developing future models and moving towards recursive self-improvement by giving up a European market that seems like a rational trade. So that's one side of the coin. The other side is that sort of they can still as long as the playing field is leveled between the American labs and there are no European labs that sort of have some unfair advantages then they can maybe all serve slightly weaker models to the European ecosystem and still get revenue in Europe without actually having to worry too much about them regulating their best models. Right? So like as you say Nathan they could do a token effort and could have some like smaller I don't know compliant model that they're serving specifically for European customers.
Why should European leaders even care? Why not just sit under the US umbrella similar to how in many cases they have sat under the US umbrella for nuclear for nuclear kind of deterrence? You have only I think two European powers the United Kingdom and France with actual nuclear weapons and the other European powers are clearly technically capable and competent. Germany, France, Italy, like they're technically obviously competent and capable but have chosen not to you know acquire those capabilities. So why not just sit under someone else's umbrella so that...
That's obviously the strategy we've been taking sort of for decades now and I think like nuclear weapons and AI are slightly different because yes AI is a very important military technology but above all it just gives you a lot of competitive economic power right so like you can imagine that if AI becomes the sort of dominant source of new scientific discovery or of new like goods and services that are trying to globally compete then there might be a scenario in which the US says it's not like we're still happy to protect Europe maybe militar like in terms of defense and we're still part of NATO but economically we are going to keep the best models for ourselves and so there I think the nuclear analogy breaks down because it like for for using your nuclear power to protect Europe doesn't really come at an economic loss so I think to avoid those kind of scenario or to at least balance things out more such that it's less in US interest to take these kind of measures, right? And it's so that's on like why not sit under the US umbrella. Then what can middle powers effectively do? So the Netherlands for example has ASML which plays a critical role in the supply chain. Taiwan has TSMC. Japan has like important materials for the semiconductor supply chain. Korea has high-end memory. So I think collectively we're actually quite well positioned to play an important role in this AI ecosystem. I do think the middle power coalition therefore is going to be important. There's also a scenario which I think was highlighted in Dario's recent essay where we have some kind of coalition of democratic countries where maybe we sit together with the US right so middle powers plus the US or sort of democratic powers worldwide and where like one of the principles is that we give each other access to frontier technology. So yeah, there are I think scenarios where this middle power coalition could work well and collectively I do think they have some important assets.
From Europe's bind to the center of San Francisco. Monday's longest conversation was with swyx, Shawn Wang of Latent Space who advises Cognition, the maker of the coding agent Devin, and runs the AI Engineer World's Fair. The subject was the practice of AI engineering. What's working, what isn't, and where the value is moving. His day job is measuring AI coders. At Cognition, he helped build a benchmark called Frontier Code. Testing not whether a model can pass a test, but whether it writes code a human would actually merge. We asked what makes it different from the benchmarks that came before.
The reason that we were so excited about Frontier Code is is it, you know, you stop being able to articulate the differences in model quality with more saturated benchmarks like SWE-bench because they're all like at most you'll get like a 1 to 2% bump and they're like cool, but like how much of that is memorization or what have you. Um, Frontier Code is all out of sample. They're not in the training set. They're all graded and heavily rubriced like we basically found like DeepSWE, Frontier Code, sorry, DeepSWE, SWE-bench, all these things they actually allow a lot of false positives in the way that models can cheat in the same way that uh you know during training they they basically have reward hacks same thing and we have an internal catalog of like 20 different ways that models cheat and so basically we just translated that to your rubrics and you know ship ship that as Frontier Code and I think like that that is like how we want to judge models going forward. Um the not just that whether they can pass the test but can they write code that we would merge right? Meter had this very very interesting blog post where they were like about 50% of SWE-bench code that passes the SWE-bench test is completely unmergeable. Like it's it's like so low quality like yeah like technically you'll pass but like you know like just on really stupid benchmarks like did you like modify a whole bunch of files you weren't supposed to touch or like uh did you cheat on the test or did you adhere to like code style what have you? uh just completely unverifiable and so like yeah we we want to guide the evolution of models towards maintainable code and against slop.
Then Pash asked when Frontier Code itself gets saturated.
So, so there's two parts of the strategy. Frontier Code 2026 will be saturated by the end of this year. We, you know, my estimate that you we'll probably hit like 80% by the end of this year. That's as designed, that's expected. Um, it is based on open-source repos, uh, which will eventually get trained on. So, they they just leak. So like you're screwed if you think if you want one benchmark that will never get saturated if you're especially if you're based in open source. So the answer is very simple just do annual cadences. So then we'll have Frontier Code 2027, 2028, all these things and I think every year we'll just move the benchmark move the goalpost from like okay this year it's rubrics for code quality that's like the easiest possible thing next year what is it my candidate right now is security I like we want people to write secure code but every year we can have a defining theme and that will be like the focus of the year very similar to what I do for AIE but like here we we set the agenda through benchmarks and then I think the other thing that I'm very keen on which we haven't like talked that much about but I've talked about it on Hacker News is the private held out set the private evals Cognition has Goldman Sachs and Citi and JPM and all these like large banks uh and and also like the rest of the Fortune 500 not only banks but banks are very big. Um creating evals that reflect the problems that they have but are not solved that are all private is good for them good for business good for the industry. So that's what Cognition is doing. So Frontier Code private uh or Frontier Code Finance, Frontier Code Retail, Frontier Code Telecom, Frontier Code Government, all these are like the the work that remains to be done for building out all these private evals that we can work with the model labs on. Uh but it is the way to communicate or translate industry problems through an agent lab like Cognition into problems that we can guide the model labs on improving the models which I I think is the function of an agent lab.
We raised a worry. If you train AI to write clean, readable, maintainable code, do you cap its ceiling? Keep it from the alien Move 37 style leaps a machine might find on its own.
When you talk to mathematicians about the math solving that and the Lean theorem generation that some of these models have been creating, they are like too detailed. Some like like no human would actually like write that and it's but it's like systematically provable. It is just doesn't contribute to knowledge. It's like well okay like you brute force this thing or something you know this is I don't think I don't think brute force is the right term but let's just kind of run with it. And yes I so I do think that at some point you should just actually step away from looking at lines of code and just say like does this work as I expected? That's great except to the point where other agents also need to read your code that is generated by other agents and they need to work together on it and except to the point where you maybe get into like debuggable like places where you have like very critical code that you're liable to the SEC for or you're liable to like whatever the healthcare authorities are and you're like I'm sorry like I've voded this thing I don't know what's going on in is that an acceptable excuse and probably not for the next 30 years, you know, after 30 years, who knows? But like so, but I I think it's very astute point. I think I do think that for most people, you can get by with like here's a black box. Do whatever you want in the black box, but the inputs of this, the outputs of this, I expect this to to match this kind of quality, and I expect this to have to adhere to some sort of coding standards so that my other agents can sort of maintain this and parallelize this.
Another harness idea getting attention is the adviser or router model. A cheaper model that notices when it's stuck and calls up a smarter one. We asked swyx what he makes of it.
This problem extends back to any problem of model routing. Like you're effectively just doing model routing. You're just giving it a different name. Um so like even like NotDiamond or like what are the other model routing companies out there? Martian and then like the the sort of ETSY lang group at Berkeley also came up with some similar ideas before. The other things I'll also mention is that actually Walden is the one of the co-founders of Cognition had this early on and they called it smart friends they released it in Windsurf not that many people use Windsurf so they don't they don't talk about it I mean still like still like over 100,000 people you know uh uh but like I think the interesting thing about the advisor strategy is that it is probably the right call as the next level upgrade from using the base cheap model, but it will still not have the level of intelligence like like you can still benchmark a lot and I think it can be like very efficient as far as like using the base model the base cheap model as a model router but this basically the dumb model doesn't know what the smart model can do. It just knows the rough shape of what the smart model can do. So like you actually need the smart model to to be able to answer what the smart model can do. And so like you like when you are asked a question that is more complex than you have the intelligence to answer, you don't even know like you're just going to straightforward answer it without knowing that like oh I'm supposed to call out to the smart model and like that's just like the theoretical like limitation.
But in practice I was so so I you know I was I I was thinking about this three years ago and I didn't do this because of this argument that like actually you need smart model first and then you can delegate to dumb model but in practice like for cost reasons for efficiency reasons you actually want to do it the other way around. You want to start with dumb model and then you go smart as a tool call. It just isn't as satisfying. It doesn't solve anything. Are is everyone using it in production? No. Will it be a trend? Maybe for three months, then then you'll get like the next Claude with like more adaptive routing and you're good. Like have one model like uh train it end to end for adaptive thinking and you're you're probably more uh better judged there than uh a systems level thing.
We turn to continual learning, whether AI systems can really learn and remember on the job, not just look things up.
An interesting very fun meta point that I actually double check every time I talk to a Frontier Lab person is what gets published and what doesn't get published. And for the consistent last two to three years, every time people have talked about Google, the overwhelming consensus is that if a paper is good or if an idea is good, it does not get published. So you should have that in mind when you read anything that's published from Google, right? Okay. So uh with regards to continual learning and memory uh so there there this is a choice like I almost did not do this track because there's a big split between the models guys and the systems guys the big split is do you update your model weights or do you not right is this a glorified RAG in another format uh where you you store things in a database and you look them up is that memory is that continual learning yeah actually like when I do a thing and I write a skill and then I do the thing again and it calls the skill. It does learn but it's not machine learning. It's it's zero gradient if you want to be like really fancy but it is not machine learning. It is in context learning for sure. Anyway, so the the point being like okay what are we talking about when we have like a community come together and do continual learning and the the sort of more machine learning side of the spectrum would be like well we will update model weights and the less less machine learning side will be the other thing. I think this is all comes down to how controllable and interpretable you want your memory to be, right? You're going to recall bad facts. You're going to not be you're going to want to forget things and can you control that? Obviously, the maximum control is you you don't update model weights and you just control what gets into the system so you can delete and monitor and debug. But uh for full internalization in the model of the things that were learned, you probably do have to train on it. And so that is a whole other discipline that Trajectory AI, Engram, all these speakers that are uh and adaptation labs these all speakers at the conference. The first half is the people that update model weights and the second half is the people are system the more systems people. And I'm not making I'm not choosing a side here. I'm just making the observation that these guys don't like each other. Like basically the, you know, the the model people don't view the systems people as legit. And systems people are like, well, the model people, you know, have fun training your model, but you're never ever going to have an a sort of a memory system that you understand because you're just updating weights. And so it's just continue pre-training or whatever. And I think that's fair. That's a fair discussion.
That split, update the model's weights or keep memory in a system you can inspect, gets a practical answer once you ask what enterprises actually want.
Quite simply, it just takes one security incident where you leak information that you weren't supposed to leak because you trained on customer data or my information was somehow exposed to my teammates's information even though if we work in the same team I'm like wait hold on like like I am not giving any of this to to a model. Uh so yeah I mean uh I think enterprises want cheap and perfect and private let's call it right those are the three things and so unfortunately that skews towards the system sites today uh but you know that's not to block like companies like Engram and Trajectory from doing uh very good POCs with some of the large enterprises I think it's still at the POC stage but you know that even at this level like POC is like a few million dollars which is great. I think that the question is like can this can they figure it out before like the money runs out or patience runs out you know I think that's that's what all startups and what R&D is for is just like figuring this out on production traffic. I mean the beauty of this is that you can just kind of run your traditional harness system that is fully you know it's like old school implemented with like basically RAG on on some memories and then you can you can run a shadow system that that has the online thing and you can just compare an AB test. So I I think all those problems are solvable. Uh they are open research questions. We are unlikely to get any papers about this because it is so valuable as a as a problem for all startups and we they they may all get steamrolled by the next architecture. Right? If like we're only doing this because context length is the slowest Moore's law in the industry. Like we've basically gone from like a thousand token context length order magnitude to a million uh in three years which is actually kind of slow as far as like everything else is concerned and I don't think we're going to see like 100 million 100 trillion you know that kind like it doesn't scale like like the other stuff scales and so I I I do think like if something comes along state space models become a lot more effective or whatever I would say that then like you you need to update weights like you can't rely on infinite context because we don't have infinite context because we don't have infinite memory and so so I I do think like we we do need to figure out the systems there in the meantime people are just stitching systems together as as they should.
From the models to the plumbing, Pash asked whether the internet's own infrastructure, GitHub, the cloud, can take the load as agents become most of the traffic.
My worry is actually that people start to wall off parts of the internet. Um so you get like not only like a dead internet but also like closed gardens closed walls internet wall garden internets where like you know in China you have like I live in like the Baidu universe and then the other your other friend go like I live in the Tencent universe the BAT is just like carve up China and like I don't think that's what we want. We want like a fully open and interoperable web and so like you know Cloudflare going like we'll just we'll just ban some agents that are not part of Cloudflare but then if you're within Cloudflare you're fine and I'm just like you know I don't know if that's like the uh the sort of ethos of the open internet that we want um and then the same by the way I'm not picking on Cloudflare like the same will happen to like OpenAI's and Vercel's and what have you uh so I I do think like we do need to scale I do think what Graphite and Cursor Origin are doing is fantastic. Uh needed to be done and like I would just observe there's a structural mismatch here. Like it's just unfair to GitHub that like anyone good at GitHub who like would be able to solve the problem can solve it at Microsoft and you can get like a really good like level I don't know 50 whatever pay of like I don't know 300k or you can go to Cursor and effectively get 5 million a year doing the same thing but like you're you're the cool sexy one and like it's very interesting right when does that switch from like oh a startup can never beat an incumbent to the startup is actually preferred to the incumbent and this And GitHub has somehow gotten itself there. The degree of scaling, right? Like we we talk about like okay, GitHub is 14x uh in terms of like number of commits, but also like 10x and every other dimension of parallelism or uh CI/CD and all those things, right? Like like it just induces so much infrastructure demand that is not human, it's probably wasteful that we're not like really set up for, but we we're just paying for it anyway, which is good because at least the economics are worked out. But like uh in terms of like number of CPUs in the world that we in our cloud that we need we're running to real shortages of everything not just GPUs but also CPU and memory and you know the sandbox companies like the E2Bs and Daytons of the world they're growing at least 50% month on month every month for the last like year and or 18 months or so. And like it's it's stupid amount of like just slop that's just like kind of spewing out from agents and like I get so much uh you know I run a large newsletter and every time I send it out you know I get so much replies from claws reading the mails and trying to reply to me and I like I know there's no human on the other side right but I have to like manually go and block these claws from replying because like this is just clogging up my inbox. So then I need an agent on my inbox to read their agents emails. And so this is just all like a huge giant recursive and like I stop I stop reading emails, right? Like it's terrible. I don't know what to do about it. I just think like you know when you talk about scaling infrastructure like that's a very human story that is really happening and we haven't give given these agents money yet. Imagine what happens when they have they all have wallets. They all have stable coins and they're all buying and selling and we don't know what the hell is going on.
That scale is pushing companies to rebuild their own tools from scratch. We asked whether they're pulling their systems of record back in-house away from SaaS.
The company should have a sovereign system of record not the individual SaaS like because like ultimately like a lot of the SaaS economy is built on we will sit on this be we will be a system of record for this kind of data like your meeting notes your um your calendar your email whatever and like we then and then now with the AI age they just slap a chatbot on top of it right like we'll we'll do the like little sidebar thing that like answers all your questions like what would you like to do today and like Yeah, like you know, Mercury just shipped a a thing on Mercury stuff, right? But like that's not integrated with anything else I do that has no memory of anything else that I prefer that I I have to set up all my skills again. I probably can't even set up my skills because there's no way to import that. So that's all that is Like what OpenClaw has is like this is my personal agent and like all the data is synced to me and then I'll decide what to do with it. Thank you very much. Right? So, how many systems of record should there be versus like, you know, like how many chap how many $20 a month subscriptions are you going to need to pay? And I think that is a big reckoning for a lot of people. What's really interesting or innovative is that you would expect the largest companies built on being the entire system of record to be most protective of their data. But Salesforce and Marc Benioff is like out there saying like everything will be available by API. You can just you can just take it. And I think that's very forward thinking. It does mean that he's going to have to change his business model somewhat, but it's like either he does that or the Salesforce killer does that, right? And so he's just heading off the uh the Salesforce issue. And I think more people should probably think about what the business model looks like beyond just sitting on your data and being uh and then slapping chatbot on it.
The last stretch is about the systems, the science, and the buildout underneath all of it. First, Bing Xu, co-creator of MXNet, now out of stealth, building self-improving infrastructure. His company runs a swarm of AI agents that write PTX, the lowest level code that tells an NVIDIA GPU what to do, and rewrites it to run faster generation after generation. His headline claim cuts against the consensus. And Nathan put it to him directly.
