Currents 038: Connor Leahy on Artificial Intelligence

Oh, thanks so much for having me back.

So it's technically not a GPT-Neo model. It's kind of a side project done by one of our Eleuther members, Ben Wang, with help from Aaron Katsutaki. I'm sorry if I pronounced your name wrong, Aaron. It's called GPT-J. The name comes from because it uses a different codebase. It uses a codebase based on the new Google framework, JAX, and is trained on TPUs. It is currently probably the best open-source natural language model. It is at six billion parameters, 6.1 or something, I'm not exactly sure. It was trained on TPUs using a mesh JAX transformers library or whatever, whatever it's called that Ben made. He did almost all the work himself, so all the credit goes to Ben on this one. He's a really crazy guy, smart guy, did a great job. And we trained a six billion parameter model with a slightly different architecture to what we do with Neo, so it's more modern. Neo is kind of closer to like old-school GPT-2 and 3. J includes some new tweaks, it changes the architecture to make it more efficient. It was also trained for longer. It was trained for 400 billion tokens instead of 300 billion. And with that, it is now pretty much on par with like the second-largest GPT-3 model, the GPT-3 Curie model that OpenAI offers. We ran them both through benchmarks and they perform very similarly. Of course, there's some differences. J is very good at code, technical things like medical information. It was trained on like medical papers, a lot of scientific papers, very good at that kind of stuff, very good at code. Slightly less good maybe than GPT-3 at like storytelling and stuff, but still very comparable.

Yeah, so Aleph Alpha is very much a European company. So we're based in Germany and we're proud of being a German company and being like a European company. And so basically the way we kind of see it is that like the US market has a very healthy, vibrant AI ecosystem. Like it's no secret that almost all of the, you know, high-tech applications you might use every day are usually American-made. There's also a pretty large sector in Asia, especially in China, who have like their own like parallel market and such. But it's almost a cliche to mention it, is that Europe has kind of fallen behind in this regard. Is that there are very few like large, you know, really successful tech companies in Europe. And we find this to be a shame. We think Europe is pretty good. Like, you know, there's problems with it, of course. We have problems with, you know, Germany and Europe, but we quite like it here. And we would like to create a better, you know, AI and tech ecosystem here in Europe. It's not always been easy raising investments in Europe and in Germany is much harder than say in Silicon Valley. But we want to ensure that, you know, also that the European market also has access to, you know, really good, really high-quality AI technologies. In our, you know, like in our perfect vision, we would want, you know, a great graduate student from Germany could come to like Aleph Alpha, it would be as competitive or like as prestigious, you know, maybe as going to like a company in Silicon Valley or something like that. Of course, the advantage of companies like Facebook and Google is that they have obvious, uh, monetizable applications for earlier AI.

So Aleph Alpha in a sense kind of is, I would describe it like most closely to, uh, at least in our ambitions, to a European OpenAI. But let's say, uh, more upfront about commercial interests in that regard. So yeah, to be clear, you know, Eleuther and Aleph are separate entities and I try to keep them separate in that regard. But Aleph is a for-profit company. The goal is to build large generalized models, uh, GPT-type models, some other things that we've been working on, and offer them to customers to perform various useful services. We basically are very strong believers and have been for a while now that this kind of technology is just on the cusp of being extremely economically important. And we want to be there. We want to have everything in place. We want to be established when that takeoff really happens, which we think is basically currently happening. So our main project is we're working on GPT-type-ish technology. I don't want to go into too many details about, you know, what we're working on behind the doors, but, uh, basically, yeah, we want to offer AGI or however you want to call it, like these types of services in Europe also. So we also have a pretty big focus on multilingual data. We don't want to have just any English models, models that can be used hopefully by lots of people all over Europe speaking different languages. And yeah, lots of projects. We're still pretty young. It's still pretty early startup, so we don't really have any like huge customer-facing things, but stay tuned.

We are looking at other things, but there's nothing I can talk about publicly yet that we... later we're ready to show. But yeah, most of all we're first looking into language modeling. Um, there are other things we're very interested in, but I can't, you know, commit to anything.

So this is a pretty common discussion I have very often. And, you know, opinions shift pretty quickly as new information, uh, gets out. But basically, I think it's kind of a null hypothesis that multi-modal will work for like, maybe not AGI, let's say like transformative AI. It's like, I think like a better word because that's like more defined by how much economic value these things can produce. I think it's pretty obvious that, you know, humans, you have eyes and ears and skin for touch or whatever, and that's enough to train a pretty powerful intelligence system. So I think it's like a null hypothesis to say that, you know, with enough image data and or video data and et cetera data, we should be able to train something that is as intelligent as a human. I think there is the other hypothesis that maybe language is enough in like a non-trivial way. I consider this to be a hypothesis that I think is more plausible than people give it credit for, in my opinion. So I'm not saying that I'm sure this is true. I'm just saying the more I deal with these like language models and the more I see like the scalings and such, the more I see like, hey, there might be, it might actually be possible. But that's a hypothesis now.

Uh, so that was basically kind of a cheeky meme. The original paper introducing transformers is called 'Attention Is All You Need', so it's kind of like a, almost a, um, it's a common thing to riff on that for titles. And basically, one of the core things that I found like in the early days of transformers, when people talked about, oh, you need multi-modality, you need video or whatever, and people would say, look, transformers, they only work for text. But kind of when I said 'Attention Is All You Need Is All We Needed', what I mean by that is we now know we have stuff like vision transformers and like 100 different variants of, you know, different like audio and spatial and whatever transformers, is that these same architectures can pretty easily be generalized to other modalities. So whether or not text is enough, it seems like pretty plausible that attention slash transformers might be still enough, or the right architecture, or a good architecture, let me say it that way, for solving these kinds of problems across multiple modalities such as video and image.

So AI risk is a very interesting topic. Is that people react to it, from my experience, kind of like one of two ways. Either you introduce it to someone and they're just immediately, oh yeah, obviously, like how could it be any other way? That's like the most obvious thing in the world, why are you even telling me this? And the other half of people get really irate and they're like, what are you talking about? This is all nonsense, it makes no sense, you're, you know, and then a bunch of like strawman arguments about what I believe or don't believe. This is a bit of a straw man, of course. I appreciate the irony there. But yeah, it seems pretty obvious to you. So like, there's lots of very, very good, very comprehensive write-ups, you know, justifying why I respect this sense. But like, a very simple intuition is just intelligence, in my definition, is kind of the ability to solve problems. It's the ability to take actions to achieve goals. And it's pretty obvious to me that if we create systems that are capable, very, very, very capable of solving problems, of achieving goals, if we even very slightly misspecify what we want these things to do, there's no limit to what these, I mean, there is limits, of course, physical limits, but like, these things may be capable of doing extremely complex, extremely powerful actions in order to achieve those goals. We already see this with, you know, comparatively very simple systems like, you know, with like, uh, economic regulation or something. You want companies to not do X, but they find some loophole in the tax code that allows them to do that, and suddenly everyone is, you know, doing something that we don't want them to do. Or another example is, uh, this comes from a guy on Twitter, Roko, who describes the food maximizer. He describes in like the 19th century we summoned a weak superintelligence that he calls the food optimizer in order to feed all the humans, and it's gotten so good at that that like 60 percent of the Western world is overweight or something at this point. That we have a goal, we, you know, we incentivize the system to produce food, food that we want to eat, but we don't incentivize the system necessarily to give us food that is healthy for us to eat. So instead, the system figured out that giving us food that tastes really, really good but is not good for us is a really efficient way for it to, you know, maximize profit. So even though we created the system in order to help us, and it did, you know, because lots of food, it also produced lots of very non-nutritious food or very, you know, sugary food because that's a way to hack human motivation. And so it seems to me pretty obvious that if we create systems that are even far more intelligent than that, it's very hard to know what these systems might be capable of and predicting what they will do ahead of time.

Yeah, absolutely, those are real considerations. So here's like one of my favorite examples. A good friend of mine, Stella Biederman, always brings up this example, is that Google Maps a while ago changed a little bit how they do their recommendation algorithm, and what it caused is that it suddenly routed a lot of traffic through this quiet neighborhood in LA, which was previously like off a highway. And because they tweaked the algorithm, the algorithm said, oh, the highway is really full of traffic, take this alternative route. And this caused this whole little neighborhood to suddenly have a very large amount of traffic. And there's a very obvious question is like, this was obviously intended by Google or something that didn't decide to do this, but this was a real harm caused to real people. And is there some kind of liability here? How could you predict things like this? I think there are many, many examples of how even like very primitive, like I wouldn't, I wouldn't consider, you know, Google Maps to be an AGI, obviously, you know, it's a pretty simple algorithm all things considered. But even these pretty simple algorithms, like the Tinder dating app, you know, algorithms, whatever, all pretty simple, even these are already basically unaligned. They're out of control. We don't necessarily know what they do and we don't necessarily know how to control them. So let's say we find out that, I don't know, like a, you know, a recommended app on YouTube does something we don't want it to do. Often we can't know that until it happens, and often we have to like, you know, just kind of like guess how to fix that. It's not obvious necessarily how to patch such an algorithm. It's not like it's necessarily code, especially if it's a machine learning model which has this big black box of a bunch of numbers, and somehow we're supposed to patch that to, you know, fix a certain problem. So to bring this back to AGI alignment, so I'm saying we can't even get these teeny little, you know, super, super subhuman algorithms to do what we want them to do. Why do we think we'll be able to control human or superhuman algorithms? That seems kind of arrogant to me.

Not necessarily, but it turns out it's like this is something that like some of my friends work on and that I'm also very interested in, is that defining the concept of agency is really, really hard. And often it like pops up in like situations where you would not expect it to pop up. A classic example is an essay by Guern, 'Why Would AIs Want to Be Agent AIs?' And he explains that like, assume we had like an oracle AI, the only thing it does is answer questions, right? You just type in a question and it outputs an answer. That doesn't seem like it would be an agent, right? It doesn't seem like this thing could destroy the world. And, you know, maybe that is safer, but there's also a story you can tell about how such an agent could still be very dangerous. For example, if the agent is incentivized to give correct answers, so you give it like reward when the answers are correct, you give it like negative reward when your answers aren't correct. Well, maybe the agent will then strategically start giving you answers that will change your behavior to make the world easier to predict. Like maybe it finds out, oh, if I advise these people to start nuclear war and they do it, then it's really easy to predict what's going to happen next. Everything's just going to be dead. That's really easy. So there's all these like really weird edge cases you can get into.

So that's absolutely a threat vector. That is one that a lot of people take very seriously. I personally like to kind of push back a little bit against that narrative. I think it is absolutely a threat vector. This is a way that harm can and will and is being done. Modern deepfake technology is very, very good. I think people still don't understand like if you have like just like 10 machine learning engineers and, you know, a refrigerator full of Red Bull, they can make some really good deepfakes nowadays. And I think this is something that a lot of people have not yet reckoned with, like just how quickly these things are going. And in a sense, this is kind of like the zone of the crux of the argument here is that, uh, you earlier mentioned a tweet from Eliezer that I retweeted. And Eliezer is one of the few people who kind of thinks this is not the biggest threat vector. And I actually agree with him on this. Like, we really want to explain why I think this. I think what you describe only will happen if superhuman AGI doesn't happen very soon, basically. I expect the way I look at AI technology, it's accelerating so fast, technology is going ahead so quickly, that by the time quote-unquote bad guys have like figured out how to use this technology at scale, there's already so much more powerful technology they can use that it's just not going to matter. So what Eliezer was talking about is that if AI technology happens fast enough, if there's, you know, we don't have like a long, you know, time of like, you know, things going wrong, us fixing it, you know, something else going wrong, fixing it, but if we have technology that is so powerful they can just irreversibly, you know, destroy or harm or control everything in one go, then our current systems are based on error-based learning. You know, it's like currently deepfakes don't get regulated until, you know, some politician gets deepfaked and then suddenly gets regulated. Like that's usually how these things go. And this is a very common human thing. And that works fine if you have like ergodic assumptions, if you can try again. But the fear that I have, I'm very concerned about like long-term future X-risk type situations. It's like, don't get me wrong, all these things are real risks. Real people are and will be harmed by these technologies by bad actors. But in a way, it feels parochial if you compare it to the threat of, you know, all humans going extinct or something like that. And if these very powerful technologies emerge soon and we don't have enough understanding of how to build them or use them safely, and we don't have the time to experiment with them without things going horribly wrong, I think we're in deep [__], really, really deep [__]. In the sense that maybe these deepfakes are, you know, causing some harm here, and maybe GPT-3, you know, propaganda messes over up over here, but then if a metaphorical paperclip maximizer emerges and paperclips everybody, well, you know, what matter, right?

So yeah, I really like the way you framed that there. I fully agree that if you accept like 100-year timeline, then yes, you should be worried about these threats more. But my timelines are more on the order of five to 15 years. So I have very short timelines. Just the amount that AI has progressed in the last two years just blows me away. It's unbelievable. And it doesn't seem to be slowing down. For the first time in my career as an AI researcher, I feel I see like a direct path to how an AGI could be constructed. Like, not that I could like do it myself right now or anything, but like there's no 'and here happens magic' in the equation. It's like I see all the parts you need and I see like at least potential solutions to each of those parts, and I don't see anyone where I have to say, uh, magic, put insert magic here. That was not the case like five years ago. Like five years ago there was many places where I would say, insert magic here because I don't know what to think about it. But for the first time, it seems to me that we're racing towards actual designs that could really become extraordinarily powerful. And I also like to just like make clear again, like it doesn't really matter if the agent has like general intelligence, okay, you know, can it climb a tree, doesn't really matter. What matters to me is, you know, can it cause irreversible harm? Like how much power can these agents have? And quick tangent here, I think important to mention is the concept of instrumental convergence. So this is like a pretty important crux of my argument is that there is a pretty, I think intuitive, there's also more formal versions of this argument, like Alex Turner has done some good work on this, but basically, you can in many, many scenarios, so for many possible goals, gaining power is a really useful thing to do. And another useful thing is staying alive. So a common example is what people that, uh, dismiss AI risk, they say, well, we'll just turn it off if it does something bad. Well, here's a simple thought experiment. Imagine you have a robot, an AGI robot, and you use it to get you coffee. So its only goal is to get coffee, right? So immediately it, you know, bursts through the wall, you know, it runs over your cat, you know, it destroys everything in its way to get to the coffee machine as quickly as possible. Like, no, no, I don't want you to do that. So you run over to hit the off button on the robot. And what will the robot do? It will stop you. Because you never gave it a will to survive or consciousness or anything like that, nothing like that. You just gave it the will to make coffee. And but the thing is, the robot will correctly reason, if I'm turned off, I can't bring you coffee. So therefore, it will actively resist you shutting it down, because then it won't be able to make your coffee. So even this extremely simple goal of making coffee can already lead to an agent that will resist being shut off. And for like more complex or powerful goals, that's the instrumental convergence hypothesis. These goals of not wanting to be shut off and also like gaining power, whatever power means, you know, it might be economic power, social power, or computation power, those are just very useful things. So we should expect most agents with most goals to by default, unless we somehow stop them from doing this, by default to pursue such dangerous objectives.

Probably not, no. Because like, let me put it this way, how would you turn that into code? Like, how would you turn those three laws into actual code run by an actual agent? Trust me, people have tried it. It's not easy. And you get into like all these weird paradoxes. Like, so, no action that could cause a human to come to harm. Well, any action could cause a human to come to harm. So any agent with that law would immediately shut itself down because any action it takes might cause human harm. So okay, now you have to have put in some kind of realization, now you have to give it like some kind of uncertainty prior.

Yeah, but then you get into the problem, which prior do you use? You know, how conservative or not conservative? What counts as a human? Is, you know, if someone is brain dead, are they still a human? If you have a simulation on a computer, is that a human? You know, again, it's tough.

I find it's difficult to think about these kinds of questions. So like two categories that people like to talk about nowadays is like the, uh, Eliezer-style fast takeoff is what people call it, where you go from like zero to, uh, you know, 100 billion trillion in like a very, very short amount of time, just like no warning almost, it just happens very, very subtly. So like the most extreme example is to kind of like, you know, someone lets their network run overnight and the next day suddenly it's God, you know, kind of like situation. Of course, something that extreme is silly. I don't expect things to go that fast. So an alternative, which is kind of interesting, is kind of like, it's kind of generally attributed to Paul Christiano. It's this concept of a slow takeoff, but ironically, the slow takeoff feels faster than the fast takeoff.

Yeah, yeah, yeah, let me unpack that for you. So in a fast takeoff, you kind of have like...

A flat line, just zero, zero, zero, and then it goes like really high. While a slow takeoff is more like a hyperbolic takeoff. Basically, what the thesis of the slow takeoff is, is that we're going to have a four-year economic doubling time before we have a one-year economic doubling time. So it would still be extremely fast, and it would feel faster to people because people could see the four-year doubling time and then the one-year doubling time before we hit the singularity. And this is, I think, more people nowadays kind of imagine situations like that. It seems also nowadays we very much have these multi-stakeholder takeoff type scenarios. We have not just, you know, one AI made in one lab by one person, which is kind of where that old-school Eliezer type thinking was. Nowadays it's more like these AI systems will be very complex, very expensive systems that are built by large organizations. And there's lots of complicated thinking about what does that mean, what is safer, what does not. So I guess I'm a bit ambivalent on this. It doesn't matter really exactly how the takeoff goes. What matters is these agents are going to appear very soon, they're going to be very powerful, and if we don't align them, we're [__]. That's kind of what I focus on.

Yeah, I fully agree. I can understand people that say, oh, maybe AI is going to take longer, or maybe we're going to run into robots. But people who say it's impossible makes no sense to me. As you just described, even these very minor changes to just a human-level agent would already make it so much vastly more powerful than a human that, you know, who knows what the limit is there. It seems very obvious to me.

Yeah, I agree, that's a good way of phrasing it. I think you're correct that the reason a lot of the early people thought differently than we do nowadays is because they thought about AI algorithms differently. They thought that there was a special math, and if you figured out this special math formula, you would get huge improvements. It's possible that such formulas exist somewhere out in math space or something, but it seems to me that at the moment, at least, the way things look is that it's more that you have kind of like computational irreducibility. It's just to get a certain level of intelligence, no matter how clever your algorithm, you still need a lot of compute, you still need a lot of data to locate the right hypothesis in your hypothesis space or whatever. I'm not saying that our algorithms currently are by any sense the limit of what might be possible. The thing that I just always try to remind myself is the space of all possible programs is one of the weirdest eldritch horror escapes imaginable. It's unbelievable, it's unknowable what things exist in the space of all possible programs. So it's best not to reason about that too much.

Maybe. I would add to that that a solution to the alignment problem is also a solution to bad guys with AI. If we had a super nice AI that knows what the right thing is for humans and robustly can follow that, and we build such AIs, and we just, you know, either destroy or make it illegal to build other kinds, that also solves that problem. It's a more general solution. And also another thing I'd like to raise is just kind of like, I think I like to say about AI alignment and safety and such, is that if you work in this field, you have to be comfortable multiplying really, really big outcomes by really small probabilities. So that's like one of the arguments about MIRI, even like the early days, they always were clear that the chance that they're at the right time or they're doing the right thing so early in a development is pretty small, but the potential outcome that maybe if they get something useful out of it is so large that it could still be worth it. So it's kind of like a risk-benefit trade-off. In that, I think working on these short-term AI problems gives you a more assured outcome, like you're more likely to do something that will have a net good. But I expect that net good, the magnitude of the net good, to be so much smaller than the possible net good of doing something very risky with long-term AI that, at least for me personally, of course, you know, all these priors come from, you pull them out of your ass at some point, is that it feels to me that working on these very risky, low probability of working but extremely high potential payout things is a very good investment. But you know, that depends on one's personal risk tolerance.

Yeah, I must also say I do look up to Eliezer a lot. He was a very great inspiration for all the work I've done. His work on the sequences is probably the number one most influential work on my personal thinking. So I'm also extraordinarily glad that someone like him exists.

For blockchain, as far as I know, I agree that I think the amount of damage done by deep fakes so far has been relatively small. But I don't think that it's because the blockchain has been robust. I talked about this in the second part of the essay. I remember, by the way, I just want to flag for any readers, I apologize for those essays being strange. These were some of my first attempts at writing long-form.

All right, well thank you, I do appreciate it. But yeah, so the second essay I talk about the truly stupendous uncreativity of evil people. I know Bruce Schneier also talks about this, like the concept of ordinary paranoia versus the security mindset. He gives this funny example. So when he was a kid, he had an ant farm, and he has a card in there where you could send an address, like a letter with an address to this location, and they'd send you a bunch of ants. And that ordinary person would think, wow, cool, I can get some ants. And someone with a security mindset will think, huh, I can send ants to anyone I want. Interesting. And it is kind of like a specific kind of style of thinking. Bruce Schneier is like one of the best writers on this topic. Eliezer has also written an essay about this, I forgot what it was called unfortunately. And it's kind of thinking that I talked about this in like the second essay of how if you are just like a little bit creative, like I am by far not the best security mindset genius hacker or anything like that, but just with a little bit of creativity, I could come up with some really, really dangerous possible attacks that I could pull off for like a hundred thousand dollars and might ruin a politician's career or something with high probability. And I could do that for a hundred thousand dollars in my bedroom. But somehow no one has done that, and that's kind of what I wrestle with in one of the essays, like why has no one done these things?

Yeah, and that's actually a big reason of why I take AI risk so seriously. Because AI, even if it's not superhuman, let's say it's just as smart as a smart human, it can still be functionally perfectly sociopathic. You can still create an AI that perfectly, consistently lies, manipulates, controls. It can run extremely complicated networks of lies and sock puppets to such a degree that it will never slip up even once. And this is not something that we are prepared to deal with. Even now, from my experience, one sociopath in your company can bring down the entire company. I've seen it happen.

Yeah, so the idea, I'm not sure if I still think that's the best name for it or not, but basically the idea is that in an ancestral environment, if you want to build trust, say there is an idea and you can't evaluate this idea, you're not sure if it's good or it's bad. And if you then see many people around you that you trust, from your tribe, your elders, your family, your friends, and they all say this idea is really good, that's pretty good evidence that it probably is true. Because anyone who tried to convince these people had to convince them, and that takes effort. And if it's a bad idea, you hope it takes more effort to convince lots of people. You know the saying that you can't fool all the people all the time. So in a sense, you have this kind of trust mechanism where seeing lots of people that you trust endorsing an idea can alleviate your desire necessarily to check the idea yourself, or it might even be an idea that you yourself can't validate. So it's kind of like our truth-making mechanism, one of our basic ways of making truth. And the problem is that this mechanism evolved in an environment where there weren't tech bots and deep fakes and organized propaganda campaigns pushing anti-vax or whatever, or Facebook algorithms optimized on sucking you in.

Interesting. So you then talk about security mindsets and the fact that all kinds of potentials for harm are out there. But then the one thing I found most interesting and caused me to think a whole lot is what might we do as humans? We can't evolve very fast biologically.

Yeah, so I would not consider this to be a full solution to anything, but I feel like it's such an obvious thing. It's also not unique to me, I'm not the first one to come up with this idea, obviously. But these ideas of using cryptography to replace some of our more informal methods by more formally powerful methods of verifying information. At some point, you have to trust people. Bruce Schneier has wonderful essays about this, about how there's no such thing as trustless technology. That does not exist. At some point, you always have to trust the computer code or the algorithm, or you have to trust the people who made the program, you have to trust the people who built your iPhone. There's always trust. You can never verify anything in a way. Trust is the most powerful skill that humans developed. The fact that we trust other people is what allows our civilization to exist. Remember, chimpanzees don't trust each other usually. I mean, that's actually nowadays considered not 100% true, but just to take the stereotype, what's clear about chimpanzees is they absolutely do not trust any chimp that's not from their band.

Exactly. All the technology I'm currently using, I can't verify what this technology is or how it works. I've never met Jim in person. He's not part of my tribe. How did I know that I should trust this email that appeared in my inbox, that I should click on this link and then go talk to this guy? How do I know? And so trust is everywhere in our society, and it is a fundamental social technology, a fundamental tool and social technology that we need for a complex society to function. Cryptography is not a solution to privacy, it doesn't necessarily make it, it's just a tool to allow us to do certain very powerful forms of privacy-enhancing technologies. The obvious ones, like encrypting your chat messages or whatever. What I'm even more interested in, and what I talked about in that essay, is this concept of a web of trust and using public key cryptography to verify and sign messages. So one of the things you can do with public key cryptography is that you can have a secret number, a key, and you're not allowed to show anyone else this key. And using this key, you can create a signature on messages that 100% guarantees you and only you wrote this message and the message was not tampered with. So what I think should happen is that this kind of technology should exist everywhere. Every tweet you send, every text message you send, every video you make should be signed, timestamped, identity-stamped. Of course, there could still be lawless places on the internet where this is not enforced, but I find it surprising that I can just go on the internet and look at a tweet and I have no guarantee who this came from or what entity wrote this and who endorses it. So the way I think trust, this kind of formalization of the weak concept that the biological blockchain was trying to implement, this idea that I could explicitly endorse people. I could say, okay, I trust this government organization but not this one. I trust this newspaper but not this one. I trust my best friend but I don't really trust my other friend because he's kind of an idiot. And then I can see who endorses what, who says this is true or who has comments, and I can verify that in an unfakeable way. So if I see thousands of people endorsing a claim, I can check, are these signed endorsements? Where does the signature come from? Who endorses these people? Is there a root node? For example, you might have a government that endorses special citizen keys that can be used for voting or for comment giving. For example, there was a great, the FTC of a few years ago had an open hearing about net neutrality, and it turned out that the telecommunications companies hired grassroots-as-a-service companies, which is a real thing, which is legal by the way, somehow this is legal, that then created thousands and thousands of fake users and fake comments in order to push them towards one possible policy. And the way you think about it, it's kind of crazy, is that you're allowing people to anonymously basically submit things in the voice of American citizens without checking. This wasn't necessarily the case when the biological blockchain was still in full force back in the day. When a town hall meeting was called, you had to physically appear in the town hall to physically tell people what your opinion was. That verified your identity. And online, that's not necessarily the case. And these keys and these endorsements by different root trust sources could be a way of allowing this kind of authentication to happen online.

Yeah, absolutely. And my answer to that is that yes, that's completely the case. I am fully aware of how politically unviable this solution would be, how complicated it would be to actually implement this system. Our institutions do not have the kind of executive capacity to be able to organize something like this in any feasible timeframe, I think. But my argument is that I think that the difficulty does not come from private keys themselves. I think it's more fundamental than that. It's just authentication is hard. Authentication is a fundamentally, irreducibly hard problem. The same way that people complain about proof of work with Bitcoin or whatever, and yeah, sure, I understand that there's a huge waste of energy and whatever, it has all these bad points, but it is an irreducibly hard problem. You can't just have Bitcoin without proof of work. You have proof of stake, but it has other problems and other trade-offs. It is an irreducibly hard problem. Trust has to be hard, because if it was easy, it also makes it easy to break. There has to be one step somewhere that is hard. That's just how it works.

Yeah, absolutely. In practice, you would have the less secure keys, maybe, and you keep those on your hard drive, and those are linked to your shitpost Twitter account or something. If you lose that, it's out there. And you have your secret government-issued key to keep on paper somewhere secret, and you only bring it out when you're voting on something or doing something that requires very high levels of trust. I'm a big privacy advocate. I think if anything, I find it sad there's less super anonymous places on the net. But I feel like there are benefits and downsides to having very anonymous places, and there are benefits and downsides to having very not anonymous places. And I think these should exist side by side. It shouldn't be everyone's authenticated everywhere, neither should it be everyone's anonymous everywhere. I think it really depends on what the use case is.

So this is a question I've been thinking about for a long time, and I still think about it all the time, and my thoughts have evolved since that essay, but I think we don't have time to get into that right now. But there is this real question, is that humans, we humans have a natural idea of a person, an identity, a singular, you know, we have citizens and they're discrete entities. We don't have continuous citizens, we don't have 0.7 people are in favor of this or something, at least outside of statistics. And the thing is, I think that is not a fundamental property of the universe. It is an emergent phenomenon that happens to be the case. It happens to be a useful abstraction because humans tend to come in one human-sized chunks and they're not easy to reproduce. But if we say had a human brain scan and I could just control-C, control-V that brain scan lots and lots of times, do each of the copies get a vote? Are they citizens? What rights do these things have? What responsibilities do I have towards these entities? And this is just one of the many, many problems we get once we start breaking down these comfortable assumptions that do hold for biological humans but don't necessarily hold once we really start building virtual entities. And one of the things I talked about in that essay is, yeah, so at some point you have to count. If you want to have a vote, you have to count. It has to be an unambiguous way of counting how many people are voting, how will our votes be tallied. That's just how voting works. And then on the more provocative side, I argue about, well, maybe not for all cases, counting humans is the right thing to be counting. Maybe there are situations where you want virtual entities to be voting or to be part of your community, or maybe these things get really complicated very quickly. And I don't propose solutions or obvious things like that. It's more like food for thought. This will happen sooner or later. Sooner or later, we will have entities sharing the planet with us that I think have a very real claim to moral patienthood, a very real claim to say, hey, I'm intelligent, I have goals and desires, I think I should get some of the things these humans get. At some point, such entities will exist. I think that is very likely. Whether those could be human uploads or AGI systems in the future, and that's going to break a lot of the assumptions we use to run our society. And I really do think we need to take these things very seriously, and we'll have to rethink a lot of the fundamental principles of how we run society.

Yeah, it was a blast.