Fireside chat with Mark Miller @ Vision Weekend US 2024

They were published in '88, but we started the work in '83.

Okay, so that's a lot. One of the things that provoked me into starting the work with Allison and Christine that led to writing the book was reading Bostrom's 'Superintelligence' book. First of all, just give a lot of credit to Bostrom for having gotten a lot of these conversations going. But there were two things that stood out to me in reading that. I'm not going to claim to speak for them, and I'll exaggerate for clarity because I'm talking about my reaction rather than necessarily what's in their point of view. On the one hand, they talk about a super intelligent singleton emerging that, because it is super intelligent and emerges first, has complete power over the world, essentially a unitary permanent military takeover, a permanent dictator over all of us that will last forever. Then having set that up and spending a lot of effort on why all sorts of other alternatives are not likely, they then advocate that wise philosophers work on the problem of how to encode a utility function such that this unitary benevolent dictator is a beneficial dictator that serves our interests. On the one hand, I felt like this was a very totalitarian view of the future that, if pursued, would go very badly. But on the other hand, I also recognized some very important commonalities between the elements of the problem as he laid it out and my perspective, which is that the world will have super intelligences in it that are very different than human beings, very different mind architectures, and that things could go very badly if the power to do bad things is distributed badly to elements that have mal intent. But the thing that stood out to me is this is not a new problem. Human beings have been dealing with super intelligences in our world that have mind architectures very different from humans and have all of these dangers. They're called human institutions. A whole variety of human institutions, both public and private, even though they're made out of humans, their mind architectures are not like human mind architectures. The alignment problem has, until the last few hundred years, a really terrible history. Most of our experience of coexisting with large scale human institutions until a few hundred years ago was that these institutions were basically tools of terrible oppression, very counter-aligned with human good. What we've come up with in the last few hundred years is a set of institutional innovations, a kind of technology that we technological innovators should be paying more attention to. We've come up with institutional mechanisms like democracy, rule of law, separation of powers, individual rights, both the public ones and the private ones, separation of duties, auditing, double-entry bookkeeping. The common theme of all of these is systems for dividing power and then formulating a system of rules such that the divided power can be recomposed together to interact in a way whose emergent properties are those that are better aligned with human benefit. I would say our civilization is very much based on those. That's essentially how I come to this. Going to the Agoric papers, that was very much the perspective we had back in the 80s as to why our civilization works at all, and that was what we sought to amplify by inventing a vision of global distributed cryptographically secured smart contracting to create a new technology base for institutional innovation, to give us new rule-based frameworks for dividing power and recomposing divided power.

So I'm going to focus on the world of cryptographically enabled smart contracting as a core technology for enabling this. But let me start with the observation about computer security. Something that's very widely believed, not necessarily in this community but in the world in general including among computer security people, is that computer security is impossible, that all we can do is manage the insecurity, we cannot build secure systems. I think that is demonstrably wrong. There are many elements of security, some are much harder than others, but seL4 as an operating system has a very good track record. As far as we know, it has no flaws. Cryptography has an excellent track record going back to the RSA papers. One way to look at all of these is in various kinds of conflict situations, there's always the question of in the arms race between attack and defense, how does that arms race play out and does one side have a long-term absolute advantage over the other? In cryptography, we've seen that the defenders have an absolute advantage over the other, an exponential advantage which may as well be absolute as far as we're concerned. In computer security, the infrastructure that entire civilization rests on, the one that's widely deployed into the world, is not only insecure, it's insecure. Within the context of that infrastructure, the attackers very much have the advantage. We're well past the thresholds of machine learning and static analysis to bring down the civilization we've got because of the pervasive insecurity of the infrastructure it rests on. seL4 is a hard piece of engineering. It took years to do, the formal proof of correctness was a much larger effort than actually writing the software. But it shows that by doing that engineering, you can build systems at the operating system and programming language endpoints where the defenders have an absolute advantage over the attackers. The reason this matters is that when the defenders have an absolute advantage, then we've got a foundation for building a peaceful, cooperative world of voluntary interactions. In the physical world, it's much more tricky and harder to play out what the balance of attack and defense is as these arms races go forward. When the attackers have the advantage, there are dangers that can only be surmounted in ways that are themselves very unpleasant. I have a lot of thoughts on that that we might get to as well.

Okay, so first of all, thank you for sending me a pointer to Vitalik's paper. It's called 'My Techno-Optimism', it's a wonderful paper, I recommend it. The overall classification of worldviews that d/acc comes from, which he says is defensive or decentralized accelerationism, I would say is very much the mission that Foresight was founded on and has followed from the beginning, which is the engine of creating a glorious future is technological innovation and adoption and deployment of new technologies. It's the thing that's going to let us hopefully grow into a wonderful world that spreads out into the universe. But there are also many dangers, there's no guarantee it's going to work out well. So while pursuing the optimistic dream of making the world a much better place and spreading that wonderful nature out into the currently very dead universe, there are also many ways we can screw it up. The foresight of Foresight Institute is that perhaps, and this is a hypothesis that we should be skeptical of, but perhaps if we try to develop some look ahead, anticipate what the upcoming technologies are, anticipate what their dangers are as well as what their promise is, we can help bias how these things emerge towards futures which defend against their dangers while realizing their promise. I'd say Vitalik's overall perspective here is very much in common with all of that. That defines the original mission of Foresight and what Vitalik classifies as d/acc.

Okay, so we are of a very decentralized mindset with regard to power. The reason I said that this premise of Foresight, that by anticipating the dangers we might be able to better avoid them, why we should even be skeptical about that hypothesis is that sometimes the things that we do in reaction to the dangers make things worse. So we need to be very careful about that. Let me take the opportunity to pick a fight with some of the things I heard yesterday. The approaches that rely on keeping dangerous ideas secret or dangerous information secret can have short-term beneficial utility, but only very short term. If the power needed to produce the dangerous idea is simply problem-solving capacity, there already large AI models which are open source. These open source AI models are completely outside all of the control mechanisms, or most of the control mechanisms that I heard talked about yesterday. Any dangerous idea that the current LLM shepherds are trying to prevent the LLM from publicly answering, all those open source AI models are outside of that control. I think that any long-term solution has to be consistent with information being public, especially information that depends only on problem-solving ability to generate. So our overall system has to be resilient against the publication of dangerous ideas. So first of all, for those of you working on AI and concerned about dangers, I want to infect you with the cypherpunk mission and vision. The cypherpunks in the late 80s and especially through the 90s, this small band of technologists, of programmers with essentially no resources, won the first crypto wars against the adversary which was the United States entire intelligence apparatus. We won. How did this small band of technologists win against that? The answer is facts on the ground. We built systems that were effectively open source, widely disseminated, the ideas were certainly open. We worked on getting them deployed, most famously HTTPS through Netscape, but lots of other earlier things. We established facts on the ground by writing code, getting it deployed and adopted into the world at scale. That means that those who would centralize power were looking at this massive decentralized adoption through open source mechanisms which were uncontrollable and realizing these are just facts on the ground they had to live with. The war was much more complicated and there have been two crypto wars since then that are messier, but the overall strategy remains. The modern crypto world is also adopting this strategy: build smart contracting frameworks, build systems like blockchains for running them in a way that is incorruptible even by nation state level attackers, get them out into the world, get people using them, and just making them the facts on the ground. I think the same thing applies to building on the open source AI models and building new open source AI models. I think the ultimate route to our safety is once again in line with what it had been: have lots of different mind architectures, lots of different kinds of mind architectures out there in the world all coexisting, so that each one finds itself in the position of having its own peculiar goals and way of thinking but being a minority player in a world mostly populated by other such architectures. That's the thing that leads to seeking a way to cooperate: how can I best serve my goals in a world mostly populated by other creatures with other goals? To a large extent, finding opportunities to help them serve their goals in a way that helps me better serve my own goals. Then these frameworks of cooperation for creating rule-based interactions that are expected to be mutually beneficial, to really accelerate the development of the ability to create new such rule-based interactions like smart contracts for these divergent parties to be interacting with each other.

Yeah, actually I want to expand on that specifically. A way to think about what the Norm Hardy Prize is about, which I'm about to explain, is you can think of it as an alignment problem in the small. The alignment problem in the large that has been laid out about philosophers creating this grand utility function that encodes benefit to all of mankind, I don't believe in that at all. But an alignment problem in the small is we know how to build secure computing systems that are secure in their own terms, secure in interaction with each other. What we don't know how to do well yet is how to bring that out through the user interface so that a person, when for example they're authorizing some action because they think it serves their interest, understands what the implications are of the action that they're authorizing. The Norm Hardy Prize is about figuring out how to help users interact with secure systems securely. Given that the system itself is secure, that last mile of the human being interacting with it securely, understanding whether what they're authorizing actually fulfills the intentions behind their decision to authorize, is an essential alignment problem to enable this world going forward to actually continue to be cooperative with human interests, to serve human interests. It requires humans to be able to act in ways that they confidently represent their interests.