Hi listeners, this is the 80,000 Hours Podcast, where each week we have an unusually in-depth conversation about one of the world's most pressing problems and how you can use your career to solve it. I'm Rob Wiblin, Director of Research at 80,000 Hours. Today, I'm speaking with a research scientist at DeepMind, which is probably the most advanced developer of machine learning systems around today. As you may know, DeepMind was the outfit that first beat top Go players with a system called AlphaGo back in 2016. Since then it has developed another ML system called AlphaZero, which can learn to play chess at the very highest level, with just a day of self-play on DeepMind's processors. And, more recently, DeepMind has been working on AlphaStar, which now plays StarCraft 2 at the level of the world's top professionals. DeepMind itself says that its aim is to 'solve intelligence' and develop a general artificial intelligence that can reason about and help to solve any problem. All of this is very impressive and exciting, but regular listeners will know that I worry about how we can ensure that AI systems continue to achieve outcomes that their designers are pleased with as they become more general reasoners, and are given progressively more autonomy to intervene in an incredibly complicated world. Naturally, that is something DeepMind takes a big interest in and has been hiring researchers to work on. If DeepMind succeeds at their mission, the products that emerge from their research could end up making choices everywhere across society, and even having more influence over the direction of Earth-originating life than flesh and blood humans. With any new technology as powerful as this, it's essential that we look ahead and devise ways to make it as robust and reliable as possible. And fortunately that's just what today's guest is trying to do. Alright, here's Pushmeet.
Today, I'm speaking with Pushmeet Kohli. For the last two years, Pushmeet has been a principal scientist and research leader at DeepMind. Before joining DeepMind, he was a partner scientist and director of research at Microsoft Research and before that he was a postdoctoral associate at Trinity Hall in Cambridge. He's been an author on about 300 papers which have between them been cited at least 22,000 times. So thanks for coming on the podcast, Pushmeet.
I mentioned that I was going to be interviewing you on social media and I have to say I've never gotten so many enthusiastic question submissions from the audience. Unfortunately for all of you, we're only going to be able to get to a fraction of those. I expect we'll get to cover how listeners might be able to contribute to the development of AI that consistently improves the world, but first, as always, what are you working on at DeepMind and why do you think it's really important work?
I joined DeepMind as you just mentioned, two years back. In the past I've worked in a variety of different disciplines like machine learning, game theory, and information retrieval, computer vision. When I first came to DeepMind, I realized that the amount of work that is happening at DeepMind is just quite at a different order from what it is at other institutions. I quickly realized that making sure that the powerful techniques that we are building are stress tested and are robust and can be safely deployed in the real world is a topic of extreme importance.
All the founders of DeepMind actually were very supportive of this particular view, like Demis, Shane, Mustafa, they're all very clear on the spot that we really need to deploy AI and machine learning techniques safely. So that became the focus of my initial work, which is making sure that machine learning techniques are robust and safe when we deploy them in the real world.
More recently, I was also made in charge of the science program at DeepMind where the idea is we want to use techniques from AI and machine learning to accelerate progress in scientific disciplines. Science, we think is a source of great challenges as well as great opportunity and it's one of the key tools that humanity can use to solve some of the key challenges that we're facing. So our AI for science program aims through that.
Yeah. So you're the research leader on these two different projects, AI for science as well as the secure and robust AI team. Maybe you could tell us a little bit more about like what exactly each of those teams does and how do you balance your time with so many responsibilities?
Let me start with the safe and robust AI team. The idea behind this team was to make sure that all the systems that we're developing and the tools and techniques that the machine learning community is developing can be properly stress tested, and we can check consistency with the properties that we would expect these tools to have. And if these tools are not behaving consistently with our specifications, how can we encourage them to behave or conform to the expectations of society? Finally, how can we formally verify that their behavior is consistent, not just based on some statistical argument, but a formal mathematical argument where we can prove that these techniques will conform to the properties we expect.
So the statistical approach is kind of sampling and being like, well, most of the time it seems like it falls within these parameters, so that's fine, whereas the formal one will be like proving that it can't fall outside particular bounds?
Talk a little about the AI for science project and maybe like what things really excite you or what are the potential outputs from these projects that you think could really improve the world?
Science is a very broad area and it is one of the key topics which gives us a way to understand about the world that we live in and even who we are. In terms of the topics, we have no constraint on topics. We are looking for problems in the general area of science, whether it's biology, whether it's physics, whether it's chemistry, where machine learning can help, and not just that machine learning can help, but a way of doing machine learning where you have a dedicated team which works with conviction towards a very challenging problem could help.
So, if a problem can be solved by using machine learning, off-the-shelf type of machine learning techniques by some PhD student or some postdoc, then that might not be a good project for us because we are in this unique position where we have some of the best and most talented machine learning researchers and we have the ability to galvanize these people towards one very ambitious goal. So we look for projects where that approach really can make a difference.
Yeah, so what are some of the concrete problems that you think machine learning can help with?
One of the problems that we have already spoken about is our work on protein structure determination. So you know about proteins, that they are the building blocks of all of life, and everything about our own bodies is informed by how proteins interact with each other. They are like the machines, they are these nanomachines that are operating our whole body. We see the effects of it, but actually these micro-machines are actually what are making us work.
So understanding how they work has been a key challenge for the scientific community. One aspect of that challenge is if you have a protein, which you have specified as a sequence, can you figure out what would be its structure? Because in many cases, the structure actually informs what kind of work that protein does, which other proteins it will bind to, whether it will basically interact with other agents, and so on. This has been a longstanding problem in proteomics: how do you infer the structure of proteins? There are people who have spent their PhDs in trying to find a structure of one protein. So, it's an incredibly hard and challenging problem. We took it on because we thought if we can make progress in this area it can have a very dramatic effect on the community. So this is an example of one of the problems that we tend to look at in the science team.
Yeah. So if we managed to solve the protein folding problem, I guess that helps a lot with designing medicines that would have to interact with any proteins that are folding because then you know their shape and then you can potentially play with them.
As I mentioned, protein structure informs protein functionality. That is the hypothesis, and in many cases it does. Then in terms of protein functionality, it has implications for antibody design, drug design, various different very challenging problems that different scientific disciplines have been trying to tackle.
If listeners in five or 10 years time found themselves saying, 'Wow, DeepMind made this amazing product that has made my life better.' What do you think that would most plausibly be? Maybe it already is like just improving maps, or improving lots of services that we use online indirectly.
I think the way DeepMind thinks about this issue is in abstraction. Like intelligence is an abstraction and in some sense, it's the ability to solve many different tasks. That informs how DeepMind is structured and how DeepMind operates. We are not looking at one specific task. Of course, we need tasks to ground the progress that we're making on intelligence, but we're working on this overall enablement technology which can enable a lot of different tasks. So we don't evaluate ourselves on, what did we do on this particular task? But we generally evaluate ourselves on what technologies did we develop and what did they enable?
So you're trying to do more fundamental research into general intelligence, or intelligence at a broader level rather than just single applications?
Absolutely, but at the same time I'd have to say that tasks are extremely important because they ground us, they tell us, they inform us how much progress we are making on this very challenging problem.
Otherwise you get disconnected?
On the safe and robust AI team, what are some of the problems with current or near future AI systems that researchers are hoping to fix?
Yeah, I think this is something that the machine learning community as a whole is sort of thinking about. If you think about the history of software development, people started off by developing software systems by programming them by hand and sort of specifying exactly how the system should behave. We have now entered an era where we see that instead of specifying how something should be done, we should specify what should be done. For example, this whole paradigm of supervised learning where we show examples to the machine or to the computer, that for this input you should provide output. For this input you should provide that output. You're telling the machine what you expect it to do rather than how it should do it.
Then it's meant to figure out itself the best way to do it?
It has to figure out the best way to do it. But part of the challenge is that this description of what you want it to do is never complete, it's only partial. This is a partial specification of the behavior that we expect from the machine. So now you have trained this machine with this partial specification, how do you verify that it has really captured what you wanted it to capture, and not just memorized what you just told it? That's the key question of generalization, does it generalize? Does it behave consistently with what I had in mind when telling it, when giving it 10 correct examples. That is the fundamental challenge that all of machine learning is tackling at the moment.
Yeah. How big a problem do you think this is? I know there's a range of views within ML and outside of ML. I guess some people think this is a problem like any other and we'll just fix it as we go, whereas other people are more alarmed thinking, no, this is like a really fundamental issue that needs a lot of attention. Do you have any views on that?
I think machine learning people have thought about, it's not as if it's a new problem, generalization has been studied. The question of generalization has been studied ever since the beginning of machine learning. Like what are the inductive biases? What will machines learn? The question becomes much more challenging when you put it in the context of the complexity of the systems that we're developing today, because the systems that we're developing today are not simple linear classifiers, they are not as simple SVMs. They are much more complicated nonlinear systems with variable compute and like a lot of different degrees of freedom. To analyze exactly how this particular model behaves or generalizes and which specifications it would be consistent with, is a new type of challenge. So in some ways, it's the same challenge that we have always been looking at, but in other ways it's a completely different part of the spectrum.
So I guess the concern might be that as ML models have to interact with the complexity of the real human world in trying to actually act and improve things, there's a lot more ways for them to act out of like how you expected them to than when they're just playing chess, where it's like a much more constrained environment?
Yes, absolutely. If you think about software systems, if you are thinking about a software system for, I don't know, like you wrote a program in BASIC, or your first program in C++, and you don't really care about what it did, like when you start it. But if that program gets installed in, say an airplane-
Or the electricity grid. You should care. So, even the software industry has considered this problem. There's a long history, remember what used to happen with Windows, the Blue Screen of Death. It was quite common, it was a real technical challenge. Microsoft at that point of time was dealing with a lot of different … Was building a framework which could interact with various different devices and it was a challenge to be able to lead robustly. The last two or three decades of work that has happened in formal verification, in testing software systems, has come to the point that we now expect the failure rate of these operating systems to be extremely small. It's not as common as we used to encounter in the 80s and 90s.
Is there a concern that when Windows 98 had a problem, it would have the Blue Screen of Death and stop, whereas it's possible that machine learning algorithms, when they have a problem, they just boldly go ahead and do things that you didn't intend and maybe you don't notice until later?
That's a problem that happens with software systems generally. Termination analysis, for example, is an incredibly hard problem. How can you verify that a method will terminate? So if your software system doesn't halt, it's still a big problem, it doesn't go away. So in some sense I don't think you should have that distinction between normal software systems and machine learning systems. I think it is the same problem, it's just that software systems are also being deployed in mission critical domains. Machine learning systems are beginning to be deployed in mission critical domains. Software systems are complex and machine learning systems are also complex, but in a different way. So I think the complexity is very different and the scale is very different. The underlying problem is the same, but the types of challenges that appear are different. As machine learning and AI techniques are deployed in many different domains, these challenges will become even more critical.
Is there any way to explain, in plain language, the approaches to AI robustness that your team is working on?
We've mentioned this particular view, that when we try to test someone, when we do test them is basically ask them a bunch of questions. Suppose you are trying to test a particular individual, you are interviewing them, you ask them a few questions and then you use the answers and how they performed on those questions to get a good sense of who they are. Then in some sense you are able to do that because you have some expectation of how people behave, because like you and I are both humans.
Because we experience it with humans.
Exactly, and because you yourself are human.
But when you are reasoning about some other intelligence then-
Like a bird, then it becomes trickier. Even though we might share the same evolutionary building blocks for reasoning and so on, the behavior is different. So that comes to the question now, if there's a neural network in front of you and you are asking it questions, you can't make the same assumptions that you were making with the human, and that's what we see. In ImageNet you ask a human, 'What is the label of this image?' and even experts are not able to identify all the different labels because there are a lot of different categories and there are subtle differences. A neural network would basically give you a very high accuracy, yet if you slightly perturb that image then suddenly it will basically tell you that a school bus is an ostrich.
So what we are trying to do is basically go beyond that simple approach of taking a few questions, the traditional view of taking a few questions and asking those few questions. What we are thinking about is, can we reason about the overall neural networks' behavior? Can we formally analyze it and can we see what kind of answers it can give and in which cases does the answer change?
That makes total sense. I guess you're trying to more formally analyze what is the envelope or what's the range of behavior that a machine learning system can engage in beyond just sampling within the normal range of questions that you might give?
Yeah. The traditional approach would be that you take a particular input and then what you do is basically you take that input and then you see how that input leads to activations in the neural network and eventually the neural network gives you an answer. Then you can see the path that the input took through the neural network to reach that answer. What we are doing, is we are saying, we are not going to ask you one question, we are going to ask you a space of questions and now we're going to see what is the response to that space of questions all throughout the network. So in some sense, we are asking the neural network infinite number of questions at the same time.
If you were to do it naively, you would spend the compute of the Universe, and still not be able to verify even a very small ImageNet network or even for an MNIST network, you will not be able to specify it even if you use all the computation in the Universe. How we do it is by basically saying, let's try to encapsulate or let's try to represent that space compactly, not by those infinite points, but by certain geometries which allow us to capture that space in some low complexity. So if you are trying to bound … If you think about all the points in this particular room, there are an infinite number of them, but they're bounded by just these four walls and the ceiling and the floor. So just these six equations of these planes bound all the infinite things that are inside this room.
So you kind of try to shrink the full space into a lower dimensionality? Is that idea or?
No. We are operating in the same dimensionality, we are just representing that space compactly. We're using fewer things to represent that space and now we are going to say, how is the space of questions? So we have now infinite questions in this particular space, but the space itself is only represented by, I don't know, like eight equations or 10 equations. But there are infinite questions that live in that space and now we are going to see how the neural network answers these infinite questions rather than just one question.
And then it becomes tractable once you've defined it that way.
I was reading a blog post that you published, I think it was a month ago, Towards Robust and Verified AI: Specification Testing, Robust Training, and Formal Verification, it's great that you have this safety research blog. I think it's on Medium, we'll stick up a link to it so people can check it out, there's a lot of great posts on there. It sounded like you don't only take that approach, you also try to like actively seek out, I think, the specific niche cases where the system might act completely differently from what you intended?
Exactly. That's like adaptive testing. So people who have given the SAT or GRE would … There is some sort of adaptive testing. You answer one question and then the question that you are asked depends on the answer that you gave. So this adaptive way of questioning also is much more efficient than just, I'll prepopulate the 10 questions and you have to answer the 10 questions. If I can choose which questions I'm going to ask you, depending on the answers that you've given me, then I'm more powerful in finding places where you might be inconsistent or you might give the wrong answer.
So you pose a question to it and then you get the answer, or you get a range of answers, then you choose the worst one and then you move from there, and then apply a bunch of similar questions that are even harder and they're like, and then take the worst answer from that and then like keep going until you just find the most perverse outcome that you can search for?
Yeah, at a simple level, this is how the technique works, but in some cases you might not even know that in the first answer. For example, you ask a car to drive from point A to point B. So what is the answer there? The answer is basically you look at how the car is driving from point A to point B and just the behavior of how the car drives from point A to point B, gives you a lot about how the car is probably reasoning. So there is no perverse thing that it did, it basically did it correctly, but it gave you a lot of insights as to what might be that policy it's thinking about and that informs your next question, rather than you selecting, going from point A to point B, point C to point D and so on. So the next point that you decide is informed by you observing how the actual car drove itself.
How confident do you feel about these methods? You're like, yeah, we're killing it, we're going to solve this problem, it's just a matter of improving these techniques.
Yeah. If the answer is obvious, then it's not a good question for DeepMind to answer. So DeepMind is, in some sense, in a unique position, where we are working in this ecosystem where there is academic research, there is industrial applied research and then there is AI fundamental research and we have a unique strength in the sense of how we are structured, the conviction with which we go for problems and so on. So in some sense, that always forces us to ask the question, is this the most challenging problem that we can work on and is this the best way we can contribute to the community?
So you want to push the envelope?
There's another group at DeepMind, called the technical AGI safety team, are you familiar with what they do and how it's different from your own work?
The technical AGI safety team reasons about AGI right, where it talks about what are the issues that might come up as intelligent systems become more and more powerful. As basically systems become extremely powerful, these questions of, does the system align with my incentives? There are other safety issues that come in at that spectrum. So the machine learning safety team and the technical AGI safety team, are two sister teams, which work very, very closely with each other. We share the techniques, but the problems that we look at are at the spectrum, whereas the technical AGI safety team is looking at problems which are extremely hard, but are going to happen in a few years' time, while the machine learning safety team is looking at maybe sometimes using the same techniques at problems which are happening today.
Yeah. So what is the relationship between this like near-term AI alignment and the longer term like artificial general intelligence issues? Did you think that one is just naturally going to merge into the other over time as the AI systems that we have get more powerful?
One would hope so, because it's all about … The fundamental issues are quite similar, so if you think about value alignment or specification consistency, we talk about these things in the near-term and long-term safety regimes as in these ways, but at the basic level the problems are quite similar.
Yeah, it's interesting. I feel like just two years ago, I heard a lot of people say that the short-term issues with ML systems are quite different from the long-term issues and working on the problems that we have now won't necessarily help with the long-term issues, but that view seems to have become much less common, much less fashionable. Have you noticed that as well or is that just the people that I know?
Yeah, I think there is a gradual realization that many of the problems are shared. Now, of course the long-term AGI safety research has some unique problems, which the short-term doesn't have.
Yeah. What are some of those?
One of the things is when we talk about specification at the moment, when we think about deploying machine learning systems, we're talking about it in a very specific domain. So the specification language, the way we express what we want the machine to do can be constrained. The language needed for specifying what the behavior should be can be limited, but when you talk about an AGI, an AGI basically can solve any problem on the planet.
In principle. So then what is the language in which you specify it?
How do you communicate to that very powerful agent. It is a unique challenge.
Do you think we'll end up having to use human language? Because that's like the most like high bandwidth method of communication that we have with other people, maybe it's the highest bandwidth method we have with an AI system as well?
It goes back to the question that our intelligence evolved in a particular way and in some sense there's the good coupling between language, human language and human intelligence. So what comes first? What came first? But there is some notion that human intelligence is able to deal with concepts expressed in human language. Now, a very powerful intelligence, a different intelligence might have its own language, might have its own concepts, might have its own abstractions, but in order for us to communicate between it, we'll need to either build a translator within those two languages or somehow try to make sure that the intelligence that we're building conforms or is very similar to a human language so that it can understand the same abstractions and concepts and the properties that we expect of such systems.
Are there any other differences between the work that you're doing and the work of the AGI, or any different challenges that the AGI team faces that are worth highlighting?
There are, at a very high level, the problems are quite similar, but the practical machine learning systems throw up a number of different sorts of issues. Some I think are shared, like the question of privacy and security and these are all questions that grew in both contexts. I don't think there are many problems which are different. The approaches and the problem instances might be different, but at the basic level, at the abstract level, the problems are quite similar.
Thinking about another division between different kinds of work. I think one framing that a lot of people have about AI safety and reliability is that there are some people who are working on capabilities like making AI more powerful and then there's other people perhaps like you who are working on reliability and safety and alignment and all of that. It's like, I guess, the extreme version of this view's like, well the capabilities people are creating the problem and then like you're cleaning it up, you're fixing it up and making it better and solving the issues that are arising. Then possibly on that view, you think, well working on capabilities is like possibly even harmful? It's not clear how helpful that is? I think this is a view that was more common in the past and it's like has also been like fading, but do you have any comments on whether that's a good way for people to conceptualize the whole deal with the ML progress?
That's a very interesting question. A lot of people see ML safety work as a sort of tax. That you have to pay this tax to make sure that you are doing things correctly. Safety is something that is necessary, you have to do it, but if you're not driven by it. I don't see it that way in some sense. So it's not … I don't think basically an organization which is doing safety work is paying a tax. In fact, it is to its advantage. How do we explain this?
Suppose, we were, you and me, took a unique mission and the mission was that we're going to drive a car around the planet. One approach is you sit in your car seat and you drive off, without putting your seatbelt. The other issue would be, you put your seatbelt on. If we were just going from point A to point B and like maybe one kilometer, the probability of an accident is so low that you might actually reach the destination without putting the seatbelt on. But if your destination is so far, we are circumnavigating the whole world, if we think about the probability of who reaches the end, you will see the person who put on the seatbelt.
Makes a meaningful difference.
Exactly. So it's about enablement, it's not a sort of a tax, it's basically enabling the creation and development of these technologies.
Can't remember who, but someone gave me this analogy to bridge building where they're like, we don't have bridge builders and then bridge safety people who are completely separate from the people, but this is like, it's not a bridge unless it doesn't fall down, there's no like anti-falling down bridge specializers. That's just part of it. I guess you're saying there's like it's not meaningful to talk about like building good ML systems without them reliably doing what you want. That's like an absolutely core part of how you design it in the first place.
Is there any steel man of this position, that like well maybe we don't want to speed up like some sort of capabilities research, we might prefer to delay that until we've done more of the work that you've done? Perhaps we want to put extra resources into this alignment work, or reliability work but as early as possible.
Yeah, I think the answer to that question actually is very contextual. In certain contexts, people have all already made that case, that when you try to deploy machine learning systems in very safety-critical domains, you ought to understand what is the behavior of the system. In other cases where you're trying to do some experimentation or the proof of concept and so on, it's fine to be able to do that sort of stuff for fun and so on and to see what are the limits of things. But I think there is a spectrum and the answer is contextual, there is no clear answer.
Yeah. Is it generally the case that work that improves like AI capabilities or allows it to do new applications or have better insights also increases safety as you go or also increases alignment as you go? Because that's just like part and parcel of improving algorithms?
Yeah, absolutely. In some sense if you think about machine learning, what is machine learning sort of … I think of machine learning as a translation service. Machine learning is a translation service where you bring it some specifications or some specifications on what behavior you want out of your system and it translates it into a system which claims to have those properties. So, what is inside that box, inside that translation system? It has various different inductive biases. They're either in the form of regularizes or different types of machine learning models, which have different inductive biases and different optimization techniques and so forth. But all of those techniques are essentially trying to solve this translation problem, converting your specification, which could be in proper examples or could be just the input examples, in the case of unsupervised or supervised learning or it could be interactions with the world and translating it into a classifier or a policy depending on the problem type.
I imagine there are some listeners out there who know quite a bit about ML and are considering doing during careers in ML and they would think that their passion is doing the kind of work that you're doing or the work that the AGI team is doing. Imagining that they couldn't actually find … There's only so many people doing this alignment, reliability work. Imagine that they couldn't get one of those positions but they could get some other general ML role to improve their skills, but then they're nervous because ultimately what I really want to do is the thing that you're doing. What advice might you give them? Will you just say dive in and you'll be able to … Well, like in any role you can find some reliability thing to contribute or at least to be able to change into a more like reliability focused role later on?
Machine learning generally and the robustness problem are not different from each other. In some sense every machine learning practitioner should be thinking about the question of generalization. Does my system generalize? Is my system robust? These are problems that not-
This is everywhere. The key advice that I would give to people is, when they approach the problem they should not approach it from the perspective of, well, if I take this particular input, apply this tool, I get this output. But think of it as to why that happens, or what are we after and how will we get that? Instead of, well here are very systematic view of like what needs to be done, but not how we should work, but what are we after?
Yeah. Are there any ML projects or research agendas that don't carry the label, safety, that you think like are going to be especially useful for safety and reliability in the long term? People wouldn't think of it as an especially like reliability focused project, but it turns out that actually it is like, it's going to have a huge influence on that potentially. Any of them which stand out?
I think anything to do with optimization. So, optimization is the general area. It's not tied to robustness or it's not tied to even machine learning. Optimization is used in a general context for various different problems in operations research and game theory, whatever. Optimization is key to what we do. Optimization is a fundamental technique that we use in safety work to improve how our systems conform to specifications. We always are optimizing the performance of our systems, not to produce specific labels, but to conform to the more general problem or to conform to these general properties that we expect. Not to the simple properties of, well for this input it should be this output. That's a simple property, but you can have more sophisticated properties and in traditional machine learning you are trying to optimize consistency with those simple properties or reduce your loss or empirical risk and in our case, we are reducing our loss or reducing the risk of inconsistency with the specifications.
What do you think the general public or listeners, like what would be their biggest misunderstandings about AI safety and reliability?
That's a very hard question.
Because there's a lot of them or just how it depends on who it is?
Yeah, it depends on who it is. I think one interesting thing that people need to think about is in the same way that we don't expect every human to be the same, we shouldn't expect that every machine learning system is the same. The second thing is that when we test our machine learning systems and then we make claims about, well, our model performs very well on a particular data set, say ImageNet or some other data set for, I don't know, speech recognition or … You are solving that data set. Solving that data set does not imply that you have solved that problem. There's a difference between solving a benchmark, or getting a high performance on the benchmark versus solving the problem.
Then the key question is, if solving a benchmark does not imply solving a particular problem, then what does? That is a question where I think a lot more needs to be done because that is the fundamental problem of what do we want or what do we expect out of systems? When do or how do we articulate? What does it mean to solve a image classification or some other problem? That is where the general public need to think about what is it that they're after and what do they expect out of these systems?
Before you were talking about this analogy between just general software debugging and security and robustness of AI. Do you want to expand on that analogy and what things we can learn from software safety?
There are certain things that we can learn. First of all like, even though it is incredibly hard, progress can be made. We knew that solving the halting problem was undecidable, yet we now have very good tools for termination analysis. That doesn't mean that we've solved the halting problem, yet it means that for certain instances we can show that things will terminate and programs do not just hang all the time and they sometimes do-
Less than they used to. So even though some problems appear incredibly challenging when you first look at them technically, overtime you make progress and you find ways to somehow approximate what we're after. So I think that is a good thing to learn from the software reliability issue. The other thing to learn is when you think about defects in software. There are those defects, it's not enough to say that, oh, there is a defect, but nobody will find it. That is something that we should learn, because there are always people who will find it.
Because they're actively trying or just because there's so many people using something that eventually they run into it?
For both reasons. So there is nature versus adversary. Nature will find your bug or the adversary will find your bug and they will both use it and you will incur a cost for both cases. So you have to think about how do you want to make sure that your machine learning system is robust to nature and to the adversary.
So you think people systematically underestimate how likely it is that the problems they know are in their software or their AI system will actually materialize and create problems?
Yeah. I think, nobody starts off by saying, oh, I should write this software, which can be hacked by some hacker and use it to steal some information. Everyone, every software engineer is trying to make sure that their program does what it says on the tape. But still-
But it's hard. Even after decades of work, we still see that defects in or certain bugs in machine learning, or like in normal software systems are sometimes exploited by people who can then use it for various different purposes.
Yeah, definitely. It seems like computer security is just an unsolved problem and like a severe ongoing problem. I guess … Would you take from that analogy like we could have just like many years where we're going to have issues with ML not doing what we want and it's just going to be like a lot of real slog for potentially a decade or two before we can fix it up.
In some sense, one thing that we have to take is we have to take it seriously. The second thing that we have to take in, is that have to learn from history. There is already a lot of work that has already happened. The third most optimistic thing that we can take is that, yes, the systems that we are building are extremely complex but they're also simple in other ways. There's simplicity in basically the building blocks and that simplicity should help us actually do much better than the traditional software systems which are messy in their own way.
Can you explain how they're simpler?
We were talking about this whole idea of not asking the system one question but asking infinite questions. That technique of asking the machine infinite questions or reasoning about how it is going to perform not on just one particular input, but a set of inputs, or a space of inputs. That is called abstract interpretation, in the software analysis and software verification community. Abstract interpretation like when I was talking to you in the context of neural networks, because our operators are simpler in some sense, there are these neurons, which behave in a specific way, we can capture what transformations they are going to do to the input. While in a traditional program there are so many different types of operators they are very different behaviors than so on. So you can do that as well, but it's slightly more complicated.
Yeah. That leads into my next question, which was, is it going to be possible to formally verify safety performance on the ML systems that we want to use?
I think a more pertinent question is, would it be possible to specify what we want out of the system, because at the end of the day you can only verify what you can specify. I think technically there is nothing, of course this is a very hard problem, but fundamentally we have solved hard search problems and challenging optimization problems and so on. So it is something that we can work towards, but a more critical problem is specifying what do we want to verify? What do we want to formally verify? At the moment we verify, is my function consistent with the input-output examples, that I gave the machine learning system and that's very easy. You can take all the inputs in the training set, you can compute the outputs and then check
Whether the outputs are the same or not. That's a very simple thing. No rocket science needed.
Now, you can have a more sophisticated specification saying, well, if I perturb the input in some way or transform the input and I expect the output to not change or change in a specific way, is it true? That's a harder question and would be showing that we can try to make progress. But what other types of specifications or what other type of behavior or what kind of rich questions might people want to ask in the future? That is a more challenging problem to think about.
Interesting. So then relative to other people you think it's going to be figuring out what we want to verify that's harder rather than the verification process itself?
Yeah, like how do you specify what is the task? Like a task is not a data set.
How do you? Do you have any thoughts on that?
Yes. I think this is something that... It goes into like how this whole idea of, it's a very philosophical thing, how do we specify tasks? When we talk about tasks, we talk about in human language. I can describe a task to you and because we share some notion of certain concepts, I can tell you, well, we should try to detect whether a car passes by and what is a car, a car has something which has four wheels and something, and can drive itself and so on. And a child with a scooter, which also has four wheels goes past and you say, 'Oh that's a car.' You say, 'No, that's not a car.' The car is slightly different, bigger, basically people can sit inside it and so on. I'm describing the task of detecting what is a car in these human concepts that I believe that you and I share a common understanding of.
That's a key assumption that I've made. Will I be able to also communicate with the machine in those same concepts? Does the machine understand those concepts? This is a key question that we have to try to think about. At the moment we're just saying, oh input, this is the output, input this output that. This is a very poor form of teaching. If you're trying to teach an intelligent system, just showing it examples is a very poor form of teaching. There's a much more richer, like when we are talking about solving a task, we are talking in human language and human concepts.
It seems like you might think that it would be reliability enhancing to have better natural language processing that, that's going to be disproportionately useful?
