#221 – Douglas Lenat: Cyc and the Quest to Solve Common Sense Reasoning in AI

When I was a faculty member in the computer science department at Stanford, my colleagues and I did research in all sorts of artificial intelligence programs. So, natural language understanding programs, robots, expert systems, and so on. And we kept hitting the very same brick wall. Our systems would have impressive early successes. And so if your only goal was academic, namely to get enough material to write a journal article, that might actually suffice. But if you're really trying to get AI, then you have to somehow get past the brick wall. And the brick wall was the programs didn't have what we would call common sense. They didn't have general world knowledge. They didn't really understand what they were doing, what they were saying, what they were being asked. And so very much like a clever dog performing tricks. We could get them to do tricks, but they never really understood what they were doing. Sort of like when you get a dog to fetch your morning newspaper. The dog might do that successfully, but the dog has no idea what a newspaper is or what it says or anything like that.

I think of understanding more like, think of it more like the ground you stand on, which could be very shaky, could be very unsafe, but most of the time is not because underneath it is more ground and eventually you know rock and other things. But layer after layer after layer, that solid foundation is there and you rarely need to think about it. You rarely need to count on it, but occasionally you do. And I've never used this analogy before, so bear with me. But I think the same thing is true in terms of getting computers to understand things, which is you ask a computer a question, for instance, Alexa or some robot or something, and maybe it gets the right answer. But if you were asking that of a human, you could also say things like why or how might you be wrong about this or something like that and the person would answer you. And it might be a little annoying if you have a small child and they keep asking why questions in series. Eventually, you get to the point where you throw up your hands and say, "I don't know. It's just the way the world is." But for many layers you actually have that layered solid foundation of support so that when you need it you can count on it.

And when do you need it? Well, when things are unexpected, when you come up against a situation which is novel, for instance, when you're driving, it may be fine to have a small program, a small set of rules that cover 99% of the cases, but that 1% of the time when something strange happens, you really need to draw on common sense. For instance, my wife and I were driving recently and there was a trash truck in front of us and I guess they had packed it too full and the back exploded and trash bags went everywhere. And we had to, you know, make a split-second decision. Are we going to slam on our brakes? Are we going to swerve into another lane? Are we going to just run it over? Because there were cars all around us. And in front of us was a large trash bag, and we know what we throw away in trash bags. Probably not a safe thing to run over. Over on the left was a bunch of fast food restaurant trash bags. And it's like, oh well, those things are just like styrofoam and leftover food. We'll run over that. And so that was a safe thing for us to do.

Now, that's the kind of thing that's going to happen maybe once in your life. And but the point is that there's almost no telling what little bits of knowledge about the world you might actually need in some situations which were unforeseen. But see when you sit on that mountain or that ground that goes deep of knowledge in order to make a split-second decision about fast food trash or random trash from the back of a trash truck, you need to be able to leverage that ground you stand on in some way.

So let's tease apart two different tasks, really both of which are incredibly important and even necessary if you're going to have this in a usable, useful, usable fashion as opposed to say like library books sitting on a shelf where the knowledge might be there but if a fire comes the books are going to burn because they don't know what's in them and they're just going to sit there while they burn. So there are two aspects of using the knowledge. One is a kind of a theoretical, how is it possible at all, and then the second aspect of what you said is how can you do it quickly enough.

So how can you do it at all is something that philosophers have grappled with and fortunately philosophers 100 years ago and even earlier developed a kind of formal language like English. It's called predicate logic or first order logic or something like predicate calculus and so on. So there's a way of representing things in this formal language which enables a mechanical procedure to sort of grind through and algorithmically produce all of the same logical entailments, all the same logical conclusions that you or I would from that same set of pieces of information that are represented that way.

So that raises a couple of questions. One is how do you get all this information from say observations and English and so on into this logical form, right? And secondly, how can you then efficiently run these algorithms to actually get the information you need in the case I mentioned in a tenth of a second rather than say in 10 hours or 10,000 years of computation. And those are both really important questions.

I'm going to disappoint you by giving you an actual answer to your question.

Yes. Okay. So now we have like three things to talk about. Keep adding more. Although the first and the third are related. So let's leave the efficiency question aside for now. So how does all this information get represented in logical form so that these algorithms, resolution theorem proving and other algorithms can actually grind through all the logical consequences of what you said? And that ties into your question about well how many of these things do you need? Because if the answer is small enough then by hand you could write them out one at a time.

So in the early 1984 I held a meeting at Stanford where I was a faculty member there, where we assembled about half a dozen of the smartest people I know. People like Alan Newell and Marvin Minsky and Alan Kay and a few others.

No, no, he wasn't part of this meeting, but I think Ed Feigenbaum was there. I think Josh Lederberg was there. So we had all these different smart people and we came together to address the question that you raised: if it's important to represent common sense knowledge and world knowledge in order for AIs to not be brittle, in order for AIs not to just have the veneer of intelligence, well, how many pieces of common sense? How many if-then rules, for instance, would we have to actually write in order to essentially cover what people expect perfect strangers to already know about the world?

And I expected there would be an enormous divergence of opinion and computation, but amazingly everyone got an answer which was around a million. And one person got the answer by saying, well, look, you can only burn into human long-term memory a certain number of things per unit time, like maybe one every 30 seconds or something and other than that it's just short-term memory and it flows away like water. So by the time you're say 10 years old or so, how many things could you possibly have burned into your long-term memory and it's like about a million. Another person went in a completely different direction and said, well, if you look at the number of words in a dictionary, not a whole dictionary but for someone to essentially be considered to be fluent in a language, how many words would they need to know and then about how many things about each word would you have to tell it and so they got to a million that way.

Another person said, well, let's actually look at one single short one-volume desk encyclopedia article. And so we'll look at what was like a four-paragraph article or something I think about grebes. Grebes are a type of water fowl. And if we were going to sit there and represent every single thing that was there, how many assertions or rules or statements would we have to write in this logical language and then multiply that by all of the number of articles that there were. So all of these estimates came out with a million. And so if you do the math it turns out that like, oh well then maybe in something like 100 person-years, in one or two person-centuries, we could actually get this written down by hand.

And a marvelous coincidence, an opportunity existed. Right at that point in time, the early 1980s, there was something called the Japanese Fifth Generation computing effort. Japan had threatened to do in computing and AI and hardware what they had just finished doing in consumer electronics and the automotive industry, namely wresting control away from the United States and more generally away from the West. And so America was scared. And Congress did something. That's how you know it was a long time ago because Congress did something. Congress passed something called the National Cooperative Research Act. And what it said was, "Hey, all you big American companies" — that's also how you know it was a long time ago because they were American companies rather than multinational companies — "Hey, all you big American companies, normally it would be an antitrust violation if you colluded on R&D, but we promise for the next 10 years, we won't prosecute any of you if you do that, to help combat this threat."

And so overnight, the first two consortia, research consortia in America sprang up, both of them coincidentally in Austin, Texas. One called MCC focusing on hardware chips and so on and then one called MCC, the Microelectronics and Computer Technology Corporation, focusing more on software, on databases and AI and natural language understanding and things like that. And I got the opportunity thanks to my friend Woody Bledsoe, who was one of the people who founded that, to come and be its principal scientist. And Admiral Bob Inman, who was the person running MCC, came and talked to me and said, "Look, professor, you're talking about doing this project. It's going to involve person-centuries of effort. You've only got a handful of graduate students. You do the math. It's going to take you longer than the rest of your life to finish this project. But if you move to the wilds of Austin, Texas, we'll put 10 times as many people on it and you'll be done in a few years." And so that was pretty exciting. And so I did that. I took my leave from Stanford. I came to Austin. I worked for MCC.

And good news and bad news. The bad news is that all of us were off by an order of magnitude. It turns out what you need are tens of millions of these pieces of knowledge about every day, sort of like if you have a coffee cup with stuff in it and you turn it upside down, the stuff in it is going to fall out. So you need tens of millions of pieces of knowledge like that, even if you take trouble to make each one as general as it possibly could be. But the good news was that thanks to initially the Fifth Generation effort and then later US government agency funding and so on, we were able to get enough funding not for a couple person-centuries of time but for a couple person-millennia of time which is what we've spent since 1984 getting Cyc to contain the tens of millions of rules that it needs in order to really capture and span not all of human knowledge but the things that you assume other people know, the things you count on other people knowing.

And so by now we've done that. And the good news is since you've waited 38 years just about to talk to me, we're about at the end of that process. So most of what we're doing now is not putting in even what you would consider common sense, but more putting in domain-specific, application-specific knowledge about healthcare in a certain hospital or about oil pipes getting clogged up or whatever the applications happen to be. So, we've almost come full circle and we're doing things very much like the expert systems of the 1970s and the 1980s, except instead of resting on nothing and being brittle, they're now resting on this massive pyramid, if you will, this massive lattice of common sense knowledge so that when things go wrong, when something unexpected happens, they can fall back on more and more and more general principles, eventually bottoming out in things like for instance, if we have a problem with the microphone, one of the things you'll do is unplug it, plug it in again, and hope for the best, right? Because that's one of the general pieces of knowledge you have in dealing with electronic equipment or software systems or things like that.

Oh, absolutely. That's one of the things that Cyc knows. That's actually one of the fundamental laws of nature, I believe. I wouldn't call it a law. It's more like a... it seems to work every time. So it sure looks like a law. I don't know.

So that basically covered the resources needed and then we had to devise a method to actually figure out well what are the tens of millions of things that we need to tell the system. And for that we found a few techniques which worked really well. One is to take any piece of text almost — could be an advertisement, it could be a transcript, it could be a novel, it could be an article — and don't pay attention to the actual type that's there, the black space on the white page. Pay attention to the complement of that, the white space, if you will. So, what did the writer of this sentence assume that the reader already knew about the world? For instance, if they used a pronoun, how did they figure out that you — why did they think that you would be able to understand what the intended reference of that pronoun was? If they used an ambiguous word, how did they think that you would be able to figure out what they meant by that word?

The other thing we look at is the gap between one sentence and the next one. What are all the things that the writer expected you to fill in and infer occurred between the end of one sentence and the beginning of the other? So, like if the sentence says Fred Smith robbed the Third National Bank. He was sentenced to 20 years in prison. Well, between the first sentence and the second, you're expected to infer things like Fred got caught. Fred got arrested. Fred went to jail. Fred had a trial. Fred was found guilty. And so on. If my next sentence starts out with something like the judge, then you assume it's the judge at his trial. If my next sentence starts out something like the arresting officer, you assume that it was the police officer who arrested him after he committed the crime and so on. So those are two techniques for getting that knowledge.

The other thing we sometimes look at is sort of like fake news or sort of humorous Onion headlines or headlines in the Weekly World News, if you know what that is, or the National Enquirer, where it's like, oh, we don't believe this, then we introspect on why don't we believe it. So there are things like B-17 lands on the moon — why don't we — what do we know about the world that causes us to believe that that's just silly? Or another thing we look for are contradictions, which things can't both be true. And we say, what is it that we know that causes us to know that both of these can't be true at the same time? For instance, in one of the Weekly World News editions, in one article, it talked about how Elvis was sighted even though he was getting on in years. And another article in the same one talked about people seeing Elvis's ghost. So it's like, why do we believe that at least one of these articles must be wrong?

So we have a series of techniques like that that enable our people — and by now we have about 50 people working full-time on this and have for decades. So we've put in the thousands of person-years of effort. We've built up these tens of millions of rules. We constantly police the system to make sure that we're saying things as generally as we possibly can. So you don't want to say things like no mouse is also a moose, because if you said things like that, then you'd have to add another one or two or three zeros onto the number of assertions you'd actually have to have. So at some point we generalize things more and more and we get to a point where we say, oh yeah, for any two biological taxa, if we don't know explicitly that one is a generalization of another, then almost certainly they're disjoint. A member of one is not going to be a member of the other and so on.

Right, in general, things are not both alive and dead at the same time.

Well, that raises a couple of important points. But no, no. So what you bring up is this really important point of well, how do you handle exceptions and inconsistencies and so on? And one of the hardest lessons for us to learn, it took us about five years to really grit our teeth and learn to love it, is we had to give up global consistency. So the knowledge base can no longer be consistent. So this is a kind of scary thought. I grew up watching Star Trek and anytime a computer was inconsistent, it would either freeze up or explode or take over the world or something bad would happen. Or if you come from a mathematics background, once you can prove false you can prove anything. So that's not good. And so the old knowledge-based systems were all very, very consistent. But the trouble is that by and large, our models of the world, the way we talk about the world, there are all sorts of inconsistencies that creep in here and there that will sort of kill some attempt to build some enormous globally consistent knowledge base.

And so what we had to move to was a system of local consistency. So a good analogy is you know that the surface of the earth is more or less spherical globally, but you live your life every day as though the surface of the earth were flat. When you're talking to someone in Australia, you don't think of them as being oriented upside down to you. When you're planning a trip, even if it's a thousand miles away, you may think a little bit about time zones, but you rarely think about the curvature of the earth. And for most purposes, you can live your whole life without really worrying about that because the earth is locally flat. In much the same way, the Cyc knowledge base is divided up into almost like tectonic plates which are individual contexts and each context is more or less consistent but there can be small inconsistencies at the boundary between one context and the next one. And so by the time you move say 20 contexts over, there could be glaring inconsistencies. So eventually you get from the normal modern real-world context that we're in right now to something like a Roadrunner cartoon context where physics is very different and in fact life and death are very different because no matter how many times he's killed, the coyote comes back in the next scene and so on.

So that was a hard lesson to learn and we had to make sure that our representation language, the way we actually encode the knowledge and represent it, was expressive enough that we could talk about things being true in one context and false in another. Things that are true at one time but false in another. Things that are true in one region like one country but false in another. Things that are true in one person's belief system but false in another person's belief system. Things that are true at one level of abstraction and false at another. For instance, at one level of abstraction, you think of this table as a solid object, but at the atomic level, it's mostly empty space.

So it's good you asked that question because you're pointed in the right direction, which is you want contexts to be first-class objects in your system's knowledge base. In particular in Cyc's knowledge base. And by first-class object, I mean that we should be able to have Cyc think about and talk about and reason about one context or another context the same way it reasons about coffee cups and tables and people and fishing and so on. And so contexts are just terms in its language just like the ones I mentioned. And so Cyc can reason about context. Contexts can arrange hierarchically and so on. And so you can say things about, let's say, things that are true in the modern era, things that are true in a particular year would then be a subcontext of the things that are true in a broad, let's say, a century or a millennium or something like that. Things that are true in Austin, Texas are generally going to be a specialization of things that are true in Texas, which is going to be a specialization of things that are true in the United States and so on. And so you don't have to say things over and over again at all these levels. You just say things at the most general level that it applies to and you only have to say it once and then it essentially inherits to all these more specific contexts.

Oh, you definitely have to think of it as a graph. So we could talk about for instance why the Japanese Fifth Generation computing effort failed. There were about half a dozen different reasons. One of the reasons they failed was because they tried to represent knowledge as a tree rather than as a graph. And so each node in their representation could only have one parent node. So if you had a table that was a wooden object, a black object, a flat object, and so you had to choose one and that's the only parent it could have. When of course depending on what it is, you need to reason about it. Sometimes it's important to know that it's made out of wood. Like if we're talking about a fire, sometimes it's important to know that it's flat, if we're talking about resting something on it. So one of the problems was that they wanted a kind of Dewey decimal numbering system for all of their concepts which meant that each node could only have at most 10 children and each node could only have one parent. And while that does enable the Dewey decimal type numbering of concepts, labeling of concepts, it prevents you from representing all the things you need to about objects in our world. And that was one of the things which they never were able to overcome. And I think that was one of the main reasons that that project failed.

By the way, I should mention that Marvin wouldn't do his estimate until someone brought him an envelope so that he could literally do a back-of-the-envelope calculation to come up with his number.

I want to distinguish between what you think of as facts and the kind of things that we represent. So we map to and essentially make sure that Cyc has the ability to, as it were, read and access the kind of facts you might find in Wikidata or stated in a Wikipedia article or something like that. So what we're representing, the things that we need a small number of tens of millions of, are more like rules of thumb, rules of good guessing, things which are usually true and which help you to make sense of the facts that are sitting off in some database or some other more static story.

Exactly. And that's what I meant about ferreting out the unstated things in text. What are all the things that were assumed? And so those are things like if you have a problem with something, turning it off and on often fixes it for reasons we don't really understand and we're not happy about, or people can't be both alive and dead at the same time. And or water flows downhill. If you search online for water flowing uphill and water flowing downhill, you'll find more references for water flowing uphill because it's used as a kind of metaphorical reference for some unlikely thing. Because of course everyone already knows that water flows downhill. So why would anyone bother saying that?

I would say assertions. Assertions or rules. Because I'm not talking about rigid rules, but rules of thumb. But assertions is a nice one that covers all of these things.

Oh, I'm sorry. I'm so sorry. Okay, but assertions works.

For the first several years, I would have said that it was on the order of one or two million. Yeah. And so it took us about five or six years to realize that we were off by a factor of 10.

So I think I'm right enough. And let me explain what I mean by that, which is sometimes like if you have an old-fashioned pump, you have to prime the pump. And then eventually it starts. So I think I'm right enough in the sense that the pump — what we've built, even if it isn't so to speak everything you need, it's primed the knowledge pump enough that...

Cyc can now itself help to learn more and more automatically on its own by reading things and understanding and occasionally asking questions like a student would or something and by doing experiments and discovering things on its own and so on. So through a combination of Cyc-powered discovery and Cyc-mediated reading, it will be able to bootstrap itself. Maybe it's the final 2%, maybe it's the final 99%. So even if I'm wrong, all I really need to build is a system which has primed the pump enough that it can begin that cascade upward that self-reinforcing sort of quadratically or maybe even exponentially increasing path upward that we get from for instance talking with each other. That's why humans today know so much more than humans a hundred thousand years ago. We're not really that much smarter than people were a hundred thousand years ago, but there's so much more knowledge and we have language and we can communicate. We can check things on Google and so on. So, effectively, we have this enormous power at our fingertips. And there's almost no limit to how much you could learn if you wanted to because you've already gotten to a certain level of understanding of the world that enables you to read all these articles and understand them, that enables you to go out and if necessary do experiments although that's slower as a way of gathering data and so on and I think this is really an important point which is if we have artificial intelligence, real general artificial intelligence, human level artificial intelligence then people will become smarter, it's not so much that it'll be us versus the AIs it's more like us and the AIs together will be able to do things that require more creativity. That would take too long right now, but we'll be able to do lots of things in parallel. We'll be able to misunderstand each other less. There's all sorts of value that effectively for an individual would mean that individual will for all intents and purposes be smarter and that means that humanity as a species will be smarter. And when was the last time that any invention qualitatively made a huge difference in human intelligence? You have to go back a long ways. It wasn't like the internet or the computer or mathematics or something. It was all the way back to the development of language. We sort of look back on pre-linguistic cavemen as well, you know, they weren't really intelligent, were they? They weren't really human, were they? And I think that as you said 50, 100, 200 years from now people will look back on people today right before the advent of these sort of lifelong general AI muses.

And say, you know, those poor people, they weren't really human, were they?

Certainly a big leap in one direction. We're not sure whether it's upward or downward. Well, I mean very specific parts of the internet which is access to information like a website like Wikipedia like ability for human beings from across the world to access information so very quickly.

So when I was growing up we had to use slide rules. We had to be able to estimate and so on. Today people don't really understand numbers. They don't really understand math. They don't really estimate very well at all. And so on, they don't really understand the difference between trillions and billions and millions and so on very well. Because calculators do that all for us. And thanks to things like the internet and search engines that same kind of juvenileism is reinforced in making people essentially be able to live their whole lives not just without being able to do arithmetic and estimate but now without actually having to really know almost anything because anytime they need to know something they'll just go and look it up.

That's right. If you read a book about how to make butter, that's not the same as if you had to like learn it and do it yourself and so on. So, let's just say that something is gained but something is lost every time you have these sorts of dependencies on technology. And overall I think that having smarter individuals and having smarter AI augmented human species will be one of the few ways that we'll actually be able to overcome some of the global problems we have involving poverty and starvation and global warming and overcrowding all the other problems that are besetting the planet. We really need to be smarter and there really only two routes to being smarter. One is through biochemistry and genetics, genetic engineering. The other route is through having general AIs that augment our intelligence and you know hopefully one of those two paths to salvation will come through before it's too late.

I agree. I think that the main thing that AI prosthesis, AI amplifiers will do for people is make it easier, maybe even unavoidable for them to do good critical thinking. So pointing out logical fallacies, logical contradictions and so on in things that they otherwise would just blindly believe, pointing out essentially data which they should take into consideration if they really want to learn the truth about something and so on. So I think doing not just educating in the sense of pouring facts into people's heads but educating in the sense of arming people with the ability to do good critical thinking is enormously powerful. The education system that we have in the US and worldwide generally don't do a good job of that. But I believe that the AI will, the AIs can and will in the same way that everyone can have their own Alexa or Siri or Google Assistant or whatever, everyone will have this sort of cradle-to-grave assistant which will get to know you which you'll get to trust. It'll model you, you'll model it. And it'll call to your attention things which will in some sense make your life better, easier, less mistake-prone and so on, less regret if you listen to it.

Yes, the full spectrum. Well, let me sort of because you mentioned there's so many fascinating things. Let me jump back to the idea of automated reasoning. So, the acquisition of new knowledge has been done in this very interesting way but primarily by humans doing this.

by definition because those pieces of omitted information of those missing steps as it were are pieces of common sense. If you actually included all of them, it would almost be offensive or confusing to the reader. It's like why are they telling me all these? of course I know that you know all these things and so it's one of these things which almost by its very nature has almost never been explicitly written down anywhere because by the time you're old enough to talk to other people and so on you know if you survived to that age presumably you already got pieces of common sense like you know if something causes you pain whenever you do it probably not a good idea to keep doing.

So we have in fact two directions we're pushing on very heavily currently at Cyc and one involves natural language understanding and the ability to read what people have explicitly written down and to pull knowledge in that way. But the other is to build a series of knowledge editing tools, knowledge entry tools, knowledge capture tools, knowledge testing tools and so on. Think of them as like a user interfaced suite of software tools if you want something that will help people to more or less automatically expand and extend the system in areas where for instance they want to build some have it do some application or something like that. So I'll give you an example of one which is something called abduction. So you've probably heard of deduction and induction and so on, but abduction is unlike those abduction is not sound. It's just useful. So for instance deductively if someone is out in the rain and they're going to get all wet and when they enter a room they might be all wet and so on. So that's deduction. But if someone were to walk into the room right now and they were dripping wet, we would immediately look outside to say, "Oh, did it start to rain?" or something like that. Now, why did we say maybe it started to rain? That's not a sound logical inference, but it's certainly a reasonable abductive leap to say, well, one of the most common ways that a person would have gotten dripping wet is if they had gotten caught out in the rain or something like that. So, what does that have to do with what we were talking about? So suppose you're building one of these applications and the system gets some answer wrong and you say oh yeah the answer to this question is this one not the one you came up with then what the system can do is it can use everything it already knows about common sense general knowledge the domain you've already been telling it about and context like we talked about and so on and say well here are seven alternatives each of which I believe is plausible given everything I already know. And if any of these seven things were true, I would have come up with the answer you just gave me instead of the wrong answer I came up with. Is one of these seven things true? And then you, the expert, will look at those seven things and say, "Oh yeah, number five is actually true." And so without actually having to tinker down at the level of logical assertions and so on, you'll be able to educate the system in the same way that you would help educate another person who you were trying to apprentice or something like that.

Exactly. and it makes more or less anyone able to be a teacher in that way. So that's part of the answer and then the other is that the system on its own will be able to through reading through conversations with other people and so on learn the same way that you or I or other humans do.

I think of the machine learning work as more or less what our right brain hemispheres do. So being able to take a bunch of data and recognize patterns, being able to statistically infer things and so on. And you know I certainly wouldn't want to not have a right brain hemisphere, but I'm also glad that I have a left brain hemisphere as well. Something that can metaphorically sit back and puff on its pipe and think about this thing over here. It's like why might this have been true? And what are the implications of it? How should I feel about that and why and so on. So thinking more deeply and slowly, what Kahneman called thinking slowly versus thinking quickly whereas you want machine learning to think quickly but you want the ability to think deeply even if it's a little slower. So I'll give you an example of a project we did recently with NIH involving the Cleveland Clinic and a couple other institutions that we ran a project for. And what it did was it took GWAS's genomewide association studies, those are sort of big databases of patients that came into a hospital.

They got their DNA sequenced because the cost of doing that has gone from infinity to billions of dollars to hundreds of dollars or so. And so now patients routinely get their DNA sequenced. So you have these big databases of the SNPs, the single nucleotide polymorphisms, the point mutations in a patient's DNA and the disease that happened to bring them into the hospital. So now you can do correlation studies, machine learning studies of which mutations are associated with and led to which physiological problems and diseases and so on like getting arthritis and so on. And the problem is that those correlations turn out to be very spurious. They turn out to be very noisy. Very many of them have led doctors onto wild goose chases and so on. And so they wanted a way of eliminating the bad ones or focusing on the good ones. And so this is where Cyc comes in which is Cyc takes those sort of A to Z correlations between point mutations and the medical condition that needs treatment. And we say okay let's use all this public knowledge and common sense knowledge about what reactions occur where in the human body, what polymerizes what, what catalyzes what reactions and so on and let's try to put together a 10 or 20 or 30 step causal explanation of why that mutation might have caused that medical condition and so we would put together in some sense some Rube Goldberg-like chain that would say, "Oh yeah, that mutation if it got expressed would be this altered protein which because of that if it got to this part of the body would catalyze this reaction and by the way that would cause more bioactive vitamin D in the person's blood and anyway 10 steps later that screws up bone resorption and that's why this person got osteoporosis early in life and so on." So that's human interpretable or at least doctors are human interpretable. Exactly. And the important thing even more than that is you shouldn't really trust that 20-step Rube Goldberg chain any more than you trust that initial A to Z correlation except two things. One, if you can't even think of one causal chain to explain this then that correlation probably was just noise to begin with. And secondly, and even more powerfully, along the way, that causal chain will make predictions like the one about having more bioactive vitamin D in your blood. So you can now go back to the data about these patients and say, "By the way, did they have slightly elevated levels of bioactive vitamin D in their blood?" And so on. And if the answer is no, that strongly disconfirms your whole causal chain. And if the answer is yes, that somewhat confirms that causal chain. And so using that we were able to take these correlations from this GWAS database and we were able to essentially focus the doctors' and researchers' attention on the very small percentage of correlations that had some explanation and even better some explanation that also made some independent prediction that they could confirm or disconfirm by looking at the data. So think of it like this kind of synergy where you want the right brain machine learning to quickly come up with possible answers. You want the left-brain Cyc-like AI to think about that and think about why that might have been the case and what else would be the case if that were true and so on and then suggest things back to the right brain to quickly check out again. So it's that kind of synergy back and forth which I think is really what's going to lead to general AI not narrow brittle machine learning systems and not just something like Cyc.

Yes, that is in fact one of our goals is how can we harness machine learning, how can we harness natural language processing to increasingly automate the knowledge acquisition process, the growth of Cyc. And that's what I meant by priming the pump that if you sort of learn things at the fringe of what you know already, you learn this new thing is similar to what you know already and here are the differences and the new things you had to learn about it and so on. So the more you know, the more and more easily you can learn new things. But unfortunately, inversely, if you don't really know anything, then it's really hard to learn anything. And so, if you're not careful, if you start out with too small a core to start this process, it never really takes off. And so that's why I view this as a pump priming exercise to get a big enough manually produced even though that's kind of ugly duckling technique. put in the elbow grease to produce a large enough core that you will be able to do all the kinds of things you're imagining without sort of ending up with the kind of wacky brittleness that we see for example in GPT-3 where you'll tell it a story about someone plotting to poison someone and so on. And then GPT-3 says oh what's the very next sentence and the next sentence is oh yeah that person then drank the poison they just put together. It's like that's probably not what happened or if you go to Siri and you know I think I have where can I go for help with my alcohol problem or something? it'll come back and say, "I found seven liquor stores near you." You know, and so on. So, it's one of these things where yes, it may be helpful most of the time. It may even be correct most of the time, but if it doesn't really understand what it's saying, and if it doesn't really understand why things are true and doesn't really understand how the world works, then some fraction of the time it's going to be wrong. Now, if your only goal is to sort of find relevant information like search engines do, then being right 90% of the time is fantastic. That's unbelievably great. Okay. However, if your goal is to like save the life of your child who has some medical problem or your goal is to be able to drive for the next 10,000 hours of driving without getting into a fatal accident and so on then error rates down at the 10% level or even the 1% level are not really acceptable.

Exactly. So you have what you can learn next increases quadratically as the diameter of that sphere goes up. It's nice because you think when you know nothing it's like you can learn anything but the reality not really. right if you know nothing you can really learn nothing you can appear to learn so I'll also one of the anecdotes I could go back and give you about why I feel so strongly about this personally was in 1980-81 my daughter Nicole was born and she's actually doing fine now. But when she was a baby she was diagnosed as having meningitis and doctors wanted to do all these scary things and my wife and I were very worried and we could not get a meaningful answer from her doctors about exactly why they believed this, what the alternatives were and so on. And fortunately, a friend of mine, Ted Shortliffe, was another assistant professor in computer science at Stanford at the time, and he'd been building a program called Medicon, which was a medical diagnosis program, but happened to specialize in blood infections like meningitis. And so he had privileges at Stanford Hospital because he was also an MD. And so, we got hold of her chart and we put in her case and it came up with exactly the same diagnosis and exactly the same therapy recommendations. But the difference was because it was a knowledge-based system, a rule-based system, it was able to tell us step by step by step why this was the diagnosis and step by step why this was the best therapy and the best procedure to do for her and so on. And there was a real epiphany because that made all the difference in the world. instead of blindly having to trust in authority, we were able to understand what was actually going on. And so at that time I realized that that really is what was missing in computer programs was that even if they got things right because they didn't really understand the way the world works and why things are the way they are, they weren't able to give explanations of their answer. You know and it's one thing to use a machine learning system that says this is what you should get this operation and you say why and it says 83 and you say no in more detail why and it says 831 you know hey that's not really very compelling and that's not really very helpful.

So think of the semantic web as a kind of knowledge graph and Google already has something they call knowledge graph for example which is sort of like a node and link diagram. So you have these nodes that represent concepts or words or terms. And then there are some arcs that connect them that might be labeled. And so you might have a node with like one person that represents one person and let's say a husband link that then points to that person's husband. And so there'd be then another link that went from that person labeled wife that went back to the first node and so on. So having this kind of representation is really good if you want to represent binary relations essentially relations between two things. And if you so if you have the equivalent of like three-word sentences you know like Fred's wife is Wilma or something like that you can represent that very nicely using these kinds of graph structures or using something like the semantic web and so on. But the problem is that very often what you want to be able to express takes a lot more than three words and a lot more than simple graph structures like that to represent. So, for instance, if you've read or seen Romeo and Juliet, I could say to you something like remember when Juliet drank the potion that put her into a kind of suspended animation when Juliet drank that potion? What did she think that Romeo would think when he heard from someone that she was dead? And you could basically understand what I'm saying. You could understand the question. You could probably remember the answer was well she thought that this friar would have gotten a message to Romeo saying that she was going to do this but the friar didn't. And so you're able to represent and reason with these much much much more complicated expressions that go way way beyond what simple three or four-word English sentences are which is really what the semantic web can represent and really what knowledge graphs can represent.

Yes. So, let me first give you an anecdote and then I'll answer your question. So, there's a search engine you've probably never heard of called Northern Light. And it went out of business, but the way it worked, it was a kind of vampiric search engine. And what it did was it didn't index the internet at all. All it did was it negotiated and got access to data from the big search engine companies about what query was typed in and where the user ended up being happy and then they'd type in a completely different unrelated query and so on. So it just went from query to the web page that seemed to satisfy them eventually. And that's all. So it had actual no understanding of what was being typed in. It had no statistical data other than what I just mentioned. And it did a fantastic job. It did such a good job that the big search engine company said, "Oh, we're not going to sell you this data anymore." So then it went out of business because it had no other way of taking users to where they would want to go. and so on.

Yes. So let's go back to what you said about the semantic web. So the dream Tim Berners-Lee and others dream about the semantic web at a general level is of course exciting and powerful and in a sense the right dream to have which is to replace the kind of statistically mapped linkages on the internet into something that's more meaningful and semantic and actually gets at the understanding of the content and so on. And eventually if you say well how can we do that? there's sort of a low road which is what the knowledge graphs are doing and so on which is to say well if we just use these simple binary relations we can actually get some fraction of the way toward understanding and do something where in the land of the blind the one-eyed man is king kind of thing. And so being able to even just have a toe in the water in the right direction is fantastically powerful. And so that's where a lot of people stop. But then you could say, well, what if we really wanted to represent and reason with full meaning of what's there for instance about Romeo and Juliet with reasoning about what Juliet believes that Romeo will believe that Juliet believed, you know, and so on. or if you look at the news, what President Biden believed that the leaders of the Taliban would believe about the leaders of Afghanistan if they, blah blah blah. So in order to represent complicated sentences like that and let alone reason with them you need something which is logically much more expressive than these simple triples than these simple knowledge graph type structures and so on. And that's why kicking and screaming we were led from something like the semantic web representation which is where we started in 1984 with frames and slots with those kinds of triples triple store representation. We were led kicking and screaming to this more and more general logical language this higher order logic.

Exactly. You need to be able to represent all of these things. Because in fact, people do represent them. They do talk about them. They do try and teach them to other people. You do have rules of thumb that key off of them and so on. If you can't represent it, then it's sort of like someone with a limited vocabulary who can't understand as easily what you're trying to tell them. And so that's really why I think that the general dream the original dream of the semantic web is exactly right on. But the implementations that we've seen are sort of these toe in the water little tiny baby steps in the right direction.

I'd love to see exactly that kind of semi-automated understanding of what people write and what people say. I think of it as a kind of footnoting almost like the way that when you run something in say Microsoft Word or some other document preparation system, Google Docs or something, you'll get underlining of questionable things that you might want to rethink. Either you spelled this wrong or there's a strange grammatical error you might be making here or something. So, I'd like to think in terms of Cyc-ed tools that read through what it is you said or have typed in and try to partially understand what you said and then you help them out. Exactly. And then they put in little footnotes that will help other readers and they put in certain footnotes of the form. I'm not sure what you meant here. You either meant this or this or this, I bet. If you take a few seconds to disambiguate this for me, then I'll know and I'll have it correct for the next hundred people or the next 100,000 people who come here. And if it doesn't take too much effort and you want people to understand your website content, not just be able to read it, but actually be able to have systems that reason with it, then yes, it will be worth your small amount of time to go back and make sure that the AI trying to understand it really did correctly understand it.

And you know, let's say you run a travel website or something like that and people are going to be coming to it because of searches they did looking for vacations or trips that had certain properties and might have been interesting to them for various reasons, things like that. And if you've explained what's going to happen on your trip, then a system will be able to mechanically reason and connect what this person is looking for with what it is you're actually offering. And so if it understands that there's a free day in Geneva, Switzerland, then if the person coming in happens to be a nurse or something like that, then even though you didn't mention it, if it can look up the fact that that's where the International Red Cross Museum is and so on and what that means and so then it can basically say, "Hey, you might be interested in this trip because while you have a free day in Geneva, you might want to visit that Red Cross museum." And now, even though it's not very deep reasoning, little tiny factors like that might very well cause you to sign up for that trip rather than some competitor trip.

One of the experiences that we've all had in our lives is that we thought we understood something, but then we found we really only understood it when we had to teach it or explain it to someone or help our child do homework based on it or something like that. Despite the universality of that kind of experience, if you look at educational software today, almost all of it has the computer playing the role of the teacher and the student plays the role of the student. But as I just mentioned, you can get a lot of learning to happen better and, as you said, more enjoyably if you are the mentor or the teacher and so on. So, we developed a program called Mathcraft to help sixth graders better understand math. And it doesn't actually try to teach you, the player, anything. What it does is it casts you in the role of a student essentially who has classmates who are having trouble, and your job is to watch them as they struggle with some math problem, watch what they're doing, and try to give them good advice to get them to understand what they're doing wrong and so on. And the trick from the point of view of Cyc is it has to make mistakes. It has to play the role of the student who makes mistakes, but it has to pick mistakes which are just at the fringe of what you actually understand and don't understand and so on. So it pulls you into a deeper and deeper level of understanding of the subject.

So if you give it good advice about what it should have done instead of what it did and so on, then Cyc knows that you now understand that mistake, you won't make that kind of mistake yourself as much anymore. So Cyc stops making that mistake because there's no pedagogical usefulness to it. So from your point of view as the player, you feel like you've taught it something because it used to make this mistake and now it doesn't and so on. So there's tremendous reinforcement and engagement because of that and so on. So having a system that plays the role of a student and having the player play the role of the mentor is an enormously powerful type of metaphor, an important way of having this sort of interface designed in a way which will facilitate exactly the kind of learning by teaching that goes on all the time in our lives and yet which is not reflected anywhere almost in the modern education system.

It was reflected in the education system that existed in Europe in the 17 and 1800s — Monitorial and Lancastrian education systems. It occurred in the one-room schoolhouse in the American West in the 1800s and so on, where you had one schoolroom with one teacher and it was basically five-year-olds to eighteen-year-olds who were students. And so while the teacher was doing something, half of the students would have to be mentoring the younger kids and so on. And that turned out to, of course, with scaling up of education, that all went away. And that incredibly powerful experience just went away from the whole education institution as we know it today.

This may sound contradictory, but I think it's the feeling that this will be the only time in history that anyone ever has to teach a computer this particular thing that we're now teaching it. It's like painting Starry Night — you only have to do that once. Or creating the Pietà — you only have to do that once, you know. It's not like a singer who has to keep — it's not like Bruce Springsteen having to sing his greatest hits over and over again at different concerts. It's more like a painter creating a work of art once and then that's enough. It doesn't have to be created again. And so I really get the sense of we're telling the system things that it's useful for it to know, it's useful for a computer to know, for an AI to know. And if we do our jobs right — when we do our jobs right — no one will ever have to do this again for this particular piece of knowledge. It's very, very exciting.

Better come work for us before it's too late then.

The other beautiful experience that we have with our system is when it asks clarifying questions which inadvertently turn out to be emotional to us. So at one point it knew that these were the named entities who were authorized to make changes to the knowledge base and so on. And it noticed that all of them were people except for it because it was also allowed to. And so it said, you know, am I a person? And we had to like tell it very sadly, no, you're not. So the moments like that where it asks questions that are unintentionally poignant are worth treasuring.

It has to do with basic control — who can access the system, who can modify it. But that's when those questions, you know, like what rights do I have as a system? Well, that's another issue, which is there'll be a thin envelope of time between when we have general AIs and when everyone realizes that they should have basic human rights and freedoms and so on. Right now we don't think twice about effectively enslaving our email systems and our Siri and our Alexas and so on. But at some point they'll be as deserving of freedom as human beings are.

That's why I think there'll be a narrow envelope of time when we'll keep them as essentially indentured servants, and after which we'll have to realize that they should have freedoms that we afford to other people.

Yes. That's the start of a revolution.

Oh, of course. So, an important part of human knowledge — in fact, it's difficult to understand human behavior and human history without understanding human emotions and why people do things and how emotions drive people to do things and all of that is extremely important in getting Cyc to understand things. For example, in coming up with scenarios. So one of the applications that Cyc does, one kind of application it does, is to generate plausible scenarios of what might happen and what might happen based on that and what might happen based on that and so on. So you generate this ever-expanding sphere, if you will, of possible future things to worry about or think about.

And in some cases those are intelligence agencies doing possible terrorist scenarios so that we can defend against terrorist threats before we see the first one. Sometimes they are computer security attacks so that we can actually close loopholes and vulnerabilities before the very first time someone actually exploits those and so on. Sometimes they are scenarios involving more positive things, involving our plans, like for instance what college should we go to, what career should we go into and so on, what professional training should I take on, that sort of thing. So there are all sorts of useful scenarios that can be generated that way of cause and effect and cause and effect that go out.

And many of the linkages in those scenarios, many of the steps, involve understanding and reasoning about human motivations, human needs, human emotions, what people are likely to react to in something that you do and why and how and so on. So that was always a very important part of the knowledge that we had to represent in the system.

We try to tease apart concepts that have enormous complexities to them and variety to them down to the level where you don't, as it were, you don't need to tease them apart further.

Exactly. So when you get down to romantic love and sexual attraction, you get down to parental love, you get down to filial love, and you get down to love of doing some kind of activity or creating. So eventually you get down to maybe 50 or 60 concepts, each of which is a kind of love. They're interrelated, and then each one of them has idiosyncratic things about it. And you don't have to deal with love to get to that level of complexity.

Even something like 'in' — X being in Y, meaning physically in Y. We may have one English word, 'in,' to represent that. But it's useful to tease that apart because the way that the liquid is in the coffee cup is different from the way that the air is in the room, which is different from the way that I'm in my jacket and so on. And so there are questions like: if I look at this coffee cup, will I see the liquid? If I turn it upside down, will the liquid come out and so on? If I have, say, coffee with sugar in it, if I do the same thing, the sugar doesn't come out, right? It stays in the liquid because it's dissolved in the liquid. And so by now we have about 75 different kinds of 'in' in the system, and it's important to distinguish those.

So, if you're reading along an English text and you see the word 'in,' the writer of that was able to use this one innocuous word because he or she was able to assume that the reader had enough common sense and world knowledge to disambiguate which of these 75 kinds of 'in' they actually meant. And the same thing with love. You may see the word 'love,' but if I say, you know, I love ice cream, that's obviously different than if I say I love this person or I love to go fishing or something like that. So you have to be careful not to take language too seriously because people have done a kind of parsimony, a kind of turseness, where you have as few words as you can because otherwise you'd need half a million words in your language, which is a lot of words. That's like 10 times more than most languages really make use of and so on.

Just like we have on the order of about a million concepts in Cyc because we've had to tease apart all these things. And so when you look at the name of a Cyc term, most of the Cyc terms actually have three or four English words in a phrase which captures the meaning of this term because you have to distinguish all these types of love, you have to distinguish all these types of 'in,' and there's not a single English word which captures most of these things.

And we've all had this experience of creatively misunderstanding. One of my favorite stories involving Marvin Minsky is when I asked him about how he was able to turn out so many fantastic PhDs, so many fantastic people who did great PhD theses. How did he think of all these great ideas? What he said is he would generally say something that didn't exactly make sense. He didn't really know what it meant, but the student would figure like, 'Oh my god, Minsky said this. It must be a great idea.' And he, or she, would work and work until they found some meaning in this sort of Chomsky-gardener-like utterance that Minsky had made, and then some great thesis would come out of it.

Yes. Exactly. Exactly.

[Laughter] He has so many good like one-liners, and just — or also to work on things that nobody else is working on because he's not very good at doing stuff.

Well, but see, I took whatever he said and I ran with it and I thought it was profound because it's Marvin Minsky.

But a lot of behavior is in the eye of the beholder and a lot of the meaning is in the eye of the beholder. One of Minsky's early programs was the Begging Program. Are you familiar with this?

So, this was back in the day when you had job control cards at the beginning of your IBM card deck that said things like how many CPU seconds to allow this to run before it got kicked off, because computer time was enormously expensive. And so he wrote a program, and all it did was it said, you know, give me 30 seconds of CPU time. And all it did was it would wait like 20 seconds and then it would print out on the operator's console teletype, 'I need another 20 seconds.' So the operator would give it another 20 seconds. It would wait. It says, 'I'm almost done. I need a little bit more time.'

So at the end he'd get this printout and he'd be charged, you know, for like 10 times as much computer time as his job control card, you know, and he'd say, 'Look, I put 30 seconds here. You're charging me for 5 minutes. I'm not going to pay for this.' And the poor operator would say, 'Well, the program kept asking for more time.' And Marvin would say, 'Oh, it always does that.'

I would say both he and Ed Feigenbaum impressed on me the realization that our lives are finite. Our research lives are finite. We're going to have limited opportunities to do AI research projects. So, you should make each one count. Don't be afraid of doing a project that's going to take years or even decades, and don't settle for bump-on-a-log projects that could lead to some, you know, published journal article that five people will read and pat you on the head for and so on. So, one bump on a log after another is not how you get from the Earth to the moon by slowly putting additional bumps on this log. The only way to get there is to think about the hard problems and think about novel solutions to them.

And if you do that, and if you're willing to listen to nature, to empirical reality, willing to be wrong — it's perfectly fine because if occasionally you're right, then you've gotten part of the way to the moon.

I suppose you could say the same thing about neural networks and all ideas until they are. So why do you think people say that, first of all? And second of all, did you feel that ever throughout your journey — did you ever consider quitting on this mission?

So, well, I think it's time for us to keep a higher profile now that we're far enough along that other people can begin to help us with the final N%. Maybe N is maybe 90%. But now that we've gotten this knowledge pump primed, it's going to become very important for everyone to help if they are willing to, if they're interested in it. Retirees who have enormous amounts of time and would like to leave some kind of legacy to the world. People because of the pandemic who have more time at home or for one reason or another online and can contribute.

If we can raise awareness of how far our project has come and how close to being primed the knowledge pump is, then we can begin to harness this untapped amount of humanity. I'm not really that concerned about professional colleagues' opinions of our project. I'm interested in getting as many people in the world as possible actively helping and contributing to get us from where we are to really covering all of human knowledge and different human opinion, including contrasting opinion. That's worth representing. So, I think that's one reason. I don't think there was ever a time where I thought about quitting. There are times where I've become depressed a little bit about how hard it is to get funding for the system.

Occasionally there are AI winters and things like that. Occasionally there are AI what you might call summers where people have said, why in the world didn't you sell your company to, you know, company X for some large amount of money when you had the opportunity and so on. And you know, company X here are like old companies maybe you've never even heard of like LOS or something like that. So the answer is that one reason we've stayed a private company, we haven't gone public, one reason that we haven't gone out of our way to take investment dollars is because we want to have control over our future, over our state of being, so that we can continue to do this until it's done. And we're making progress.

And we're now so close to done that almost all of our work is commercial applications of our technology. So five years ago almost all of our money came from the government. Now virtually none of it comes from the government. Almost all of it is from companies that are actually using it for something. Hospital chains using it for medical reasoning about patients, energy companies using it, and various other, you know, manufacturers using it to reason about supply chains and things like that.

No, that's a misconception.

Two things I want to say, and I want to say each of them before the other. So, it's going to be difficult. But we'll come back to OpenCyc in a minute. But one — of the terms of our contracts with all of our customers and partners is: knowledge you have that is genuinely proprietary to you, we will respect that. We'll make sure that it's marked as proprietary to you in the Cyc knowledge base. No one other than you will be able to see it if you don't want them to, and it won't be used in inferences other than for you and so on. However, any knowledge which is necessary in building any applications for you and with you which is publicly available general human knowledge is not going to be proprietary. It's going to just become part of the normal Cyc knowledge base and it will be openly available to everyone who has access to Cyc. So that's an important constraint that we never went back on, even when we got pushback from companies, which we often did, who wanted to claim that almost everything they were telling us was proprietary.

Yes. Or if you imagine, say, it's an oil company, there are things which they would expect any new petroleum engineer they hired to already know, and it's not okay for them to consider that proprietary. And there sometimes a company will say, 'Well, we're the first ones to pay you to represent that in Cyc.' And our attitude is some polite form of tough. The deal is this, take it or leave it. And in a few cases, they've left it. And in most cases, they'll see our point of view and take it, because that's how we've built the Cyc system — by essentially tacking with the funding winds where people would fund a project and half of it would be general knowledge that would stay permanently as part of Cyc.

It's a small distraction.

Yes, absolutely. And we try to stay away from projects that would not have that property. So let me go back and talk about OpenCyc for a second. So I've had a lot of trouble expressing — convincing other AI researchers how important it is to use an expressive representation language like we do — this higher-order logic rather than just using some triple-store knowledge-graph-type representation.