Universally Accessible Intelligence with Character.ai's Noam Shazeer

Our goal is to be like an AGI company and a product-first company, and the way to do that is by picking the right product that forces us to work on the right things.

Hey, I know him. That is all right. I don't know what I'm going to be answering.

Sounds good. Okay.

Oh, yeah. They... I went on the basketball... no, I'm joking. They recruited me because they wanted to win the math competition, so I ended up going on scholarship.

Well, I've always been excited about AI. I just wanted to push technology forward. I saw this large language model thing—well, at the time it wasn't large language models, but neural networks and language modeling was just the most promising thing on Earth. It's the best problem ever. All you have to do is predict the next word, and yet once you do it, you get something that can just talk to you about anything. So I knew you can make this technology better in a lot of ways. It's going to be amazing. We can improve it with model architecture, distributed algorithms, quantization, all of these things. I was working on that, but then it struck me: the biggest thing is just scale. Can you throw a billion dollars or a trillion dollars at this thing instead of a million dollars? And what you need for that is to show a massively valuable application. There are a few options. Coding looks super useful, but also dialogue—it's the world's number one pastime. This was back in 2019, 2020. My co-founder Daniel De Freitas has been on this lifelong mission to do chatbots and dialogue systems. He had built this system he called Meena, it was like his 20th project, and it was really good. I thought, okay, gotta help Daniel, let's build something amazing. We got it on giant models, it was this internal viral sensation at Google, and then they forcibly renamed it to LaMDA over some sort of social justice thing. But anyway, at some point I realized there's just too much brand risk in large companies to ever launch anything fun. So let's do a startup and maximally accelerate.

Well, I think we just need a sort of global pause of about six months—no, about four months—until we get enough H100s online to train our next model.

No, no. Yes, there's a lot of potential benefits, and we're going to work on those as the technology improves.

Oh, that's pretty good. Yeah, that's better than I would do.

Yeah, better. Some of the big unlocks we're working on are just training a bigger, smarter model. The scaling laws are going to take us a pretty long way. The model we're serving now cost us about two million dollars worth of compute cycles to train last year, and we could probably repeat it for like half a million now. So we're going to launch something tons of IQ points smarter, hopefully by the end of the year. Also, more accessible—multimodal, maybe you want to hear a voice and see a face. And then also able to interact with multiple people. We want a virtual person in there with all your friends, or the experience like you got elected president and you get the earpiece and the whole cabinet of advisors. Or like you walk into Cheers and everyone knows your name and they're glad you came. There's a lot we can do to make things more usable. Right now, the thing we're serving is using a context window of a few thousand tokens, which means your lifelong friend remembers what happened for the last half hour. And still, there are a lot of people who are using it hours a day. So that will make things way better, especially if you can just dump in massive amounts of information. It should be able to know a billion things about you. The HBM bandwidth is there, we just need to do it.

I think the way to understand this is that entertainment is a two trillion dollar a year industry, and the dirty secret is that entertainment is imaginary friends that don't know you exist. The reason people interact with TV or any of these other things is these parasocial relationships—your relationship with TV characters, book characters, celebrities. Everybody does it. There are billions of lonely people out there. So it's actually a very cool problem and a cool first use case for AGI. There was the option to go into lots of different applications, but a lot of them have a lot of overhead and requirements. If you want to launch something that's a doctor, it's going to be a lot slower because you want to be really careful about not providing false information. But a friend? You can do that really fast. It's just entertainment. It makes things up—that's a feature. So essentially, it's this massive unmet need. One thing that's very important is that the thing kind of feel human and be able to talk about anything, and that matches up very well with the generality of large language models. And one thing that's not a problem is making stuff up. So perfect. And if I want to push this technology ahead fast, that's what I want to go with, because it's ready for an explosion right now, not in five years when we solve all the problems.

Exactly. People like that human experience of very mixed use cases, talk about everything. It's not that we want to fine-tune to some particular domain or use case. People want this experience of everything, which is what the technology is perfect for.

Yeah, the more you get to mission-critical particular use cases, the more you get tempted into writing particular rules and doing things that will not generalize well. So it was kind of important to stay away from that. Our goal is to be like an AGI company and a product-first company, and the way to do that is by picking the right product that forces us to work on the right things—things that generalize, make the model smarter, make it do what people want, and serve it at massive scale and cheaply. So I think this was the right product for the right goal.

I love being a full-stack company. It means we get to mess with every layer and do the code design. If there's something that's going to affect something at the end, we get to mess with it at the beginning. We get to pull in lots of user data as feedback. A lot of us invented this stuff, so of course we're going to do a full-stack company. A lot of us are motivated by launching. People who are attracted to work at Character.AI are people who love inventing stuff and love launching it. Some people are motivated by publishing. I was frustrated I couldn't launch at Google, so that's where I'm coming from.

Just make the things smarter, it's going to have a better theory of mind. I think that's definitely something massively important. It seems like one of these emergent properties that is just going to come with scale. But I see this stuff massively scaling up. It's just not that expensive. I saw an article yesterday that Nvidia is going to build another one and a half million H100s next year. So that's 2 million H100s. That's 2 times 10 to the 6th, times they can do about 10 to the 15th operations per second, so 2 times 10 to the 21. Divide by 8 billion people on Earth, that's roughly a quarter of a trillion operations per second per person. That means it could be processing on the order of one word per second on a hundred billion parameter model for everyone on Earth. But really, it's not going to be everyone on Earth because some people are blocked in China and some people are sleeping. But things are not that expensive. This thing is massively scalable if you do it right, and we're working on that.

Yeah, I think it's like a Wright brothers first airplane kind of moment. We've got something that works and is useful for a large number of use cases, and it looks like it's scaling very, very well without any breakthroughs. It's going to get massively better as everyone scales up to use it. And there will be more breakthroughs because now all the scientists in the world are working on making this stuff better. It's great that all this stuff is accessible as open source. We're going to see a huge amount of innovation. What's possible in the largest companies now can be possible in somebody's academic lab or garage in a few years. The technology gets better, and there are going to be all kinds of great use cases that emerge, pushing technology forward, pushing science, pushing the ability to help people in various ways. I'd love to get to the point where you can just ask it how to cure cancer or something. It seems a few years away for now.

I don't know. It's impossible to predict the future, but I don't think anyone has seen these scaling laws stop. As far as anybody has experimented, stuff just keeps getting smarter. So we'll be able to unlock lots and lots of new stuff. I don't know if there's an end to it, but at least everybody in the world should be able to talk to something really brilliant and have incredible tools all the time. I can't imagine that that will not be able to build on itself. And definitely, at the core, the computation isn't that expensive. Operations cost like 10 to the negative 18 these days. If you can do this stuff efficiently, even talking to the biggest models ever trained, the cost of that should be way lower than the value of your time or most anybody's time. There's the capacity there to scale these things up by orders of magnitude.