Well hello, hello. Is the mic working? Check, check, check, one, two, three. All right, first hard technology problem of the day down. Yeah, yeah, well the Wi-Fi is the other one. Everyone here knows... So Greg, welcome to AI Engineer. Thank you so much for taking the time.
We're going to go a little bit chronologically. A lot of people sent in questions and I've sort of grouped them up for you, so we'll just get right into it. So you know, I did some deep research on you. You started with deep research... I called it 'Peep research' because you were researching a person. You actually did theater growing up, and chemistry and math, and you wrote a calendar scheduling app and that's what got you into coding. But what really inspired your love for coding? Why are you the coding guy?
Well, the funny thing is I thought I was going to be a mathematician when I grew up. You know, I'd read about people like Galois and Gauss, working on these hundred, two hundred, three hundred year time horizons, and I was like, that's what I want to do. If anything that I come up with is ever used while I'm still alive, it wasn't long-term enough, it wasn't abstract enough. And I was writing this chemistry textbook after high school, sent it to one of my friends who'd done something similar in math, and he said, 'No one is going to publish this. You can either self-publish.' I was like, 'Ah, sounds like a lot of work, a lot of capital.' Or you could make a website. And I was like, 'Guess I'm going to learn how to make a website.' So I literally went on W3 Schools and did their PHP tutorial. How many people here remember W3 Schools? Yeah, decent number of hands. And I remember the very first thing I built was a table sorting widget. I had this picture in my head of what it would be, and I remember the moment that I clicked the column and it sorted according to that column, which was exactly the thing that I wanted, and I was like, that was magic. And I was like, this is so cool because the thing about math is that you think hard about a problem, you understand it, you write it down in an obscure way, you call it proof, and then like three people will ever care. But in programming, you write it down in an obscure way, we call it a program, and then maybe only three people ever read that program and care about the code, but everyone gets the benefit. No one has to understand the details. That thing that was in your head, it's real, it's in the world. And I was like, that's the thing I want to do. Forget about that hundred-year time horizon. I just want to build.
You do just want to build. You were so good at it that somehow, somewhere, you got cold emailed by Stripe while you were still in college. That's right. What's the story? How first of all did they find you, and what was it that convinced you to drop out to join them?
Well, I had mutual friends with all the people at Stripe, the giant company of like three people at the time. And they had asked, you know, the usual thing where they'd asked someone at Harvard who the people around campus to talk to, who they might recruit, and my name came up. They asked the same for the people at MIT because I actually had dropped out of Harvard to go to MIT, so I had the advantage of getting up votes on both sides. But I remember when I met Patrick, and it was late at night, it was storming, and I showed up and we just started talking about code. And it was just like one of those moments where you're like, this is the kind of person that I've wanted to work with and been looking for. So I ended up dropping out of MIT, flew out, and been out here ever since.
We have some guest questions sprinkled along the way. Question from someone named Matthew Brockman, I've heard of him, CTO of Julius AI. 'When do you think our parents will give up on the dream of you finishing your degree? Maybe Harvard or UND will take you back.'
Yes, well, never. It was definitely... I think it was no matter where you're going, if you tell your parents you're leaving Harvard, it's going to be hard. You tell your parents you're leaving school altogether, it's going to be difficult. And I think that, to their credit, even though it was difficult, they were like, we trust you, you must see something and understand something from where you sit that's hard for us to see from halfway across the country. But yeah, I think that as you know, I did Stripe and had a good time and learned things, and it turned out as a real company and not just dropping out doing nothing. I think they really have warmed up to it. I think they're very proud of you.
Absolutely. So you were with Stripe from 4 to 250 people as the first CTO eventually. One thing I found recently that Hacker News maybe doesn't know is apparently the call installation only happened like a handful of times, it wasn't like a thing at Stripe.
I think that's true. It is the thing that survived, it's an urban legend because it's so cool, it's like you're so customer obsessed. Anyway, what else do people get wrong about early Stripe? Why do we want to clear the air?
Yeah, well, I think people don't understand how hard it was. Like, I remember, first of all, the kind of thing that we did a lot is that we added all of our customers on Gchat, and so it was very much the case that we were in constant contact with them. So even if you're not literally sitting over their shoulder, you're doing the next best thing. But I remember one day we realized that the payment backend that we were on just wasn't going to scale. We absolutely needed to be on Wells Fargo, and we got the deal done, but now we need to do a technical integration, and they said this technical integration is going to take like nine months because that's how long it takes. And we're like, that's crazy, we can't sit around waiting nine months to get this thing done. So actually in 24 hours we completed it by just basically treating it like a college problem set. I was implementing everything, John was working from the top of this test script and testing everything and being like, this is broken. Daryl was starting from the bottom and working his way up. In the morning we got on with the certifying person and we sent some test messages and there was an error, and the person's like, 'All right, I'll see you next week,' because that's how all their customers operate. There's an error, you need to send it to your dev team. And we were like, no, no, no, there must just be some sort of glitch in the system. Patrick was just talking to keep her on the line and frantically I was there editing the code. We got like five turns in and we actually failed, but fortunately she was nice enough to reschedule two hours later, and then we passed. So you realize that was like six weeks worth of normal dev work that you got done in that moment because you didn't just accept the arbitrary constraints of how other organizations would work.
Yeah, yeah. Do you think there's a lot more opportunity like that in most jobs? How do you advise other people to be that fast or to cut that many cycles?
Yes, I mean, I think the way I think about it is that if you think from first principles, you can find where things need to be slow or done the way that they're normally done. Those things exist. The general principle of 'just don't worry about the constraints and just do the thing' is not 100% true. I think it's really about mapping to where is there unnecessary overhead that's there for constraints that are no longer applicable, that don't apply to your specific circumstance. And I think this is especially true in this world that we're in now with AI that's accelerating productivity so much. Yeah, just fire off a codex, why not?
One last thing about your pre-OpenAI life was independent study. I just found that it's a recurrent theme from high school, you did it in college, and your sabbatical as well. You've just done it repeatedly. What makes independent study effective? I think there's a lot of people who don't do a good job of it and kind of waste a year. What do you do that makes it so effective?
Well, I think it was a key part of how I grew up. In sixth grade, my dad taught me algebra. In seventh grade, I showed up at the high school as the first time you track into advanced math, pre-algebra, and we went to the teacher like, 'Can he skip this and go directly to the eighth grade course?' And the teacher looked at my mom and me very condescendingly and was like, 'Every parent believes that their child is special.' After a month of being in this teacher's class, I was paying no attention and just doing calculator games in the back, and she'd try to trip me up and call me to answer questions from the whiteboard, and I would just get them all right. She was like, 'All right, fair enough, your child should be in the next year.' But then when I was in eighth grade, there was no more math left in my middle school. I didn't have a car, so I had to do online courses, and in that one year I ended up doing three years worth of high school math. So I think that for me, a lot of it is about suddenly, if you're excited about something independently, it's something you want to do that you can break the constraints there as well. You can do three years of math in one year, and then it compounds because the next year I was at my high school, finished math there, and then all through 10th, 11th, 12th grade, I had no more math, so I did have a car and I was able to go to University of North Dakota and take whatever classes I wanted there. So I think that that kind of compounded to learning programming, and the way I learned programming is very much self-study, just building things and experiencing things out in the world. So I think the thing I would advise is if you have an opportunity to explore and you have a passion you're actually enjoying, just go deep. And by the way, it's not always fun. I think it is very easy to feel like you got kind of bored, but if you just push through those hurdles, then I think the reward is worth it.
You self-studied machine learning too. That was a whole period of your life. Any particular highlights from there? It sounds like you talked to Geoff Hinton at one time.
I did talk to Geoff Hinton, yes. And like, was that helpful? Or what was the most helpful thing? You became a machine learning practitioner.
Well, so when I started out, I'd been at Stripe, I was reading Hacker News posts about deep learning, and it was like there's a deep learning for X every day, it felt like, and this was 2013, 2014. And I was like, what is deep learning? And I knew like one person in the field, so I talked to them, they introduced me to some more people, and then they introduced me to more people. And the thing that surprised me was I kept getting introduced to a bunch of my smartest friends from college, and I was like, that's interesting, all of these people ended up in this field, what's going on? And I started to realize that there was something real that was building, that people were really making these systems do material new things that computers were not able to do before. And I was like, that is the thing. So after I left Stripe, I knew I wanted to do something in AI, start an AI company, but I didn't really know how to contribute, what my skills would be useful for. So I was in New York and I was like, you know what, I'll build a GPU rig and see if I can do some Kaggle competitions. So I went on Newegg and bought some Titan X cards. It was really cool, physically assembling this machine. You can find some tweet from 2015 when I powered it on, you see all this green and all the fans going, and I was like, this is what computers are meant to be. I think many folks in the audience have that experience as well.
Awesome. Okay, so what convinced you that AGI was possible? You had a point where you were sort of disillusioned with it. You tried to write a chatbot, it didn't work. But what made you go all in on it?
Yeah, well, part of the journey for me was reading Alan Turing's 1950 paper, 'Computing Machinery and Intelligence.' This is the Turing test paper. How many people have read it? Fewer hands than W3 Schools, but equally as important, worth reading. The thing that is so fascinating to me is he lays out in the beginning, okay, Turing test, this idea of just does a machine think, is it intelligent? And you can say it's intelligent if a human can't tell the difference between talking to it and talking to a human. Fine. But the thing that has not really become as embedded in pop culture, but to me was so astounding, was he said, 'Well, how are you going to program an answer to this? You will never be able to write down all the rules. But what if you could build a child machine that learns like a human child, and then you just apply rewards and punishments, and boom, it's going to be able to pass the test.' And I was like, that is the kind of technology that we have to build. Because as a programmer, you have to understand everything, you have to understand the rules of how to solve the problem. But what if the machine can understand things and solve problems that you yourself cannot understand? That feels fundamental, that feels like how you actually solve problems that are important to humanity. And this was 2008 or so that I read this, and I went to my professor, who was an NLP professor, and I asked if I could do some research with him, and he said, 'Yeah, here are some parse trees.' And I was like, okay, this is not what Turing was talking about. This is like word nets and the whole thing. So it's definitely a little bit of trough of sorrow there. But with deep learning, the thing about deep learning that's magic is that it really started to show promising results in 2012 with AlexNet, and it just blew everyone out of the water in the ImageNet competition. So suddenly you have this general learning machine. It's got a little bit of a prior in there of convolutions, but it's better than 40 years worth of computer vision research, people trying to write down all the rules as well as possible. And then people are like, well, okay, it works in vision, but it's never going to work in my field, it's never going to work in machine translation, never going to work in NLP, never going to work in this or that. And suddenly it starts being the best in all of those areas. Suddenly the walls between these departments are being torn down, and you're like, that is what Turing was talking about. So I think for me, just seeing the type signature of this technology, and by the way, this technology is not new. Neural nets were really... if you go back and read the McCulloch-Pitts neuron paper from like 1943 or so, I told people to give homework to people, okay, yeah, there you go. The images in there look just like the kinds of images that you see now, just layers of neurons and things like that. So you just realize there's something deeply fundamental about what we're doing. And you can find this paper from the 1990s talking about what caused the deep learning winters, and that it was these neural net people, they have no new ideas, they just want to build bigger computers. And I'm like, yes, that's what we need to do. So I think that all of this together just feels like we are to some extent continuing this wave, this 70-year history. And in many ways, the whole computing industry has been really trying to build up to the point that you can have machines that are able to perform the kinds of tasks that we're just starting to scratch the surface, to solve new problems that humans cannot, to be assistive to us in our daily lives, to not have to be typing with our meat sticks, but instead to have something that you can interact with just like a person, where the machine comes much closer to you rather than you closer to it, having to learn assembly language or whatever it is. So to me, it felt like all of the factors were lined up and now we just need to build.
I like that consistent theme that you keep coming back to: we just need to build. So in 2022, you wrote that it's time to be an ML engineer. Actually, I have a personal friend who read that post and cold emailed you and joined OpenAI and all that. You said that great engineers are able to contribute at the same level as great researchers to future progress. Is that still true today? You know, I think a lot of engineers look at the researchers who are making millions of dollars and they're like, how do I contribute as much?
I think it's absolutely, if not even more true. I think that if you look at the phases of deep learning research since 2012, I think at the beginning it really was, and this is kind of what I expected when we started OpenAI, just research scientists who had gotten a PhD who would go and kind of come up with ideas and test them out. And there's engineering to be done. If you actually look at AlexNet itself, it's fundamentally the engineering of let's get fast convolutional kernels on a GPU. And fun fact, people who were in the lab with Alex at the time actually felt very bad for him because they were like, he has some fast conv kernels for some image dataset that doesn't really matter. But Ilya was like, well, clearly we just need to apply this to ImageNet, it's going to be great. So it's the combination of great engineering together with the idea of what to do with it that makes the magic work. And the thing that I think is still true and even more true is, okay, so the engineering required now is not just let's build some kernels, but let's build a system, let's actually scale to 100,000 GPUs, let's actually do this crazy RL system that orchestrates things in all sorts of ways. So the idea, if you don't have the idea, you're dead in the water, there's nothing to do. But if you don't have the engineering, that idea is not going to live and see the light of day. So you need to have both of these coming together harmoniously.
Yeah, I think that Ilya-Alex relationship is really emblematic of the research-engineering partnership that now is the philosophy at OpenAI.
That's right. And if you look at how OpenAI operates, from the very beginning we had this ethos of engineering and research being valued and working together as partners. And I think that is something that we really work at every day.
It's my explicit goal to try to throw curveballs in this stuff. So in terms of the relationship between engineering and research, what did OpenAI do wrong in the early days that you do well now?
