Something I've learned is that when people are willing to push to the extent of something, to push to their limit, to be unreasonably intense about something, to continually take on difficult problems, usually these people are defined by some kind of experience. Something happened in their lives that made them that way. Something gave them a chip on their shoulder. Something happened that changed their internal world and allowed them to see life in a way that is different from how other people see it. It's just really interesting to me to talk to people and learn what that is.
In this episode of The Other Stuff, I had the opportunity to speak with Aidan Gomez, who is a very talented computer scientist and also the CEO of Cohere. In this conversation, I thought we were going to talk about things like the company he's creating in Canada. I thought we were going to talk about the scientific papers that he's worked on. A notable one is 'Attention Is All You Need.' And we did talk about those things. But one of the things we talked about was this small town he grew up in just north of Brighton, Ontario. And how that experience growing up there informed his entire life. It's not just because of the place. It's a remote place with a very small population. But it's not just because of that. It's because of how his family ended up there and what they were processing at the time. I think it's a place that represents his family defining who they are and their values. I think it's a place where his family was processing some kind of grief. And I think that experience defined everything about who he is, or a lot about who he is, everything he's done up until this point. And I think it will define everything he does for the rest of his life. It's always really fascinating to talk to people who are not only working on interesting things, but have had these interesting life experiences that shape them and how they see the world. I think we all have this in some way. And it was cool to learn from Aidan about his past and the way he grew up. Very interesting individual. This was a great conversation and I think I'm just scratching the surface on who Aidan Gomez is. I would love to keep talking to him and keep learning and seeing how he processes and thinks about things. I hope you enjoy this episode. Thank you for everything.
First of all, thank you for doing this. I'm so stoked to be here. I've missed Toronto. I've been away for too long. I'm living in London now, in the UK. But I've been watching from the outside what you've been doing, what others have been doing. And I have so much FOMO. It's just so cool what's happening here. I miss this city. I don't know when it changed, my sentiment towards Toronto when I lived here. It was home. It was warm and cuddly, but you would describe it as like New York but shitty. New York but worse. But I don't describe it like that anymore. There's something, there's a bunch of things really starting here and starting to bubble up. And I think you've been a huge part of catalyzing that, inspiring people.
Thanks, man. I think with Toronto, the thing I started to believe was that it's not as fixed as people think it is. That's the weird thing about it. When you think of a city, you think it's this existing cultural idea that you're participating in. The reality is that at least in Toronto, it's very malleable, far more malleable than people think. I'm also learning that your relationship with a place has less to do with the inherent qualities of that place and more to do with whether you have a sense of meaning there, whether you have friends you can hang out with and crack jokes with, whether there's significance to what you're pursuing in that space. For me, it's over money or clothes or a nice car. What I want to know is, do you have the capacity to walk into a random plaza in Mississauga and be like, 'Yo, I know a spot to get Korean fried chicken that you have no idea about,' and the Korean fried chicken is really good? And not only that, you don't know about this spot. To me, that became the top of the hierarchy. If you can shift what is valuable and what is not valuable, then it really shifts your relationship to a place. I think a lot of people in the technology scene had a hierarchy projected onto us: this is what is good, this is what is not as good. If you're building a startup, you should go to San Francisco. If you stay here, it's not as good. So we started shifting the hierarchy relative to what we wanted to do. We started being excited about it and not shying away from the idea. We just don't want to participate in that existing hierarchy anymore. Whether people look at what we're doing and think they want to do it or not, or if they see it exactly the way we see it, that doesn't really matter. It reveals that you don't have to be subject to someone else's perception of a place. You can just start making it your own. It's really interesting to see what happens when you do that.
Yeah. You can be the author, the sculptor. I think Toronto is still a city that's being built. Its tone hasn't been set. Its brand is still malleable, like you say. The people in it were so hungry for a story, for action, for someone to come in and start doing things and organizing people, not towards any specific objective but more towards a cultural one. Like setting a new tone. I think you were the one who really started that trajectory and made people proud to have come out of the Toronto tech scene. It's such a culture-setting city. It's defined a lot of what's cool. It makes it exciting to be here because you can be a founder of the city. You can be one of the ones that really built it. It's not crowded. There aren't a thousand startups that come through here every quarter. There's no startup mill. Instead, it's a small community of really ambitious, high-taste individuals trying to build stuff, while also being the fourth largest city in North America.
That's crazy. I didn't know that.
Yeah. Right, Robbie? I don't even know. Maybe there's still so much upside.
Oh, am I wrong about that? You got to get in an order because that just says top.
I think it is because... Yeah. There you go. Fourth.
Yeah. Like what do you mean?
No, not even close. Even though it's the fourth largest city, there's still so much headroom to grow into. There's so much good in the future. It's not like the juice is squeezed. It is what it is. Which is kind of how I feel about a lot of other big cities. They're defined, you already know what they are. They're set in stone. But this place is so malleable. The story is still being written.
So I think when you are actively participating in this, you have a company, you're CEO of a company in Canada. You're participating in it in a very real way. A pattern I'm noticing is that when I talk to founders who are doing stuff in the city, they all get it. They're not just here because they think, 'This is my city and I want to participate in it.' They actually know something that other people don't know, which is that there's a higher potential for return. It's an advantage to be here. There's so much more opportunity here. You can only truly understand that when you start to do stuff and observe the feedback. You realize, 'Oh, there's so much talent here, but not that many interesting projects for them to work on. So if I create an interesting project, a lot of people will value that.'
Yeah. It's so high alpha. There's so much talent. Canadian universities in general just keep pumping out generations of the best computer scientists on the face of the planet.
Before we talk about that, you grew up in Brighton.
Brighton, Ontario. That's interesting. And your dad was like a mathematician or something?
No. My dad, he was interested in math and physics. He was a high school teacher in math, physics, and biology.
And your mom was an artist.
She was an artist and she studied dance. She was also a librarian. And she taught Pilates.
She taught Pilates like early days.
My mom, for me, my brother. If she was in Pilates early on, she's actually kind of a genius.
Yeah. No, she was. Because if you're in early, you're in it because of Joseph Pilates. You're not in it because Pilates is a thing. You're in it for the correct method.
Yeah. She was like the first one there.
So, let me just zoom out of Brighton. So, that's Brighton.
So, how far is this from Toronto? It's like an hour and a half away.
Is this like Prince Edward County?
Yes. It's above Prince Edward County, but super close. All my high school girlfriends worked at the wineries in Prince Edward County.
Oh, okay. So, I'm getting a feel for this. I can kind of understand this. So, where is Prince Edward County?
Yeah. Well, there was only like one, right?
Yeah, there is only one for 50 kilometers in any direction. We actually tried to call the principal. I think he's going to get back to us.
Yeah. So there's only one, East Northumberland Secondary School. We called it ENSS.
Okay. So, you grew up in this place called Brighton. So, this is kind of a remote place, right?
Yeah. It's on the 401, so it's pretty connected, but I didn't actually grow up in Brighton. Brighton is a township of a bunch of little mini towns. I grew up north of Brighton, close to Codrington.
Okay. So, you grew up in this smaller town. So, the story is my dad and his dad. My grandma, my dad's mom, she passed away. Both of my grandparents and my dad, they were immigrants from Spain to Toronto. They were like the only Gomez in the phone book at that time. They kind of only had each other. There weren't a lot of Spanish people in Toronto back in those days.
Your dad and his dad.
Yeah. Then when my dad was like 21 or 22, his mom died. My grandpa, he just sort of opted out of the city. He didn't want to live in Toronto anymore. So my dad took a huge loan and bought 150 acres of forest just north of Brighton, close to Codrington. They went there, cut down some trees, and built a cabin together, the two of them. Then his dad moved out there. My dad stayed in the city and kept teaching. But my grandpa, when his wife died, he moved into the woods in this cabin. He didn't speak English and he's moving into the whitest little farmer's village. He plowed some soil, made a little garden, dug an outhouse, and had a little wood furnace for heating. Somehow he survived with a cat and a dog in the middle of the woods, in a place where he didn't speak the language, by himself. I find his story really inspiring. He's so... if you want to talk about someone who's a max autonomy individual, this guy was wild. He fought in World War II, became a vegan after that. In his house in the woods, there were flies and bugs. He would catch them in his hand and not kill them, and then let them outside. He wouldn't hurt anything. He was so kind and peaceful, and he was largely alone and independent. When his wife died, he just said, 'I'm going.' I find the metal you need to have to thrive and live like that so inspiring.
Yeah, for sure. Me too. Do you think there was something when his wife died that was an inciting incident between your dad and your grandfather? Like, why do you think they decided to do that?
I just think my grandpa wanted to be alone. The woman he loved and built a life around died, and he wanted to be alone. So my dad helped him do that. Eventually, my dad and his dad had this extreme... my dad loved my grandpa so much. My grandpa is my hero, my brother's hero, my dad's hero, my mom's. So when my dad and my mom had me, when I was one year old, they left the city. My dad and my grandpa cut down trees and built a second cabin. From one year old, that was my house, a second cabin on this property. So I grew up with my grandpa my whole life.
Wow. So this is hugely influential on who you are. So your relationship with Brighton, when I look at it, I think, 'Oh, this is some small town.' But for you, you grew up in a very different way. This was an act that your family took. This was a processing.
Yeah. If you turn around, that's beautiful, man.
Keep turning. That is the library that my mom works at still.
That's it. The interesting thing is when people look at this, any normal person will think, 'Yeah, you just grew up in a small town.' But that's not what happened.
Yeah. You grew up in a process.
Yeah. And I didn't appreciate it at all. I thought it was normal. You grow up from one year old in the middle of the forest in a cabin your dad built, you think everyone has this experience. In the summer and spring, you go out into the woods, look for trees that naturally die, cut them down with a chainsaw in the spring so they bake dry over the summer. Then in the fall, you chop it into rounds, load them in the truck, bring them to the house, split them with an axe into logs, and stack them to dry out more. That's what you have to do because in the winter, the only way to keep your house warm is with a wood furnace. It's like that or die. So that's what you grew up doing. My dad did most of it.
Oh, damn. That's a deep cut. What is this from? 2019.
Yeah. This is me showing Ivan how to chop some wood.
Wow. That's like a perfect chop.
Thank you. So you grew up like this. But at some point, you would have realized this is not a normal way to grow up. You thought it was normal, right?
Yeah. Once I got to school and hung out at friends' houses, even in Brighton, it was odd even for Brighton. Nobody lived like me other than me. Even the farm kids had internet. We had no internet. We were in a dead spot, so we only got a phone line for dial-up. Later we got satellite, pre-Starlink, so it was the most shitty internet you can imagine. You could watch photos load pixel by pixel. I would go to my friend Nick's house and his internet was so fast. I was so pissed off. I was like, 'Why the fuck do we live here? There's no fucking internet. I can't play COD with my friends.' I was so mad. But now I realize how stupid I was and how lucky I was.
Why do you think you were lucky?
I just don't think many people get to grow up like that. Not a lot of people have the option to just run outside naked in the woods, but I did.
What was your day like? A normal day. You had a wood furnace for a while.
Yeah. Up until I left the house. They put in a natural gas heater, but I was already at uni by then.
Oh, so in high school it was a wood furnace. That's crazy, man.
So what was your normal day like? Getting ready for school.
Wake up, shower maybe. It's a septic tank. There's a well with hard water, so you have a softener. So much shit would happen. The power would go out randomly if a storm knocked out the power lines. My driveway from the road to my house is a kilometer and a half long. We had a snow plow for our driveway. My dad would attach a plow to the front of whatever car. At some point it was a Kia Sportage with a snow plow on the front. He would clear the driveway with that. Also on a Dodge Caravan minivan. We had this old Chevy truck from the 70s. It finally died, and we hooked up a Dodge Grand Caravan.
How is this possible that this is the way you grew up? You must have been pissed.
Yeah, as a teenager, for sure. I went through phases. Before I turned 13, my house was the sickest house. Paintball guns, BB guns, we'd be out in the woods shooting shit. Friends' parents were cool with it. We had machetes and dirt bikes on 150 acres of property to explore. We knew the whole area. We'd build a teepee, start a little fire, survive. We were 10 years old. I remember with my friend Nick, we had a sleepover on the weekend and didn't want to leave. So we said we're going to camp out in the woods. Take some string cheese. 'You're not going to find us. We're just going to live here. This is our new life. We live 200 meters from the house, and we're going to figure this out. We can sneak in when my parents are asleep, get more cheese strings, and we'll be fine.' So up until I was 13, it was the best. Then from 13 to when I graduated, it was hell. There's no internet. I can't talk to anyone. Everyone has whatever.
So that was Instagram. When you're growing up, a lot of the way you interact with the world is by playing outside. But at a certain point, people start using the internet, and there's this dual social layer. There's the reality you're moving through, but then this other layer of the internet that people have access to. Your friends had access to it, and you didn't.
Yeah, I was pretty much cut out. But I tried really hard. Even with dial-up, the reason I learned to code was because I thought, 'There's no way my computer is meant to be this slow. There's something wrong with it. I'm going to figure out how to make my computer faster.'
So I would spend hours researching how to make a computer faster, going into settings, changing things.
Why did you want to make your computer faster? For games?
For anything. I just wanted to be on the internet. It was so hard to do that on dial-up or satellite internet. If a cloud passed over, you're gone dark.
So what was the first experience that made you want to be on the internet? What's the first thing you remember where you thought, 'I need this?'
I don't know. My exposure to the internet was at friends' places and at school. You play those web game things, RuneScape and all that. I think the thing that really made me want to have the internet was just gaming with friends. That was such a social, fun thing to do. It was totally inaccessible unless I was somewhere else, out of my house.
Yeah. And your friends were playing it at their houses.
Yeah. So there was this FOMO. How do you... when you think about... Man, that's so weird.
Just the snowplow. Just growing up like that. It's kind of odd that you ended up in this situation. At some point, you were trying to get access to the internet, but you must have been good at school because you ended up at U of T. What is the process? How do you go from trying to make your computer faster to getting to U of T?
I was super lucky. I had my dad, who was a math and science teacher, so I had a private tutor. To get my first Game Boy, I had to learn the multiplication table up to a certain point when I was five or six years old. He always set up these challenges: if you accomplish this hard but good thing, you unlock this thing. It was a big lesson in delayed gratification. You can get anything you want, but you need to achieve things to unlock them. He was right there with me, teaching me, testing me, growing me. Having that was an insane advantage. I just got taught.
Why is your family like this?
I don't know. My dad came when he was 10, so he was a kid. But there's something about this. They clearly like doing really hard things, and they clearly value their own understanding. There's a tolerance for pain. My mom is very much an artist, and all my love of beauty and aesthetics comes from her. So I got these two role models. One hardcore science, math, technology, and the other side just the beauty in the world, taste, and art. My dad is a hardcore atheist, and my mom is super spiritual. No particular religion, but we would go out into the woods to hug the wishing tree, make a wish on this big ancient 200-year-old oak tree. When her father died, she would go out and spend time with a beech tree that was her father's. I got so lucky.
Yeah, that sounds really insane. It's like you were almost groomed to be this person that's really good at something.
Maybe. I don't think they ever expected me.
Well, they're clearly driven by a set of values. You have to be very driven by a set of values to do something like that.
They definitely have very strong values. They had strong enough values to be like... It sounds like the inciting incident is when your grandmother died. For whatever reason, they decided that it's unacceptable for us to not live our values at this point. The only thing that matters is living our values. It sounds like they leaned into whoever they were. It sounds like it was a key bonding moment between your grandfather and your dad to say, 'I see you, and we're going to do this.' There's a lot of implications to that. There has to be a very strong set of values and a certain level of conviction and commitment to lean into that. It's not just going out there and cutting down trees. There are very real implications when you're bringing a child into the world in that kind of thing. They clearly knew that.
Because you know what I mean? So it's like it takes a certain level of commitment and for your mom to be involved in that too.
Mhm. There requires a certain level of alignment and commitment that our values and the way we see the world is more important than anything else.
Yeah. Yeah. I think they... Yeah. I was super lucky.
I was super lucky to have the parents that I had, to grow up in the place that I grew up.
It's really cool. Makes me wonder a lot about how it influenced your work. I wonder what it'd be like to work with you. I wonder how you approach problems and difficulty levels, like when you feel like something is done or not done. It would have to change your thresholds of what is acceptable or not acceptable.
Did that... like when you were in school and you were trying to get... and your dad is teaching you this stuff and he's putting you through these very difficult exercises. How did you end up at U of T in that program? How did you end up there? Because I think you were at U of T at a pretty... like what happened was you were kind of at the right place at the right time a lot of times.
Yeah. It's kind of intense.
Yeah. So after you went to U of T, you were at U of T at a critical point because it's not just the guys you were hanging around with or the people that you were around. It's not just that they had just... they were always there. It was like these are people that had been doing this for a very, very long time and it seemed like their work was at a tipping point.
Yeah. It just started to work.
It just started to work. Right. And they all happened to congregate in these clusters of areas. How did you end up at U of T and can you just describe what it was like and what the energy was like? Can you just tell us, for other people who weren't there, what was happening, who these people were?
Well, my dad went to U of T, so that kind of made it my dad's school. I always had a halo around U of T and what it meant. I really wanted to go there. That was the only place I applied. I didn't apply to any other school. What was the scene like? Was there a moment where you recognized that something special is happening here? Because you're kind of the new entrant, but these guys have been on a path for a while. Was there a moment where you recognized that this is a tipping point or this is something interesting? So I didn't take any ML or AI classes until third year of U of T. Before that, I had no idea what was happening. I was totally out of the loop. I didn't know who Geoff Hinton was. But independently, I was just really interested in AI. It felt like intelligence was the last unanswered question. There was no satisfactory answer of why our species is like this. It's the last mystery. For me, it was the world's last mystery. Physics, we could explain all that. We knew down to the lowest level how these systems work. We could predict how they evolved. We just understood them so well. But this one thing of what are we? Why are we so different than every other animal? How do we explain this? There was no satisfactory answer. Everything around us was so dumb and it was just us. I couldn't think of a more exciting problem to work on. So I went into AI and I started doing it independently. I actually left school. I spent a year in Vancouver working for a startup. They were building an app that would turn your sheet music as you play piano. I grew up playing piano. My dad forced me to do that. This was a merger of two things that I loved. To do that, we were making neural nets that listen to audio streams, try to triangulate where in the song you are, track that over time, and then turn your pages for you. Turns out it's a very difficult problem. But I had to learn how to do machine learning and I started reading all these papers and it was all U of T. It was all U of T. I was like, wait, what? At the top of the paper, you see the emails of the authors and it's just cs.utoronto.ca. I was so blown away. When I was in Vancouver, I shot an email to Geoff. I was like, 'Hello, Professor Hinton. I read your paper on the ReLU activation function. I think it should be periodic instead of monotonic.' I gave him this idea and the guy responded. This legend who last year won the Nobel Prize. I was a second-year undergrad, borderline imbecile, and he responded to my little research idea and he told me why I was wrong, very respectfully. Then he put me in touch with some others in the lab. From there, I got to see the U of T ML group's vibe. I would sit in on paper readings and people presenting their work. I started to see this ecosystem, a scene of incredibly smart, incredibly passionate people. I noticed they were all going to Google. All these guys would go to Google Brain. Ilya was there. Geoff, everyone had gone over to Google at that point. So I just started emailing people.
Yeah. Emailing these dudes.
What would the emails say?
Hey, I'm Aidan. Here's some of the stuff I'm working on. What I'm thinking about. I read your paper on this. I think it's super interesting. Did you try this? The next thing might be this. I'd love to work on it together. Do you have any spots? 99% of those emails went unanswered, but a few got answered and then I got in.
That's cool. It takes a certain level of confidence to read a paper and say, 'Hey, you did it like this, but you can actually do it like that.' Why do you think you had that confidence?
Oh, I see. But still, confidently naive. Still interesting that you were confident.
Yeah. I guess so. I never felt like anything was out of reach for me. I never had a sense that... I was confident thanks to my naivety. But I felt like I could engage with the material. I was obsessed with it. I would carry around a stack of papers with me everywhere I went. I'd be in the gym.
Like what's in the stack? The scientific papers.
Yeah. Research papers from arXiv. I would print them out. I would have a pen and paper and I would be reading it. Initially, it was super hard. I didn't understand anything that was happening in the document. It would take me weeks to read one paper. I would be on my phone Googling every third word, trying to understand it. But over time, it got easier and easier. I thought the ideas were so beautiful. We're dealing in intelligence, what makes a system smart. The researchers I was reading, their inspiration wasn't some abstract theory or mathematical derivation. It was people, humans. It was 'this is what people do, so maybe we can create something similar.' Those sorts of analogies. If you read a Geoff paper...
What's your favorite one? Or what's an example of one that communicates what you're talking about?
Yeah. He talks about sex and reproduction. 'Dropout: A Simple Way to Prevent Neural Networks from Overfitting.' Abstract: Deep neural nets with a large number of parameters are very powerful machine learning systems. However, overfitting is a serious problem in such networks. Large networks are also slow to use, making it difficult to deal with overfitting by combining the predictions of many different large neural nets at test time. Dropout is a technique for addressing this problem. The key idea is to randomly drop units along with their connections from the neural network during training. This prevents units from co-adapting too much. During training, dropout samples from an exponential number of different thinned networks. At test time, it is easy to approximate the effect of averaging the predictions of all these thinned networks by simply using a single unthinned network that has smaller weights. This significantly reduces overfitting and gives major improvements over other regularization methods. We show that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification, and computational biology, obtaining state-of-the-art results on many benchmark data sets.' This is cool. It almost feels like graffiti, like a flex. How can you not only find a breakthrough?
Wait, at the bottom here. Oh yeah. So there he's talking about sex and... 'A motivation for dropout comes from a theory of the role of sex in evolution. Sexual reproduction involves taking half the genes of one parent and half of the other, adding a very small amount of random mutation, and combining them to produce an offspring. The asexual alternative is to create an offspring with a slightly mutated copy of the parent's genes. It seems plausible that asexual reproduction should be a better way to optimize individual fitness because a good set of genes that have come to work well together can be passed on directly to the offspring. On the other hand, sexual reproduction is likely to break up these co-adapted sets of genes, especially if these sets are large, and intuitively this should decrease the fitness of organisms that already have evolved complicated co-adaptations. However, sexual reproduction is the way most advanced organisms evolved.' What the fuck, dude?
So it's interesting because you're learning. It's almost like reading a science fiction novel.
Yeah. No, he draws so much inspiration from these high-level observations about the systems that do work, which is our own. I think Geoff is such a beautiful mind, with the ability to draw these connections and pull inspiration from really disparate areas into what is basically just applied math. But he can go from such a high level to such a low-level idea. This is literally zeroing out weights in a forward pass. He can draw from there up to reproduction. That skill, I think, is what makes his ideas so effective, because he can work by analogy so effectively.
So when you read this paper, did it reframe how you saw your own skill and what you were doing with math? How that could be applied, what you were learning about computers? Did this core idea extrapolated out to these broader concepts change the significance of how you saw what you were doing?
No, I always thought it was very significant. I always had a view on the problem we're working on as understanding the last mystery.
No, I mean when you were at U of T walking with these stacks of papers, trying to understand your relationship with AI, were you already at that point seeing the beauty of it?
Yeah, I definitely did. That's what really pulled me in and made me so obsessed. I just think that time, that place, there was this group of people who had a secret. It was deep learning and neural nets. If you were in it, you just felt like... nobody else was really talking about this. No one really cared about AI. No one took it seriously. But you were watching this process of discovery. There was a new breakthrough every few months.
These things were just getting smarter and smarter. It was such a cool time. It shifted dramatically. The impact now comes dramatically less from research and academia and much more in industry. That pollutes some of the beauty, I think, because it pulls in different incentives. But the reality is, to make progress, you need so many more resources that it has to be done in these settings. There's just no other way to aggregate resources at sufficient scale to pursue the work you need to do. But back then, this was run on CPUs, on your laptop. It was very accessible.
Was this stuff being dropped in real time? When did this paper come out? 2014? So is this something where you're at U of T and it's like, 'Oh shit, they dropped a new paper,' and you read it? Were these breakthroughs happening in real time? You would see a paper that had just come out and you're around other people reading it for the first time as well.
Yeah. No, that was happening constantly.
He just had so many good ideas. The most amazing thing about Hinton is that the guy just wrote hits. He had such high taste in what he worked on. A non-obvious thing about this paper, Dropout, is just naming it that term. The branding is so interesting. It's hard to do that, to recognize that you're not just figuring out computational breakthroughs, but you're also creating the scaffolding for other people to participate in these ideas and understand the significance. Research is one part the idea, one part the execution, and one part the branding. Like 'Attention is All You Need' — that's a really good name. Or 'Transformers.' In the moment, it was sort of a joke, like 'haha, that'd be a funny name.' But in retrospect, would it have gotten that much attention if it was 'Sequence Transduction Models for the Modeling of...'?
So you're writing emails. The first email is to Geoff Hinton. Then you get in there and start emailing other people. You're reading scientific papers, trying to understand every sentence. Through these papers, you're starting to understand this scene, this group of individuals working on this thing where breakthroughs are happening one after another. Then what is the story of how you went from that to working on 'Attention is All You Need'?
I got into Google. There's a guy, he's right there next to me on the paper, Łukasz. A guy from Poland. You want to talk about high-taste individuals, this guy is unbelievable. He's constantly half a decade to a decade ahead of himself.
What do you mean? For a normal person like me, what does that mean?
Look up this paper. It's called 'One Model to Learn Them All.' This is another paper at the same time as the Transformer that Łukasz and I worked on.
Can I read this abstract?
'Deep learning yields great results across many fields, from speech recognition, image classification to translation. But for each problem, getting a deep model to work well involves research into architecture and a long period of tuning. We present a single model that yields good results on a number of problems spanning multiple domains. In particular, this single model is trained concurrently on ImageNet, multiple translation tasks, image captioning, COCO datasets, a speech recognition corpus, and an English parsing task. Our model architecture incorporates building blocks from multiple domains. It contains convolution layers, an attention mechanism, and sparsely gated layers. Each of these computational blocks is crucial for a subset of tasks that we train on. Interestingly, even if a block is not crucial for a task, we observe that adding it never hurts performance and in most cases improves it on all tasks. We also show that tasks with less data benefit largely from joint training with other tasks, while performance on large tasks degrades only slightly, if at all.'
Mhm. So basically, before this, you would build a model for translation. That's the only thing it does. You would build a model for image classification that looked completely different. Build a model for each of these different tasks. Łukasz's idea was, why are we doing this? What if we just had one model that did all of those things? It could consume audio, images, text. We only got there two years ago with GPT-4 and Omni, but this was being worked on eight years ago. He's always so far ahead. The results we saw were that when you put everything together, it helps. Things that were really hard to make good because you couldn't collect enough data got dramatically better when exposed to and augmented with these other capabilities. Łukasz has been thinking about this stuff for so long. Anyway, Łukasz hired me. I went into Google, worked on this, worked on the Transformer. When I was leaving Google four months later...
Wait, you did both of those within four months?
It was. That period of my life, I slept at the office. The night we submitted the papers to NeurIPS, I was at the office and I got woken up by a cleaner opening a door into a phone booth. I was sleeping on the floor of the phone booth and it whacked me. That's what woke me up. I got hired, four months later I was leaving, and then I found out that Łukasz thought I was a PhD student. He's like, 'We don't hire undergrad interns.' So he was so confused. Apparently there was some sort of administrative error that put me there. I sort of slipped through the cracks to be present.
Holy shit. What is the process of writing a paper like this? How does it go down? At first it's all experiments, code, setting everything up, running it, trying, and then it's an iteration process. But what is it like? Obviously this Łukasz guy had an idea. He's like, 'What if we brought all the models together?' There must be some inciting incident. Maybe he's in the shower and has this thought.
You should have him on. You should ask him. I don't know how he had that idea.
No, but I just mean in general, how does it work?
Yeah, you just have moments of inspiration. You're like, 'What if we did that? Would that work?' You just wonder. You're curious. You want to run it and see. That's how research and these discoveries happen, just curiosity. Suddenly you have a 'what if' and you try it and see.
So that experience, those four months at Google, changed your life, right?
It was definitely a formative period of my life. After that, when I got back to Toronto, I started this research group at U of T for undergrads called For AI. I started that with Ivan. I messaged the U of T computer science student union and said, 'Hey, I want to do some research in machine learning. I need some help. Who would want to do that with me?' And Ivan messaged me.
Damn, that's crazy. So this is the third year.
Yeah, starting third year.
He had dropped out at that point. He was working at a startup.
So I guess he was still on the emails.
Yeah, he was still in the Slack.
So that's your first relationship with Ivan. Then Geoff invited me to Google in Toronto and that's where I met Nick. I was so jealous of Nick because Nick was like Geoff's first rapport, like Geoff's godson. They had such a close relationship. I was coming into this lab and Geoff and Nick were like this together. I really admired Nick and his relationship with Geoff. That's how I got to know Nick. Eventually, the three of us, Nick and Ivan actually lived in the same building right next to the Google office.
They didn't know each other though.
Yeah. I don't know if they knew they lived in the same building, but I did. Then one day the three of us just sat down and said, 'We should fucking do this.'
How does that happen? When you did the For AI thing, that was your first thing with Ivan. He messaged you and said, 'I'll help you.' That's where you guys worked on together.
