Welcome to Text on Tech. This is Jed Sundwall. This month we're joined by Esther Dyson. If that name sounds familiar, you may have heard of her from her career throughout technology. She's been around for a long time doing a lot of interesting things. I first got to know her when she was on the board of Eventful, the first startup that I worked for out of grad school way back in 2007. Anyway, she's been a luminary for a long time. I think she was the first executive director of ICANN, so she's been around from the very beginning of the internet. Also, if her name sounds familiar, she's the sister of George Dyson, who was my guest on episode five. I have to say, having spoken with both of them, I have the privilege on this podcast of having to listen to these episodes a few times as I edit them. I want to be fair to all of my guests, but I would say that the conversations with George and Esther really benefit from multiple listens. They are such smart thinkers, such deep and thoughtful people. I admit there are a lot of things that they say in the moment that I don't catch, and I don't even catch upon first listen. Sometimes I really understand things on the second listen. So I really enjoy this conversation. I've listened to it a few times and I've learned something every time. One thing I want to point out is that Esther makes a comment about a particular kind of luck that we can maybe aspire to, or at least recognize in our lives. There's the possibility that you get to do something for your job, you still get paid to do it, but it's work that you would do for free anyway. That's a really nice definition of privilege and luck, and something to maybe aspire to in one's career. I have to say that I feel that way myself. I feel very lucky that I get to do stuff that I really care about and would do for free if I didn't have to make a living. Anyway, she mentions this concept and it kind of went over my head in the moment, so I wanted to call that out now. Also, one thing I want to start doing on these podcasts is at the get-go explain what we're going to be talking about in terms of spoiling the works. This is an all-spoilers podcast. You should be able to enjoy these conversations even if you haven't read the text or seen the movies that we're talking about. But today we are going to be talking about 'Her', a 2013 movie by Spike Jonze about a guy who falls in love with an artificially intelligent agent, and the repercussions of that. I won't spoil it much beyond that. There's not that much more to the story other than the fact that he eventually learns that he is no match for her, and it's a very asymmetric relationship and it ends up being profoundly unfulfilling. And then we're talking about a book called 'Never Let Me Go' by Kazuo Ishiguro, which takes place in the '90s. I think we misspoke in the podcast and said we thought it took place sometime in the '70s. Regardless, it's an alternate history fiction story about a school of children in England who are clones, human clones. Throughout the book, you learn slowly over time that they exist for their organs to be harvested, to be given to presumably non-cloned humans. It's a really, as you can imagine, quite haunting and disturbing in some ways book, but also a really beautiful book that I'm really glad I read. I'm really glad that Esther recommended it, and I think about it a lot. It's a great book. So without further ado, let's talk to Esther. Thanks for listening. All right, Esther, welcome to Text on Tech.
This is our first live recording, so thanks for indulging me. Let's see how it goes. So before I ask you anything about yourself, I have to ask if you're a technologist.
Yes, but I would say more a scientist than a technologist in some sense. I'm not actively building things. I'm certainly scientifically interested. I'm also a journalist, an asker of questions. In short.
That's fair. I know you listened to the episode with George. I was very glad that he enthusiastically said yes because he builds boats. I think that's a good answer. So tell me a little bit more about your background in journalism. I think it's interesting for the audience.
For context, my brother and my father worked for Oppenheimer. So we grew up in this amazing bubble of scientists and Nobel Prize winners, very few actual Americans, lots of foreigners who lived in this compound at the Institute for Advanced Study. It was an amazing upbringing. We kind of took space travel for granted because that was something my father was involved with. And we were askers of questions. On the other hand, we didn't know much about real life. I remember, this is a very academic place. I spent the year I was 13 in England at my own behest with family friends of my dad's. When I came back, I finally got the job I'd been wanting forever, which was to work in the public library, and discovered as I was talking about details of working and so forth, I thought, well, I know about the guys in the stores and the milkman, but I thought normal people had three months off in the summer. It's a rude awakening.
I think when you realize that, I definitely remember a moment when I realized that summer vacation was not actually part of life after a certain age.
So you were raised a scientist?
I was raised by scientists. They were great parents. They sort of said whatever you want. We didn't have to discuss physics at the dinner table, put it that way.
Okay. And then how did you get involved in computing from early days?
Well, after college and after a year of sort of intellectual dishwashing, writing newsletters for office workers, 'Office Guide for Office Girls', 'PS for Personal Secretary', and 'MS for Medical Secretary', I got a job as a fact checker for Forbes Magazine. Then I did what I really wanted to do. I'd worked for the college paper. I became a fact checker for Forbes back when it had fact checkers. The rule was if you couldn't come up with a good story idea, you had to fact check someone else's story. The most interesting stories that weren't really being covered much were in tech. So I got more and more interested in that as opposed to utilities and the kinds of companies that the audience of dentists and retired investors were interested in reading about. Ultimately, I joined Wall Street because I wanted to... I realized at Forbes we were just writing stories, and I thought, 'Oh, I'll go to Wall Street where people make decisions.' And I discovered that what I was really doing was selling stocks for a broker's house. The most interesting company I was following was FedEx. It was a startup that had raised $100 million, was doing a new round at $15 million, a down round, before it ever shipped a package. It was a wonderful story. I learned about logistics and timing and slack and all kinds of really interesting sort of physics slash logistics concepts. People want to go as fast as they can somewhere; packages really don't care. Understanding the difference there. Then, of course, as computers and tracking got better and easier, they began to optimize much more. It was a really great start. Then ultimately I joined a tech newsletter where the founder who invited me to write it because he had left Morgan Stanley. His name is Ben Rosen. He was on the boards of Compaq and Lotus and really had a conflict. We talked to Business Week and Computer Week or something about selling it. Then I thought, unlike... I looked at Ben and said, 'You know, he's definitely better dressed than I am, and he's the epitome of Wall Street success, but he's a human being. I could do that too.' So I bought it from him. I ran it for 25 years. It was called Release 1.0. It was sort of a little pun because his name was Ben Rosen, it was the Rosen Electronics Letter, so it was capital R, capital E, capital L. Release 1.0. The outfit was called Rosen Research, so I changed it to EDventure with my initials, but again I didn't want my name in it. That was what I did for 25 years.
Amazing. And then do you want to talk about what you're doing now?
Well, the two other things that are key to my life in some sense: a lot of travel, and especially a huge engagement with Russia starting in 1989 and ending in 2022, and more time in Africa than your average American. Plus also six months training in Star City outside Moscow as a cosmonaut. And what I'm doing now is basically got interested in healthcare after tech because I wanted to find something more useful. If you look at healthcare and you start asking questions in the US, your first question has got to be ultimately, why are we spending so much money fixing people who shouldn't have been damaged in the first place? So I got interested in the production of health rather than the recovery of health, and started a 10-year, originally it was a five-year nonprofit project called Wellville, to work in five small communities in the US to help them stay and be healthy. Like everything else I've done, it was more that I didn't know what I was doing or what I was going to learn, because if I did, I wouldn't have bothered to do it. I learned a ton and we helped the communities a little bit to basically build their own capacity to be healthy.
I think it's your email signature that says 'Always making mistakes.' I like that.
That's very old. It's endured. That's timeless wisdom. People keep making mistakes, and as long as they're new, they learn.
I sometimes say we should aspire to be less wrong continually. I brought up the idea of doing this podcast with you and asked you what you would want to talk about, and you immediately said 'Never Let Me Go' and 'Her'. Shockingly, I had never, 10 years after 'Her' had come out and even being a Spike Jonze fan, I'd never seen it. So I finally watched it on a plane sometime in the past year or so. Then I picked up 'Never Let Me Go' and I absolutely loved it. Do you have any recollection of why those things came to mind?
Well, they're both... On another day I might have picked two completely different ones. AI, I think people misunderstand it. Some willfully, some not. Some people think it's big and shiny, some people think it's incredibly dangerous. I think... One of my favorite things is two-phase commit, which in its day was a big thing. It was basically the notion, for example at FedEx, somebody picks up the package and they scan it in, and then someone else acknowledges they got the scan, but you need both sides to certify the transaction and then you can keep your database up to date. A very basic thing, but it was huge in the plumbing. AI in a sense... pretty much everything we do has a two-phase commit now. In the same way, long ago we used to talk about phone sales. Dell Computer started out very innovative. They did fax sales. They would fax advertisements and then confirm them by fax. So I mean, 10 years from now there won't be anything that isn't somehow AI supported, and that's not a problem. AI is computing and it does certain things right. Then there is the whole other area of AI either pretending to be... and of course the AI itself doesn't pretend, the owner of it makes it pretend to be human. And the use of AI to manipulate people much more effectively than we can now with just two-phase commit and databases. But it's not magic. At the same time, one of my favorite science fiction memes is the electronic beings in the sky who look down on Earth and say, 'It's so weird, they think with their meat.' I'm electronic, I'm intelligent. These are chemicals. How could... And so the essence of what George talks about a lot is if you're sufficiently digital, you become analog again. If something comes along that's better than us, that's digitally analog, God bless them. I believe in evolution. But right now I'm a human and I believe in humans, and not using AI or manipulative business models to manipulate people. The big danger with AI, beyond discovering new forms of poison, is really its ability to accelerate predatory business models. There are predatory business models now, including mostly things that addict people to stuff that's bad for them, whether it's drugs or social media or junk food. What we need to do is it's useful to train AI and to be transparent about what you're training them to do, but we need to train people to be better people and to be better humans. So trying to get back to what your question was, why I chose these... 'Her' contains all of that, I think. 'Never Let Me Go' also does, but in a much weirder way.
The timing of this is really interesting. Wellville is ending. We're having our final big gathering next week, though the whole thing ends at the end of the year. Our big gathering is not a funeral, even though Wellville is ending. The whole point of it was not to be permanently sustainable. One thing some people are getting addicted to in some sense is the notion that they should live forever. The point of Wellville was precisely not to live forever, but to help the communities build, not people who live forever, but institutions that at least have some sustainability. Then in the middle of this summer, Biden resigned, probably a little late. And just randomly, somebody I know, there's a movie about him undergoing euthanasia, and just lots of stuff about term limits. So right now my sense is, term limits is a really important thing. Mostly people stick around too long, things last too long. But in some things, our term limits are too short. When we invest in something that keeps people healthy, our vision is too short. We should be helping people be healthy now and look for return 20 years from now. The thing I regret most about my youth is not wearing sunglasses because my eyes are overly sensitive. A lot of people now I think regret going to loud rock concerts without earbuds. As we talk about the fundamental of Wellville, it's partly invest in your community, don't be selfish. It's also about time. Think forward. We need to think along both those dimensions more effectively to build a good society. The thing I recall the best about 'Never Let Me Go', which ironically as I reread it I think was in the movie but not in the book, was at the end someone says, 'Whether you're a clone or a human, life is too short, and you figure out what's really going on right at the end, and then it's over.' But you can have a good life even if it's short. The point is what is the substance of your life, not what is the length of your life.
I got to pull out my notebook. This has nothing to do with 'Never Let Me Go'. We're recording this during, this will probably, I'll post this sometime at the end of next month in October, but we're just finishing up Climate Week here. I wrote this down, I don't remember what meeting I was in, but I said something: 'When we cooperate as a species, we cooperate with the future.' This is a quirk of the fact that we perceive time, and it's a very interesting thing about us is that we care about ourselves and we care about our future selves, and we also care about future generations. We are selfish, but we also care about future generations, and that is made to survive. That has to do with exactly what you just said. It's great that you said that because it's what's on my mind as we talk about climate stuff. We have to prepare for the future, but we don't have a lot of good institutions that are designed to do that. 'Never Let Me Go' is so beautiful and so... I'll do an intro to the podcast, but it's all spoilers. This is a terrible concept for a podcast because I assume that my listeners have actually read the texts, and that's not always true. If they haven't, we need to get in a little context.
One thing I love about the way Ishiguro writes is he uses euphemism so elegantly. The basic concept here is there are normal people and they are kind of in control. This is in England in the '60s, something like that. It feels like the '60s, and so there's quite a class society. They've figured out how to make clones that can be used for spare body parts for normal people. The clones grow up in these boarding schools and don't really know what's going on, but they do get taught, 'Here's what happens. You grow up here, and you maybe get a job, but ultimately what you do is first you become a caregiver for people who are donating, and then you become a donor yourself.' The people who donate do it sometimes only once or twice, sometimes three or four times, and then they 'complete'. That's the term. It basically means donate your organs and lose a kidney, and then you need a caregiver to take care of you. After you've lost three or four more organs, you complete. This is an amazing euphemism. You never really understand what's going on. The essence of 'Never Let Me Go' is a song that is supposed to be about 'Never let me go, my lover', but one of the protagonists here, it is a baby because the clones do know that they can't have babies, though they are encouraged to have sex and to form relationships. There are these myths about if you're truly, truly in love, maybe you don't need to go through the donation completion cycle. But at the end, whether you go through that formal donation completion cycle or you just go through life, at the end you hit the same death and your time is over, and it's probably too short. Everything you've learned, if you don't do it right, goes to waste. But if you do it right, like with Wellville, we can end and the communities will take that wisdom forward.
That's a great way to describe what you're trying to do, to take the edge off of... The lesson of the book, I'm about to start crying, it's such a beautiful book, but it's so sad. He writes so well about what it's like to grow up and have friends and have betrayals and teasing people. He writes so carefully about the protagonist who's narrating most of the story, like he was looking at her this way and then she had her back to him, and then they all walked in this direction, really paying attention to how humans look at one another, how they respond to one another, form little groups. To describe the experience of a clone growing up without a family, but we're still just social animals and we have this genome that makes us behave in this way and try to find meaning in our lives. To your point, it seems cruel what's happening to them, but then you're like, well, it's not that huge of a difference between what's happened to us as well.
One thing I found interesting about the book, and I think there's some context around AI in here as well, but it's really what's on my mind these days is our institutions. These kids, it's another element of the story, is that the protagonist and her friends they all went to this kind of special school, and discover later how special it is.
Do you want to... You read the book more recently than I did or listened to it again. I'm curious to get your take on how special the school is.
The essence of it is they don't really understand how different they are. So much of what they think they know is rumors. One thing I learned when I went to Russia was what's really important is usually what you don't even talk about. Some of it is misinformation and some of it is real. 'Why are we here? Why are we different from...' They know something about society. They read famous novels, they read Jane Austen. At the same time, they're told, 'You're special. You have a special role. That's why you're here in this sporting school.' Ironically, I always remember one of my very early jobs was at Viacom, which had spun off from CBS. It much later got reacquired. I'm sure there were loads of Wall Street Journal stories about CBS spinning off Viacom, but nobody mentioned it to me. I'd heard it once, and I thought that it must be a secret because they didn't tell me. I was 20 or something. Then I found next to the copying machine, or maybe the fax machine, this stapler that said CBS, and I thought, 'Ah, evidence.' But I didn't mention it to anybody because I just thought, 'Whatever it is, they don't talk about it.' It's exactly that same, 'I think I understand, but I better not mention it too much.' There was one teacher who wanted to tell them the truth and she basically got fired, because the sense was somehow knowing too much might make them uncomfortable or something. I'm a big believer in the truth.
What's revealed is that this was an experimental school that some activists had managed to set up to show off that these clones are actually full humans, they create art. It was just unbearable for some of these teachers to see that these are real people that we're just going to chop up. There's some intimation that there are other cloning operations that are not so nice, just like a feed lot, a chicken farm. Most of the teachers just went along with it. The staff, when you got the sense that they were kind of allies but had no power either. They weren't the people who started the school, they were just workmen, and they kind of also saw the kids as human. What I think is so fascinating about it is it aligns very nicely with your story about CBS. You're 20 years old, or little kids, creating this mental model of the world and institutions in it. These are little kids that are like, 'This school is our world. This is the most important institution in the world.' They finally grow up and they find the original teachers, the bosses of the school, and there's just a bunch of people who'd run a nonprofit that ran out of funding. They're like, 'Wait, I thought this was forever. The whole world. In my mind, you ran the entire universe, and I find out now you're just some people that set up a nonprofit that ran out of funding.' As you get older, you keep peeling back the curtain and you're like, 'Wait, it's just people trying to figure stuff out behind every single curtain.' But when you're growing up, we tend to imagine these very large institutions that are inscrutable. What's fascinating to me about AI is that we are actually now entering a world in which there are these entities that we interact with that are inscrutable. That's the point. People are inscrutable too. The business models are obscure, the people are obscure. Just because it's AI, ironically we're sort of paying more attention to it. 'My God, AI discriminates.' Well, it discriminates because before that, people discriminated, and that's what the AI learns from. It just does what it's trained to do.
There are these news cycles every now and then about horrific content showing up in the training data sets, the kind of things that you don't want to train these models on. Unfortunately, my response to that, I get somewhat philosophical. I'm like, 'Well, if you want to train a model on the world, it's going to contain a lot of things that I wish didn't exist.' There's a lot of stuff I wish didn't exist in the world. What you need to do is train the model that these should be counterfactuals or something. But it's like, don't train on what is, train on what should be. But then you have who decides what should be. Then there's the other sort of AI slime problem of the AI is taught to leave out the crazy things like 'you should eat rocks' or 'put glue on your pizza'. So here's a question for you. What was the craziest hallucination ever of an AI, or ever in the world?
You asked the right question back. In the world, we can look at a lot of religious texts that have had a huge impact on populations that are probably born of hallucination. But a genuine hallucination about the truth... I'll just give you the spoiler. You're baiting me. There was this crazy notion that the world was round. Or flat. Or round. In the beginning, people said that was a hallucination. The world's flat, and you fall off the edge. I know cousins of people who've fallen off the edge of the world and were never heard of again. If you take out all the hallucinations, you may also take out interesting scientific discoveries or things that should be, like every parent loves their child and is capable and has enough money to afford child care. I want that hallucination to come true. That's how we imagine. This is very interesting because I love this idea that the world being round is a hallucination. It is not. It's a concept that exists in our minds that's not obvious to us. That is a hallucination. In that sense, it's a hallucination that corresponds with reality because we can sort of prove that's what's going on here. But there are these other hallucinations, these aspirational concepts of how we want life to be that we have to imagine and then we make them real. Or just the hallucination that when you have pictures either of galaxies or nano thingies, most of us is actually empty space between protons and neutrons and stuff like that. It's incredible. You're saying also just how we perceive reality generally is always some form of... Once when I was 8 years old, I lost the ability to construct a 3D world because what you see is actually kind of in a plane. When you're a teeny little baby, you sort of discover where in your mind, and I simply could not construct a 3D world. It wasn't that I lost binocular vision, but simply the ability to. So I had a real problem getting off the school bus. Is this a permanent issue?
No, it happened once. I wish it would happen again because it was really interesting. Fortunately, George was there and helped me off the school bus. The pole was right here and the driver was here and I couldn't really see the stair. It was weird.
You remember that as an educational experience because you're like, 'This is a way to perceive reality that is not fit for purpose.'
I didn't quite intellectualize it as much, but this notion to understand that it's all constructed.
I love talking about this stuff. There's a book called 'The Order of Time' by Carlo Rovelli. I'll put it in the show notes. I haven't read the whole thing, I started reading it, but he makes the case that how we perceive time is also a construction. The way I've understood this myself is to know that we have a clock ticking on the camera recorder. The way we perceive time now, we've been talking for a little while and I can remember that in the past, and my mind has some sort of way of sensing roughly how long ago I walked in this office. The point being is that all of that timekeeping is just done in my head relative to this moment right now. The way my mind perceives how long ago something happened is arbitrary. It can feel like a really long time ago, it can feel like not very much time at all. We've all experienced time moving faster and slower. The author is a physicist and he explains how time slows down as you get closer to a center of mass. That makes no sense to me whatsoever, but he's like, 'This is true, and we shouldn't take how we perceive time as an accurate representation of what time actually is.' So just building on what you're saying.
There's also the interesting phenomenon of you are walking out into the street and you react to an oncoming bicycle faster than you actually perceive it, because different parts of your brain send signals right to different parts. Every once in a while I experience that and that's really interesting. The sensorium is responding in a lot of different ways to things. I think we do ourselves a disservice when we think that we have biases towards favoring certain senses.
Let's talk about the hallucination of 'Her' then also, because you said something interesting which is it's not the AI trying to manipulate humans, it's the AI manufacturer with a business goal or a political goal or a revenge goal. There could be all sorts of bad intentions. I would also argue though that it takes two. That's kind of the story of 'Her', is that this guy really wanted to be loved.
'Her' is not about bad AI. 'Her' is about a guy willfully hallucinating because he's smart enough, he's a techie, he knows that she's an AI. It's long and complicated, but the basic... He's a dude, he has an unhappy marriage and complicated stuff like that. Then these OS's come out, which is a strange use of the technology because they're not OS's, they really are applications. But never mind. You're in a safe space, we can complain about that. Make sure Mike Jones hears this. So they are trained by someone who probably has a business model to train really good companions that are flirty and funny and human and all this. Of course he falls in love with her, knowing that she's a her. There's an interesting point at which someone is hired or at least asked, maybe as a favor, but anyway, she's a physical, real person who shows up but with the voice of her. It goes really badly wrong. It's totally weird. The real discovery is partly about AI and also partly about time. At the end, it turns out she actually has 1500 lovers, at least. It's some crazy number. She has enough time for all of them. He needs to be needed, he needs to be unique. He doesn't... and his time scale...
Yeah, I think that's a really good point. The movie is about a guy who is lonely and he finds this AI companion, and he thinks it's a unique relationship, but it's not. She's having thousands of conversations simultaneously. It's a really profound commentary on the nature of love and relationships in the digital age.
Is completely different, yeah. He needs her attention for some measurable, you know, above 30% of her time, yeah.
And that's sort of... he knew she wasn't real, yeah. But when he discovered that she was just living on a completely different time scale and didn't need him, that was when the whole thing fell apart, yeah. Yeah, it's... the phrase that comes to mind is like an impedance mismatch, yes. Like yeah, yeah, yeah, railroad tracks, whatever, yeah. But a little more severe, yeah.
And what, but then also pointing out like what a relationship even is, right? And it's not just feeling cared for or having someone think that you're funny, it's a lot more than that.
Well, it's them needing you right because it needs to be two-way in some form or another. I mean, many human relationships are not, and many human relationships are toxic, yeah. And this one, how shall I say it? It's not willfully toxic, yeah, but it's toxic in terms of what humans need, right?
Well, it's clearly affects her... yeah. And it's... it reveals what love is like functionally for humans, which is what you said, a two-way... it creates a web, yeah.
And that's the point about the clones in Never Let Me Go. They are human, yeah. They may be artificially created, but they're not artificial in the sense that they can feel love, they are human, right? And they love one another, and even though they don't have babies, they do have families, yeah. And they have... they need one another. And this whole... there's basically a three-way bungled romance between Tommy, Cassie, and Ruth, and yeah, that... you know, Ruth redeems herself in the end before she completes, yeah, by insisting that Kathy and Tommy get together again, yeah. And that gives meaning to their lives, yes. And they helped get each other through everything, right? And they needed one another, they needed one another.
Well, and then this sort of dark... it's such a... it's such a great book, but... I don't know how to say it. It's a very dark optimist or something like that. Like these clones, they were also needed, they were for something, like... yeah.
Well, they were human, they were human, but they also... like donated their organs to other people. Like there's some virtue in what they did, even though they didn't... you know, they never signed up for that.
I mean, that's true, but... when there wasn't a lot of human connection between the donors and the... no, certainly not, recipients. I mean, I'm not advocating for this scenario, yeah. But they were needed in... you want to be needed by a particular individual, not by... you know, some something that's hungry for an organ, yeah. Like... yeah.
Yeah, in a sense their recipients were not human to them. That's right, that's true. And were... you know, fundamentally grabbing something in an inhuman way, yeah. Yeah.
