AI Knows You Too Well: Is Privacy a Lost Cause? | Andy Yen, Founder of Proton

AI like social media is intentionally designed to be addictive. In fact, you know, some of these AIs will even change the way they talk to you to cater to, you know, your personality and sort of what it senses is the sort of answer that you want to hear. And that's a really scary thing. You know, you mentioned that AI maybe knows you better than some of your best friends. I would actually argue that in not so long from now, the AI could know you better than even you yourself. A lot of us humans are not so self-aware about who we are, right? AI will actually be able to exploit the weaknesses in your personality that even you are not aware of in order to compel you to keep using it and, you know, do what it wants.

Hey, thanks for having me. It's a pleasure to be here. And I think between the world of crypto, the world of encryption which you're in and the world of privacy, a lot of intersections. So hopefully we'll have a fun conversation to go through some of these pretty important topics as you say.

I think it really depends on who you are, right? The average person is probably quite screwed if I'm being completely honest. And I would say the more technical sophisticated people have more means to protect themselves but it's going to become harder and harder as you know if the current trends go on. So what we see actually, you know, people talk about AI and I think you mentioned AI as well and what AI is actually doing is it's simply an extension of a trend that's been going on for 50 years because fundamentally AI is actually a better and more efficient way for humans to communicate with computers. And so it's not dramatically changing any of our business models but actually it's accelerating the existing models that already exist. So if you think about search, the information that you give into Google search, which allows Google to build a very detailed profile about who you are, well an AI conversation with say Gemini that is way more intimate, that is getting much more information about you. So what Google is able to do is they're probably able to accelerate by a factor of five or 10 their existing business model with the advent of AI. And this is something that is happening, you know, sort of across the board. So then it becomes harder and harder because AI now becomes the central tool of our lives. It's everywhere. So do you live in the stone age so to speak from a tech perspective or do you participate in the celebration of the loss of our privacy globally? And this is why I think in some extent the average person who doesn't have any knowledge of this is pretty screwed. They can't take means to protect themselves. But if you're aware then you can do something. The problem is I would say probably 80% of the population, 90% of the population simply doesn't understand some of the issues that we talk about, you know, for example on this show.

Unfortunately, it's kind of all of the above and that's kind of the scary thing, right? So, let's break that down piece by piece. Of course, tech companies can see it because they are recording and essentially, you know, analyzing and saving every single conversation you ever have. So, everything you ever type in is being recorded and more or less permanently retained. They're not anonymizing it. They're not doing something to kind of mix it in because you're logged in. It's your user ID. And they are actually, you know, looking at this information because that is how they try to improve these programs. So it's there and they actively look at it and they use it because that is part of their business and it's also being used to profile you and send you advertisements and in the case of, you think of ChatGPT, even shopping recommendations so they can tell you who to buy and recommend you products and you can actually directly buy them through ChatGPT now, right? So the companies definitely see all of that. Now the thing about what that implies is anything that a company has it's actually obliged to give to law enforcement. So if the FBI asks for it, if the feds come asking for it, if any, you know, police or prosecutor requests it, that information is also available. The government has access to this information. But a private party that sues you can also get access to that information. And there's also in fact a famous New York Times case where New York Times I think sued OpenAI and as part of the lawsuit they tried to require OpenAI to actually retain all the conversations because they wanted to use those conversations as evidence in their lawsuit against OpenAI. So that's the thing. Now what is even worse sometimes is then there's also the inadvertent breaches that happen when you give something into OpenAI in ChatGPT you're actually contributing to the knowledge of ChatGPT so the information that you give it becomes part of its brain so to speak and if it's talking to somebody else there's actually a very real possibility and in fact quite high likelihood that information that you've given in it can then be regurgitated out and shown to somebody else as part of another conversation because that information is now in the corporate information that is used to train and give answers to these models. And that has happened, right? If you put a password into ChatGPT and someone's very smart in engineering they can get ChatGPT to spit out the information that you gave it, you know, on accident. And then of course there's also bugs. I think there was, you know, a couple examples of data breaches where, you know, an AI company accidentally left something open and revealed all the chats or sometimes the chats were, I think there's even one case where the chats were accidentally opened and indexed by Google, in which case anybody can get access to them. And that's the nature of information. Once you put it out there, it's out there. You cannot really take it back. You may be able to sometimes force them to delete it. But that may or may not be too late depending on your threat model. So unfortunately the answer is it's all the above in that whole list of things that you gave me because when you put it into ChatGPT it is unfortunately no longer your data.

Yeah, they would have your chat logs and depending on what you say to ChatGPT, that could be quite compromising and you know there are some people today who use ChatGPT for relationship advice, for personal advice, it is their psychologist. It is maybe even their virtual girlfriend or boyfriend as the case may be. The information that you would give a child is incredibly intimate. It's literally a private conversation with somebody who in some cases is your best friend. And that is all potentially leakable, accessible, subpoenaable and also available to hackers as well.

2014.

AI like social media is intentionally designed to be addictive. In fact, you know, some of these AIs will even change the way they talk to you to cater to, you know, your personality and sort of what it senses is the sort of answer that you would want to hear. And that's a really scary thing. You know, you mentioned that AI maybe knows you better than some of your best friends. I would actually argue that in not so long from now, the AI could know you better than even you yourself. A lot of us humans are not so self-aware about who we are, right? AI will actually be able to exploit the weaknesses in your personality that even you are not aware of in order to compel you to keep using it and, you know, do what it wants because what is the purpose of something like ChatGPT and Gemini? Well at the end of the day it's engagement they want you to keep using it it's sort of a hamster wheel that once you get on they never want you to come off so it is designed to tell you what you want to hear to keep you coming back and to ultimately make you dependent. That is not a bug that's actually the core feature of this project and we've seen this play out with social media algorithms which were just basically AI light.

It is and it's also a fundamental human right. So privacy in many ways is our last defense against the encroachment of surveillance capitalism which today dominates the world. If you look at the largest companies on the stock exchange globally by market cap, they are really all companies who are actively engaged in AI. That is the biggest business and the market cap of these companies added together is bigger than most countries, right? You add them up, you get for example you take a top two or three companies that's bigger than the GDP of Germany. So we are really at the stage where these companies have gotten so big that they are actually probably in many cases more powerful, more influential than even governments themselves. So the ability of governments to even regulate these companies is quite limited. You know, we think often about oh, you know, we used to think about privacy as we need privacy and encryption to serve as the last safeguard against the encroachment of power of government on individual freedoms. But these companies today are bigger and more powerful than most governments. So you can almost say that the government part is irrelevant. It's the corporates that are probably even worse in many cases. And government actually in some sense at least in a democratic world is supposed to serve the will of the voter. So you have some control over that. But for OpenAI or Google your vote doesn't count. You don't have a vote. So it's actually even worse.

I think a subscription doesn't mean that they want to violate your privacy because no honestly if from a better take on your perspective if I can trick this person into giving him his data for free but I can also make him pay me for that privilege why wouldn't I do both, right? Yeah this global media here optimistic will actually pay me to abuse this data hell I'll take the money I'm going to leave the money on the table right and also harvest just his eyeballs and sell him some ads. Do all the above. Find out we can collect even more money and that's what these businesses are about. It's all about money. So fundamentally it's a question of business model. The business model here is monetization at all cost. These are profit driven companies that care only about profit and they will squeeze a dollar out of you any way they can. So they're happy to take your subscription money and abuse your data at the same time and this is what they do. So subscription I think it's not a close enough safeguard to say that you know oh it's private it really you have to go down to the business model the business ethics also what the business stands for that's the key thing. Now I think Sam is talking about oh you know it'd be great to have some government regulations around privacy etc etc he's probably more thinking about his New York Times lawsuit right he wants the government to protect him from third parties subpoenaing his information that he's collecting so what he's basically saying is I want regulation to ensure that the only person that can abuse your data is me and nobody else. Right? That's effectively what he's saying. So it's regulation in sort of a let's say very self-serving way, right? He's definitely not asking oh let's have regulation that prevents me myself from abusing your data. Now he just wants everybody else to be locked out of his ecosystem so he can have a monopoly on your information. And that's like so yeah I wouldn't say that just because he says that he's actually going to go in that direction because you know history has proven over and over again and Sam's a known quantity right he's been the value for a long time people know what he's about by now.

I think in the long run. So here's the interesting about AI in sort of all technology shifts. You know, in computing, there's a concept called Moore's law. Are you familiar with Moore's law? It's basically that computing power doubles every 18 months. And somehow it's amazingly held up for the last 30 years. It's a bit similar with AI. AI is going to rapidly commoditize. What would today cost maybe a billion dollars to train may in five, six years time only cost 10 million. So I think there's two sides. One is the cost of AI is going to go down probably exponentially with time. So these giant cost projections that people are putting out today of how much it costs to build AI that may not even be true. They may say oh it's a trillion dollars but that could end up being only a hundred billion over time so I do think there is a way to make the business model work over time this I think is we can be pretty confident about that but I do think some of the promises that have been made and advertised today in terms of how much money we're going to spend how much we're going to invest how much we're going to build those are unrealistic, not grounded in reality. And the businesses today, most businesses are highly unprofitable. And that's okay in the early phase, but at some point investors are going to want a return and investors may want these returns before the costs have dropped far enough on the exponential scale and then you enter into a situation where these companies are going to be under increasing pressure to generate as much money as possible. And then this is why ChatGPT gets into shopping. It gets into promoting different things to you. This is why they get into browsers so they can track your browsing activity and pop up other things to sell you things because they're under this pressure to generate money. But I do think long-term the business model works. It's just going to take a while and these companies may not have so long. So they're forced to be increasingly aggressive in abusing your data and unfortunately that's their business. And this is why when Proton, you know, we built our Lumo AI we did it in a different way because we realized that you need to have a different business model or else it simply is going to lead to really bad outcomes for the users.

Yeah, there's no technical limitation in fact that prevents them from doing what we're doing. What really prevents it is a business model limitation. And if we're being completely frank, it's a problem of capitalism. Capitalism drives them to make the highest possible profit. And Proton, of course, being predominantly owned by a nonprofit doesn't have those same type of constraints, but there's no technical barrier to say why you couldn't do the same thing as Lumo, you know. Is there any legal barrier? There's also no legal barrier as well, in fact. So governments around the world aren't saying, 'Hey, we need subpoena rights just in case, you know, we're dealing with someone who's cooking up a bio device that's going to kill thousands of people.' Well, depends on the government. Russia and China would definitely have those requirements. But in the US, here in Switzerland, here in Europe, we thankfully are not there yet, right? I say yet because it's hard to tell the future, but we're at least not there yet. Let's put it that way. So what Lumo does that is quite unique is number one, we don't keep a record of any of your conversations. Anything that is in your chat history is encrypted in a way that we cannot decrypt. So we are technically prevented from accessing your history. And number two, we don't use any of your conversation or any of your prompts to do model training and refinement. So there's no chance that your information kind of gets leaked out. And that also means that number three our staff don't read your conversations because they can't. And if we can't get access to your information, it means a government coming with a subpoena or a court order even if fully legal well we cannot disclose information that we ourselves don't have access to. And so it's sort of the only chat bot AI chatbot where there's a strong technical guarantee that your conversation stays private and that is what is different. And back to your point Google could technically build a similar system. They just don't have the economic or business model incentives to do so because their business model is flipped. You know I'm a subscription business and I take all of my money directly from subscribers and that means that my incentives are aligned with our customers. Our customers pay us because we're private and the instant we're not private they stop paying us. So I have a financial incentive to keep doing that. But if you use Google you're not actually Google's customer. You're the product they're selling to the real customer which is the advertiser. And that is a misalignment of incentives which is never going to really put any real pressure to compel them to protect their privacy because protecting your privacy unfortunately goes against their fundamental business interest.

Yeah. So, you're talking about Lumo your chat history is saved, but it's encrypted in a way that we cannot decrypt it. So, it's encrypted essentially with your private key that we don't have access to. We just have an encrypted copy of that that we cannot decrypt. And that is the technical difference and by the way it's all open source so people can take a look and see how it works but yes your history, you know, whatever sensitive discussions that you have had with Lumo that's your data and we cannot get access to it.

We use all of them. So, you know, but only open source models we're strict on open source models so we have basically, you know, for example we have models of Mistral we even have actually OpenAI's open source model. We've got the Chinese open source models as well. Yes. And Kimi K2, you know those models like that. But of course then we bring them in, we modify them and we make sure actually we use the best model for every single query. And this is important because you know all the models have their own biases. So if you were to ask a Chinese model certain questions, it wouldn't give you let's say the correct answer, right? And we ensure that we do give the correct answer and we also try to be as neutral as possible. You know we don't want to be a right-wing model. We also don't want to be a woke model. So we try to train and calibrate our systems to be sort of let's say as neutral as we can with as few biases as possible. And this is something that we actively do.

Well, Lumo has been on the market for five or six months and these guys have, let's say, been on now at this point for over three years. So we will always be a little bit behind let's say the cutting edge because we simply haven't been around as long, right? But as you say what is interesting about the AI revolution is the gap between the best open and the proprietary is really really quite small and that means that even if we're not 100% of the way there it's actually pretty close and we're quickly adding the feature set to complete the missing features. So you talk about memory. Memory is something that we're actually working on right now. That is probably going to come to Lumo, you know, within the next month or two. We have a major release essentially every two months. And it's a field that is improving very rapidly. So I think it's a gap that we close and it's a gap that is pretty easy to close because of how good open source solutions are in this particular tech revolution which is quite rare when it comes to tech revolutions overall. And I think that's quite important. On the topic of memory and sort of personalization. In fact, there is a possibility for you to personalize Lumo. You can tell Lumo how you want to talk to, how you want to respond. But there are certain features that we are going to decline to copy because we don't think they're right for society. So, I don't really want Lumo to begin to change his answers and behave differently because of inferences it has made about your personality. You can do that optionally if that's what you really want by opening up the personalization and instructing it to do that. But by default, it's not going to do that because I want to avoid some of the filter bubble stuff that we had on social media manipulation. That's what we call it in a human relationship. Manipulation, right? I want to avoid some of the manipulation because I think that's harmful. And you know, social media, the problem was if you were on the left, it kept feeding you more left content and it made you more and more extreme. It did the same to people on the right. And then we live now in a completely polarized world and then we wonder why, right? So I think Lumo and AI that responsibly it should not cater and try to reinforce our worst impulses even if that gives higher engagement. It should actually try to do more to steer people in society towards the center to seeing both points of views. And that means I would say probably less personalization by default because personalization is what gives rise to a lot of that. So some people say oh well but the fact that you don't do this is a bug. It may be a product bug but it's probably a feature for society and this is a trade-off that we try to get correct.

Yeah. It doesn't get there because it's not profitable enough and it's not their business model fundamentally. And the reason why at Proton we transformed the business into being having a nonprofit as the bigger shareholder is because that is the way in which you resolve this conflict. So it's kind of funny because you can sort of see intentions from direction of travel. Sam Altman has spent the last three years trying as hard as possible to stop being a nonprofit whereas Proton went the other direction and actually went from a for-profit company that could have stayed for profit into something that was primarily owned by a nonprofit. And I think that speaks a lot about intentions of course but you need to have that structure that is the essential structure you need to have in order to ensure that you can carry out the vision that you spoke in the long run. And a foundation structure really puts it what it's mandating is it's a legal structure that obliges you to put society's interests above financial self-interest and this is the basis of a nonprofit foundation under Swiss law and that's a key I think innovation from a business model standpoint because it gives you the flexibility and the freedom actually to make the decision that is good for the customer but maybe not always best for the bottom line.

They're all here. They're all here.

So maybe a popular opinion among your audience but I would actually reject a bit the comparison with the crypto foundations because if we're being let's say completely frank a lot of the crypto foundations that were created in Switzerland and this was a lot of them in 2017 around the ICO era we call it. Yes. I mean let's be honest most of those are scams right many of them were the vast majority and these foundations were not here for social benefit right these foundations were created because it was a convenient legal structure to legally launder large amounts of money received from unwitting investors who 99% of the time were defrauded at the end of the day. So I actually feel it was a bit of a mistake for Switzerland to cater to this business. But it was early days. They didn't realize a lot of those things happened and some of these foundations are still around. They still around having cashed out tons of money to the founding teams. And then what happened? They went off and they did things that ultimately didn't have utility. Ethereum maybe is kind of one of the rare exceptions where I suppose some value was created, but a lot of these promised we're going to have this new blockchain to do this thing that will transform the world and then it's vapor where it hasn't shown up and they just been sitting around collecting salaries year-over-year, not shipping anything and these networks are dead.

We did. We did. The problem is we had all the bankers and lawyers in Zurich we're in Geneva so we're not in the ecosystem but Switzerland same country right they all showed up and say oh you know we're going to do this great ICO for you you're not going to have to give up any equity you're going to create a new token or coin or blockchain and you had to resist this and then we promise you at least 100 million and some of these things are raising billions you have a big brand you can do all this and it's like wow This is great. Actually, I went to Zug in fact because I was intrigued and like if someone offers you a billion, you got to go and hear them out, right? So I went there and I went for the board and of course it was tempting and they were signing entrepreneurs up left and right every single day. But I realized like if I go down this path, what you're essentially asking me to do is defraud the customers and the community that has put great trust in the business for essentially personal gain. And it was something that was simply incompatible with what Proton stands for. It's something that is incompatible with my values. I could not personally do it. So we refused and then people looked at us like we were idiots. Like, okay, yeah, why are these people so stupid? The people raising hundreds of millions left and right and these guys are going to sit here and struggle in Geneva, barely having any money to get by and trying to grow their business. Like we look like idiots, right? But in retrospect, I think it was the right thing to do because it just wasn't correct. It was simply a fraud and we didn't want to be involved in that at all.

And now what makes the Proton foundation structure different. So, first of all, it's not created to defraud people, right? This is number one. And as the foundation didn't come from outside, you know, investors or users who put money in that wasn't what we did, right? But you're correct. It has to be sustainable. This is the key thing. You know, a foundation technically doesn't have a profit interest, but if all your revenue is coming from a couple donors, guess what? You're in the pocket of those donors. You work for those donors. And this is why Proton has what I call a hybrid structure. It's not purely a foundation. Actually, it's a foundation as the bigger shareholder of a for-profit company because profit still needs to be there. But this is sort of a self-fulfilling prophecy in that the two sort of reinforce each other. And the way it works is the company does not have to take decisions that are bad for users because it's not going to come under pressure from its primary shareholder, a nonprofit to do that because a nonprofit actually cannot do that. And if the company were to go out and let's say tomorrow we decided to go into surveillance capitalism business model. Well, the foundation as the controlling shareholder is just going to block that and that's the end of that, right? It's never going to happen. So the foundation actually gives the company the freedom to do the right thing but the foundation has as this asset actually a big shareholding in a highly profitable business and that means the foundation doesn't need to go out with a pen to beg for money just in itself it can collect the money from the company to operate on its own and that means the foundation is actually independent completely independent of any outside force and so I think it's a novel structure if it was just a company it wouldn't work it was just a foundation also wouldn't work but the combination of the two of them together with this shareholding structure. This gives us a solution that is so you know I say it's a self-reinforcing and that is what we did that was really kind of innovative from a business setup standpoint. It hasn't actually been done before there were really I think very few examples of this. So I'm under some pressure because if we fail then we've shown the world that this model doesn't work. If you succeed hopefully others will say hey you know I can do business differently. This worked for Proton it could work for us. And I think this is the future of how you get to responsible capitalism.

Ship sailed for them. Yes.

Proton has from the beginning always been involved in unsustainable businesses that we're allowed to continue because we have stakeholders who are not purely profit driven. And I'll give you kind of an example of this. We today have one of the world's largest free VPN services and unlike most free VPNs which monetize your data and abuse your privacy to make money our free VPN is in fact not monetized doesn't log doesn't track doesn't even have a bandwidth limit right you can use it for as much as you want there's no limit but not only that we also spend millions of euros every year in building R&D to ensure this VPN works in Russia and works in Iran. And by the way, these are two countries which are under sanctions. So even if these users wanted to upgrade to pay us, they can't because there's no PayPal, there's no credit cards. They're locked out of the bank and said, yeah, they can pay in crypto but that's kind of a small percentage of population, right? So we're actively investing millions to be present in a market where there is no prospect for monetization because legally it would be illegal for them to pay us. So that's clearly money losing business and Lumo well it's a bit early to see in Lumo but you know so it may yet turn a profit at some point. But what I guess the point is we don't need to turn a profit on all of our businesses because we are not under pressure to do that, right? If I was backed by VCs, they would have told me years ago, you know, kill off your Russian business. This is stupid. This is a waste of money.

And you saw all the Western companies simply left Russia when they couldn't collect money anymore. Because that was the prudent business thing to do. But our structure allows us to engage in activities in business lines that are not profitable, have maybe very little prospect of being profitable, but are aligned with our mission and are good for the world. And this is why I do want people to pay for if you're a happy user, please pay us, right? Even though you're not obliged to do it because we also have a free version of it. It's subscription based paid version. Yeah, there's a paid version. But let's say the free version is pretty good. Like all of our products are probably too good to be honest, right? But that is our mission. Our core mission is to make privacy accessible. I don't want privacy to be a luxury good. This is maybe Apple's model, right? Although they don't really believe in privacy. Privacy for the mission is marketing. I think privacy is a fundamental human right that needs to be available for everybody. It can't be a luxury good and I want it to be open for anybody that needs it. And that comes at a cost but it's a cost that Proton is willing to pay and we are very fortunate to be positioned that we're able to pay that because we do have other paying customers who make us profitable and they are essentially subsidizing the other parts of the business that today do not make money and cannot make money.