Generative AI, Privacy and Data Protection | Oxford Generative AI Summit 2024

Thank you very much for coming back after lunch for this particular panel, where we're focused on the topic of generative AI, privacy, and data protection. Another important area of regulation for generative AI. We've got another important set of questions about the intersection between this fast and bold innovation and how data protection law, which has been around for a long time—you can see Simon and I have been around for a long time because we had hair when we started off in this profession over 20 years ago—is now facing the challenge of how to apply this long-standing set of rights and principles to this new emerging context of the technology, and how we can reap the benefits of the technology but still keep those rights and principles in data protection intact, which the public still value and think are important. In the panel today, we're going to look at some of the questions about risks, safeguards, and mitigations, and how they can be balanced. We're also going to think about data protection regulation in the context of other forms of AI regulation, and how data protection law is going to remain relevant and how it may need to intersect with new forms of AI regulation, e.g., the EU AI Act. We've got a great panel here today. I'm going to get them all to introduce themselves in a minute. We've got perspectives from business, from the regulator, and from an academic and someone who's working in a think tank as well. Really looking forward to the discussion. Also, we actually had a panel here last year for those of you who attended the 2023 Summit on generative AI and data protection that I hosted as well. I think what's going to be interesting to reflect on is what's moved on in the last year, how sharp some of the questions have got, or have we got some solutions to some of the things we were discussing last year as well, because I had the pleasure of chairing the panel last year. To open this up, I'm going to go to each of the panelists to ask them this first question, and then they'll introduce themselves and talk about their current work and role in this topic. The first question for the panel is that data protection really intersects with a number of different parts of the life cycle for generative AI in terms of the inputs, the enhancements, and the outputs, and in development and deployment. So from your different perspectives, where do you see the most significant data protection risks, and how does data protection governance need to evolve to address those risks? I think we'll go to Sophia first from the ICO, from the regulator, to kick us off.

Thank you. Not a difficult question at all. Thanks for inviting us. Just to briefly introduce myself, I'm the Group Manager for AI Policy at the ICO, the data protection regulator. I manage our team of policy and technology experts who are tasked with developing the ICO's emerging positions in terms of AI. We monitor the technology as it evolves and it surfaces new issues. We are updating our guidance, we lead our cross-regulatory cooperation, so we represent the ICO at the Digital Regulation Cooperation Forum, which I think was one of the legacies of Simon's work at the ICO as well, and our cooperation with regulators in the UK and internationally, and a variety of other stakeholders. We're producing guidance, for example, at the moment with the EHRC, the Equality and Human Rights Commission in the UK, and we continue to engage with counterparts internationally at EU level as well. Going back to the question around what are the most significant risks that generative AI surfaces in terms of data protection, I would say it's one of those iterations of AI that has presented most challenges for data protection because there are things that are implicit in how it's being developed that are contradicting or creating tensions with requirements under the GDPR. More specifically, the issue around information rights. Under data protection, you need to be able to fulfill information rights where data subjects are making requests such as exercising the right to erasure. Someone may want to ask you to remove their data from your model or from the training data, but in the context of a model that takes days to train, millions in terms of computing capacity, obviously that comes with market forces that make it practically really difficult to respond to information rights requests on time. You also have accountability issues. There is a lot of ambiguity around who is the controller, who is the processor. What I'm trying to do right now is nudging the conversation towards our generative AI consultation. Back in January this year, we launched a consultation on that specific point—the intersection of data protection and generative AI—and we chose to focus on those five areas where we see the most risk because there is a lot of regulatory uncertainty at the moment. Those areas are: the information rights piece that I just mentioned; accountability—who is accountable for what; purpose limitation—the fact that under data protection you need to be specific in terms of why you process data, but what we're seeing in practice is a lot of ambiguity or confusion from stakeholders around what is a clearly defined purpose, in a way that is also intentional with the notion of general purpose AI; you also have the fundamentals of the technology itself, like the fact that you consume a lot of data and that creates tensions with data minimization; our consultation also touched on accuracy—what our expectations are in terms of accurate outputs and inputs; and I think there is also a question around what are the downstream implications of this technology, because the more we move into an area where those models are being used to power AI agents, we will have to start to think about what are the implications for Article 22. So I can go on forever, to be honest, so I'll just pause here.

Thanks very much, Sophia. I think that gives you a really good insight in terms of what the regulator is considering in terms of those key risks and areas of regulatory uncertainty, and also obviously, if you're new to some of these topics, go on the ICO's website—there's lots of information there about the ICO's current positions that you can start to look at as well. I'll turn to you now, Elena, to give your perspective on these data protection risks and some of the changes to governance and organizations.

Thank you. My name is Elena Simperl. I'm the Director of Research at the Open Data Institute. I'm here with my colleague Emma Thwaites, who just took a picture of me. She is, among others, our Director of Policy, so possibly more of an expert to respond to some of these questions. I am also, for my sins, a Professor of Computer Science at King's College London, and so I've been working in AI. I'm going to talk a little bit about the work we've been doing at the Open Data Institute as part of the Data-Centric AI program. This is a program of work we launched at the end of last year. For those interested, there's a 15-point plan of things that need to happen to create a healthy, trustworthy data ecosystem for AI, and that plan is not just for the ODI and shouldn't be just for the ODI to tackle, but it will be a multistakeholder effort. First of all, let me just congratulate you on the question, because often when a very challenging question is asked, it is imposed in such a nuanced way. Yesterday, Tuesday, we published a first version of what we call our AI Data Taxonomy. In that taxonomy, the aim is to unpack what we mean by the 'no data, no AI' headline that many of us have embraced over the last year. You are already making some of these distinctions: inputs, enhancements, outputs, also looking across the whole AI lifecycle. All these different types of data sets and the way data is collected, managed, assured, governed in all these different scenarios is different and tackles important challenges. Whatever we do, we will need to genuinely put an effort into understanding the characteristics and the challenges associated with—I'm just going to give some examples—data that is in the public domain and has been used to pre-train a foundational model; synthetic data that has been created because the actual data can or should not be shared; fine-tuning data that possibly a different organization has created and used on top of an existing foundational model to tailor it for a particular purpose; prompting data that a consumer is adding to a consumer-facing tool like ChatGPT to ensure it responds to whatever needs it has. All these different types of data come with different challenges. The processes, technical and otherwise, to create, maintain, and assure these data sets are different. So without making your job more complicated, because you will need to look into all these different scenarios individually, this is also an opportunity for the providers of privacy-enhancing technology, data protection technology, to come up with really interesting innovative solutions to make a difference. I will just say a few things now with my scientist hat on. In terms of data protection and inputs, I think there's a lot of research that needs to happen. Techniques like model unlearning, for instance, are supposed to help with some of those challenges you mentioned around managing to make those models more mindful of privacy without having to actually bend the weights and spend another hundred million on retraining. On enhancements, we've published a public policy intervention in June around rights, which includes people's rights but also includes rights of those people involved in the supply chain of a data set. In particular, if you're thinking about safety, everyone's talking about AI safety. Safety testing includes a lot of contributions from gig workers that work on digital platforms that, for instance, are supposed to make judgments on questions around harms like toxicity, racism. They do that using their knowledge and experience, so they add their own data to these data sets that are then part of fine-tuning and safety tailoring these models. On outputs, I think there is an opportunity and a risk. I don't know how much the general public understands that at the moment we are putting our knowledge, our data, and our experience in the form of prompts into a handful of proprietary digital platforms. As an academic and a tech optimist, I would like us collectively not to make the same mistakes we've made with previous digital platforms, because we're just prompting into ChatGPT—hundreds of millions of users, several times a day—we're putting that data into these systems, and that data is not available in the same form across the ecosystem.

Thank you very much, Elena. Just to pick up on some of your points there around potential technology solutions and what we call privacy-enhancing technologies to address some of the concerns that we've got, particularly how can you technically erase personal data connected with a generative AI model because of the way it's constructed in its tokenized form? How confident are you, as a computer scientist, that we may be able to find some of these solutions which can still give effect to people's data protection rights? Where do you think we are on that road?

Okay, so let me start with a disclaimer. I'm not working in privacy-aware AI; I'm working in symbolic AI, that's the old-fashioned AI that didn't seem to work but now everyone needs. But I think it's quite challenging, and it will very much depend on how that model was trained and what sort of safety testing was applied before it was released. In some cases, depending on the type of data we're talking about, it is going to be hard to almost impossible. Correcting the models is a huge challenge, both from the degree to which the end user or even the developer is able to intervene in a predictable, deterministic way, but also because it is quite hard to locate in those models, in those weights, the exact point where this happens. I'm speaking to an expert audience, but just in case I can't make my point clear: with deep learning, imagine you have a fruit salad and a smoothie. Google search, the AI deep learning AI that is in Google searches, is like a fruit salad. If you want to take a piece of apple, you will take the piece of apple; bits of it will have spread in the salad, but you still can take most of it out. In the smoothie, which is the generative AI part, good luck with that. You will be able to find some bits of the skin, but it's much, much harder. Still, I think there are some ways, depending on the training data and depending on where else that data is available, to isolate those effects.

Thank you, great. We may have to come back to this next year. I think you heard it here first anyway: regulating generative AI and data protection is like regulating smoothies. Trademark that one. Thanks, it was really great. It's not mine, by the way. Great insight and follow-up to that. I'm now going to turn to Simon, who I used to work with when I was the Deputy Commissioner at the ICO before I came out to work in the private sector. Simon is now working in the private sector for a very data-focused company. You're really dealing with some of these AI governance issues at the coalface in terms of products and services you're launching, Simon. So look forward to hearing your perspective.

Thanks so much, Simon. Just to drill down into that supply chain issue right now: how do you think companies are grappling with it in terms of the contracts that they're having between the controllers and the processors, or this concept of joint controllership which the ICO has perhaps put forward as a concept that may need to have more prominence in the generative AI space?

Okay, thanks so much, Simon. I'm going to move on to a different topic now, which I think a number of you would have been hearing about, and this is actually focused on the inputs and the training phase for generative AI models. Obviously, given the scope of data that goes into the training process, personal data will be harvested during that process. The companies obviously don't really target personal data in building a number of these large language models, but personal data is involved because of the sweep of the data. So we have a question under the GDPR, under Article 6, in terms of what is the lawful basis for that process of AI training. It's an interesting question because obviously for those of you who know a little bit about GDPR, there are a number of options in finding a lawful basis in Article 6. One is consent, another one is necessary for the performance of a contract, and another one is whether it's necessary for a legitimate interest pursued by the controller, e.g., the company, or in the interests of other third parties. But it also needs to be subject to a balancing test against the rights and freedoms of the individual as well. The discussion and the debate that has started to emerge has really looked at what lawful basis is feasible. Can you feasibly get consent for large-scale training of AI data? Obviously, that could be challenging. It could be challenging as well whether it will always be necessary for the performance of a contract—a contract won't be in place between the individuals and the company undertaking the AI training. So the focus has actually fallen on this legitimate interest condition in the law. That's why you've seen some big news stories recently in relation to Meta of PA, the training relating to their models in the EU, because of the challenge they've got currently engaging with the European data protection regulators in terms of whether that is actually meeting the legitimate interest condition in the GDPR. So we're seeing quite an important moment now in the trajectory of generative AI about how it's interacting with data protection law. So my question to the panel really is: what views and thoughts they've got about this tricky question of organizations relying on legitimate interests, and what they might need to consider? And of course, it may be in different use cases and context as well. So Simon, could you start us off on this one and talk about your perspective on it?

Thanks, you kicked off really well. In terms of that risk point, is part of your message there that we really must consider each use case on its merits as well? Because I suppose the concerns which have risen particularly about the social media companies, the particular maybe heightened risks about using social media data for training as opposed to other data—would you differentiate between any the context?

Okay, thanks. Sophia, over to you. The ICO has put some guidance out on legitimate interest as Simon talked about, so it'd be great to hear about the ICO's approach to this question.

Yeah, I think this is a really interesting discussion. Legitimate interests in terms of generative AI is an area we looked at as part of the consultation, but we looked at that as a lawful basis when you web-scrape data. So there's a difference in our approach when it comes to first-party data—for example, user data that Meta may want to use to develop their models—and a difference between that and the context in which data brokers or developers themselves are basically web-scraping social media or other unidentified sources, because they rarely do actually disclose what they actually scrape. So starting with the first-party data context, in terms of legitimate interest when you're using that basis, you still have to be able to fulfill people's information rights, but those rights are not absolute to the extent that they are, for example, when you're using consent as your lawful basis. When you use consent, you have to ensure that consent is revocable, which we do say is really difficult to actually enact in a generative context. That's one of the reasons we turned toward legitimate interest as the most likely available basis to train generative AI. I think we have been public about the fact that we engaged with Meta on their use of first-party data, and our experience has been that it was a really productive dialogue. In none of our engagements do we end in a place where we can actually provide some kind of certification for the lawfulness of the processing in perpetuity, but we did manage to improve aspects of transparency towards data subjects, and that's why people were alerted to the fact that Meta was planning to do that. I think that's a direction that regulators should be heading towards: do a lot of upstream engagement so you kind of nudge a company into compliance before bad things happen. So that's in terms of first-party data. When it comes to web-scraped data, I think the harm we see is the loss of control of someone's personal data. I think we have to be realistic here: when someone scrapes your data to develop a model, these institutions most of the time will not only use your data for that specific model. There was a press release by one of the big players in the space who was saying that they have trained millions of models in the last couple of years. What we're trying to do as regulators is to translate the legislation in a context in a way that somehow mitigates the risk of too many gateways that enable abuse of the legislation, if that makes sense. We don't—perfection is a really tricky thing to achieve in data protection, but we're trying the best we can.

Thank you very much, Sophia. I think that's really useful. Flowing on from what Simon said about how you can differentiate the use, and actually learning from how the UK regulator has approached this question of how Meta have used social media data in their model and the improvements that the ICO got from that process, I think that's interesting to learn and contrast from what's happening in the EU as well. Elena, I'll turn to you. I think you were touching on some issues about inputs when you spoke previously, but have you got any particular views on how this GDPR question could be approached from your distinct perspective?

Yes, following from what you were saying about the fact that lots of these conversations are very technical and we're focusing a lot on the law, I think I would agree with that. I think even if there is this guidance, even if this dialogue with the relevant technology providers is happening and is encouraging, public perception and public trust are different. So it's nice that a company is now more transparent about the fact that they've started to acknowledge that they're using public data to train models, and maybe strictly speaking from a legal point of view they are within their rights to do that, but that does impact the trust that the public has, including myself as a user of some of these platforms, not just in the provider of the service but also in the regulatory and political actors who are supposed to protect my rights. So maybe some of that could be improved with more awareness campaigns, with more digital literacy and data literacy. But if we're thinking about harms down the line, misinformation is a huge issue, and these models are perpetuating and exacerbating misinformation. A lot has been said about the role of these models in elections, in how we engage in the democratic process. Some of that can or should be regulated—I'm not going to comment on that—but I would encourage us to think beyond the law, because there's what the law says and what the regulator can and does enforce to show that regulation is working in practice, and then there's the public perception and trust. As a scientist, I think there could also be a huge opportunity to think a little bit more disruptively. We are talking about a time when the way in which data is used to innovate with technology is changing. Lots of the assumptions are changing because of how the technology works. Isn't that an opportunity to actually think a little bit more about the whole architecture and technology stack of these platforms? Why do we have to have the sort of data-scraping behavior and centralized data controls that these platforms have been built on? Could this not be an opportunity to actually think about different architectures that give people from the very beginning more agency and control over their data? The two things aren't directly related, but there is a relationship, and with everything in technology, timing is everything. If everything has changed and a lot has, this is an opportunity to actually rethink the whole how personal data management is implemented in these platforms as well.

Can I come in? I probably should have clarified more about the part of our engagement with Meta. One of the things we achieved is basically, apart from the fact that Meta contacted their customers to tell them they're planning to do that, you also had an easy way to object to that processing. So I personally have objected to that processing. So in a way, we empower data subjects to just stay out of it basically. And in terms of, I really support the more kind of disruptive and innovative way of looking at the problems we're facing, because for a lot of what we're seeing at the ICO, we're aware of the fact that law may not be the actual practical solution. It may be the case that we need to think more creatively around new institutions when we want to introduce, for example, data trusts or data intermediaries, or what kind of innovation do we want to promote and tell developers that we actually need, so we don't impede their experimentation with this technology.

Yeah, thanks. I think it's really an interesting discussion about where this nuanced position currently is in the middle, where we can see the technology can continue to grow and evolve, but we are actually able to build in practical safeguards. Sophia actually referenced one there in terms of the right to object to that processing by Meta. So I think all of those points are really interesting. And then going to the bigger picture, I think it's a great wider discussion about also the policy aspect of maybe we don't longer term have to accept the status quo. We've got discussions about, you know, Nigel was mentioning this morning about the utility of smaller models which may therefore use less personal data, whether data protection regulators can actually enforce technically their way through the law. But I think data protection regulators can have a voice in this conversation about it being a sort of data protection by design conversation. So I think some really great points there on the panel. We want a bit of time for questions, so I'm going to do a reasonably quick final question to finish off to the panel, which is just to reflect on the fact that there are other forms of regulation emerging. We've got the EU AI Act obviously coming towards implementation, and that's focused particularly on safety risks, particularly on high-risk models, etc. So it'd be good to get the panel to reflect on what's the position: do you think data protection should take against other forms of regulation which are also going to be important related to generative AI? Elena, could you just give us your thoughts on that quickly?

So I'm going to focus on a very specific point of the EU AI Act that I know a little bit about, and that is the transparency requirements, and in particular transparency requirements about data. Now, with foundational models, one area that we have seen flourishing where there's a lot of innovation and a lot of potential is synthetic data. Synthetic data is used in situations where the actual data can or should not be shared, but it is also extensively used in creating fine-tuning and other types of data that would be too costly to create. Well, in some of those situations, actually collecting data, including personal data, genuine data, is quite vital. So we need to distinguish between these different use cases, but more importantly, we need to be clear about what things like a sufficiently detailed summary of that synthetic data looks like. So yes, there is a law. I mean, everyone says that the AI Act tries to regulate AI as if it were a microwave or a product safety law at the end of the day. But still, it does create more awareness on the needs for transparency across the AI lifecycle, and we need more guidance on what's appropriate and useful for the different types of data sets, in particular synthetic data sets, which are playing a huge role because at the moment that's the only way or the cheapest way to deal with the fact that model providers are running out of real data.

Okay, thank you. So that's quite a good point: the EU AI Act is going to ask about transparency which maybe GDPR won't ask, and that can be complementary. Sophia, what are your views? Working in a data protection regulator, I'm sure you have a view on the relevance of data protection, but how do you see these different forms of regulations sitting with data protection?

I mean, the AI Act itself says basically that it complements and works alongside the GDPR. So in a way, there is no tension there. Transparency is already a legal requirement in data protection, but the kind of transparency that has data subjects as its focus—data protection regulators will never tell you to release all your training data for everyone to see because that is disclosure of someone's personal information. I think a really useful addition that the EU AI Act provides is this notion of systemic risk, which is something that we have discussed—the systemic risks that AI gives rise to in the context of the DRCF, for example—but data protection is not really the best vehicle to address that level of risk. I think the EU AI Act has the ability to really empower data protection authorities in the EU because it really makes explicit requirements that DPAs should have anyway when it comes to the processing of personal data in the context of AI development and use. One advantage of the GDPR compared to the EU AI Act is that point around being a principles-based legislation rather than a product safety one that is more rigid and not really amenable to change or interpretation. And the fact, I would contest, that data protection as a framework has the capacity to look across the supply chain. So we're not interested just at the moment that you introduce a model into the market; we're interested in everything that comes before that.

Thank you. I think that's a really good overview of how the two can complement each other and what each of the pieces of legislation brings. What's your view on this, Simon?

Okay, well that's an interesting note to finish on. Let's throw out and see if anybody's got any questions for the last few minutes.

I've really liked your point about being able to use engineering as another tool in addition to law and standards. To what degree can we as engineers get ahead of this cycle and insist upon a higher standard? I know Karl Friston and his work on active inference is one way they're showing great results—90% less data for the same result. How can we as engineers help?

I'm happy to try. I was told that I'm not supposed to touch the microphone because then something would change, but anyway. So I think we can and we should. What I observe in my own practice or when I teach is that unfortunately at the moment there's still a bit of a gap between two communities: the engineers or the AI scientists who sit typically in a computer science department, and researchers or practitioners in responsible or trustworthy AI. In that latter area, there are a number of tools like impact assessments and the likes that have been around since 2017, 2018, so they're quite established already. There are things like data cards and model cards, a number of transparency and accountability tools, ways to check and analyze your data before it goes into a model. But those frameworks aren't very much used by engineers, and there's lots of research that shows that even when they are used, they're not very well understood. I'm going to give one example, which is explanations. There's a lot of research that shows that it's actually the engineer, the expert, who tends to over-rely on those explanations even when they're basically random. So a lot of work to be done. I think the way forward would be to take these responsible AI frameworks and tools and test them in practice. I think companies and managers of engineering teams should make an effort to see how these toolkits apply to them and what small measures they could put in place now. I'm going to give one example very quickly. Nigel mentioned small models. Now, with small models, even if they're deep learning, you have a different control, you have different remedies that you can put in place. And as most companies will in time use these sorts of models or retrieval-augmented generation methods, where accuracy and control over the data and the provenance are different, if we manage to take this responsible AI toolkit and really apply them in practice, I think a lot of these issues will not disappear but will have fewer impacts down the road.

Okay, I'm not going to go too close to the microphone. I agree with Simon. I think basically getting this cross-disciplinary conversation to work in a productive and well-informed way is a really complicated exercise, and we're actually learning it as we go along. A lot of the time when we're discussing new policy positions, we will have our technical advisers in the room with legal advisers. I think as long as there is goodwill from all the teams involved and we are humble and accept that there are certain things we do not understand, I think we can move forward with this, but it is going to be a learning process.

Okay, thank you very much. Do we have a question over there?

Thank you. You sort of touched on this a little bit towards the end, but I essentially wanted to ask if you think that the current explosion in generative AI that we've seen in recent years could have happened in the current GDPR landscape that we're seeing here in Europe, or whether, as you sort of alluded to, that's stifling the ability for such new explosive innovation to occur.

That's directly to you. First stab.

As the regulator, I will basically reference something that I think Nigel said earlier today: that necessity is the mother of invention, and regulation basically articulates what is necessary to be done. Responsible innovation, I would say, is innovation that operates within the bounds of the law. I wouldn't call the EU an environment that's overly regulated, but obviously I work for a regulator, I'm biased. I think one pattern we see in general is that a lot of stakeholders who are not founded in the EU or the UK start processing data without actually having EU legislation at front of mind, to be honest, and they do move fast. If they break things or they don't break things, sometimes we may learn about it in a couple of years rather than now. So wait and see, I would say.

Okay, well I think we're just about at time, so I can't take any more questions. I think we've really delved into some of the key questions in data protection in this panel. Hopefully, I think you've provided some different perspectives and really tried to illustrate how data protection is practically interacting with generative AI. It's a reality, it's here, but equally we can see some pathways forward. We look forward to seeing how different solutions emerge in the coming years. It may be that there are solutions on a range of areas: some of it can be addressed by regulators through helpful guidance, engineering solutions, and Simon's positing that maybe eventually we might see some law reform as well. But I think we can see data protection will play an important role in this area because ultimately it's about protecting people's data and it's about trust and confidence, and that's always going to shape how we use the technology. But thank you very much to my excellent panelists.