Michael Kratsios, Managing Director, Scale AI | Oxford Generative AI Summit 2024

Yeah, absolutely. First off, thank you so much for having me. I'm delighted to be here. My fiancée actually is a graduate of Oxford, and I always try to find excuses to come here so she can show me around this beautiful campus. But you are correct. I think the debates around AI policy had started long before ChatGPT was released, and a lot of the debates, if you remember, in the sort of 2017, 2018, 2019 period were more centered around, I think, two things. One was a horse race national security sort of question or debate between the US and China. So, you know, who is leading, who's investing more money, who's doing the right things to sort of create a national strategy, you know, who's going to come out on top. And the second set of conversations were really tied into a lot of this debate of the impact of automation on the workforce. So at the very tail end of the Obama administration, if you remember, in the sort of 2016 timeframe, there was a big report that was released which discussed all about sort of what are the potential impacts of AI on the economy, on workers, on the workforce, and so on. So I think when we were in government, what we spent a lot of time thinking about is, you know, the US does not have a national AI strategy yet. It is something that's incredibly important, and what can we do to develop one? And we spent sort of the first year and a half doing a bunch of sort of like listening, touring around the country, bringing people into the White House, and trying to like build attention around the issue. I mean, it feels kind of crazy today to say this, but it took a lot of energy to get people to care about AI at that moment. Like it wasn't front page news, there weren't hearings on the Hill. Like you really had to twist people's arms to say like, this is really serious for the country, what can we do? And to President Trump's credit, he signed not one but two executive orders on artificial intelligence. He released one which was a national strategy called the American AI Initiative, and the second was an executive order around the use of artificial intelligence in government. So I would say I think there was a lot of effort done during that period, which I think we're all very proud of what we were able to accomplish, and I think set up this next administration to kind of build on all those efforts. Just a quick summary of that strategy, so you kind of know where we were, where our heads were at. I think first and foremost was around research and development leadership. So actually, for the first time ever, accounting for the amount of federal dollars that are spent on AI research and development, and to actually double the spending of AI research over the course of two years, which was a really big deal back in 2019. The second was on sort of workforce-related issues. How can you use the federal government's grants and fellowship programs in order to direct people towards more AI-related secondary sort of programs? The third was around sort of regulatory issues, and we'll get into that in a second. But sort of the first ever binding regulatory guidance around artificial intelligence was released by the Trump administration in 2020. And the fourth was around international engagement, this understanding that, you know, we had to work with our partners and allies if we wanted to counter the threat of an aggressive CCP.

Yeah, if you're an AI policy wonk, the last two years have been probably one of the most thrilling roller coasters of your life. So we started with no one caring about AI regs, and then suddenly you had a bunch of sort of Silicon Valley technologists running to Capitol Hill saying that we need to create sort of a new IAEA for artificial intelligence. There were draft ideas about entire new departments or agencies at the US government to sort of do all this stuff. You had Breton and the brilliant folks in Brussels deciding that, you know, they need to take charge and modify a rule they've been working on for years to somehow account for technology that they had no idea how it worked or they even understood how to use it. So I think you had sort of all these people kind of like, you know, jockeying for, you know, who could get a quote in the front page of the New York Times around kind of what the future was for AI regs. And it's been an interesting watch because you sort of had this sort of crescendo of desire for sort of these centralized regulatory apparatuses. And at least in the US, I think there's, you know, some sanity has come to the conversation, and we've sort of veered away from that. And I think you could argue there's probably bipartisan consensus on Capitol Hill that the idea of a singular AI regulation is definitely untenable, and that the future of AI regulation in the United States is centered around a sector-specific, risk-based approach to regs, meaning that AI is going to be sort of permeated across every facet of our society, and we already have a set of agencies that cover all sorts of different use cases. So the folks at the FDA, for example, are probably best suited to deal with AI-powered medical diagnostics. The people at the Department of Transportation are probably best suited to work with AI and autonomous vehicles. And I think kind of that's where we're going now. The second-order question of that is like, do we have the skills necessary at those regulatory agencies to be able to account for these new use cases? And that's something that's going to continue to be a work in progress. But I think we pretty definitively have moved away from this sort of like centralized single rule. I think the one area of it's still a little bit open to debate, and I think people are still trying to sort out what to do, is, you know, I think there's a lot of clarity, or generally in my mind, on how to think about the use of AI. The real question is what do you do with, or if anything, with the foundation models themselves. And there was action taken by the Biden White House to essentially — now it's in rulemaking — to require any US model developer that trains over a particular threshold to disclose to the federal government by law the testing that they have done for risk to the federal government. So that's, you know, you could argue that's kind of a lightweight approach to thinking about it. If you're more on the acceleration side, you may argue that that's a burden, that that particular threshold is one that many people are going to be surpassing in the very near future, so it's not a very future-proof rule, for example. So that's kind of where we stand today. My general view is you need a sector-specific approach to it, and I think that's kind of how it's going to work out in the years ahead.

Yeah, so I think to the UK's credit and to Prime Minister Rishi's credit, I think they spent a ton of effort in sort of launching the first AI safety sort of institute in the world and trying to sort of normalize the policy priority of examining existential risk on behalf of a country. You know, the government itself has to do that rather than the companies themselves doing it on their own. And, you know, I think there's an argument to be made that there are certain types of use cases that these models have that, you know, the government probably wants to look into. And you can imagine sort of extreme sort of biothreat issues or things related to nuclear or whatever may be. There's lots of them. But I think what's interesting about the tone of the conversation was if you were speaking about this last year, that was the dominant thrust. That was what everyone was fixated on. That was where the regulations needed to be focused on. And I think we're sort of over time getting a little more back to sanity, where, of course, the government should be thinking about these issues, but that isn't a reason to sort of hold back the entire industry as a consequence of it. And I think since the UK launched theirs, the US, in great fashion, showed up to the safety summit in Bletchley Park and had our Secretary of Commerce announce their own on stage at the British event. But in any case, everyone is sort of competing to sort of create their own AI safety institutes. And now the big question is how are these going to function in a way that actually supports and promotes the AI industry rather than holds it back. And to me, I think the core thing that every one of these safety institutes need to be thinking about is something that remains unanswered, and that is a measurement science question. If you actually dig into how you red team these models, you know, it is not a particularly empirical process. You have some biotech experts partnered with some prompt engineers sitting together in a room interrogating a model for a couple hours and then writing a report and trying to get some reporter to write about how dangerous this particular model is. And like, that is not a repeatable, sustainable, empirically defensible way that we should be doing policy. So what I've always argued for, and I think that a lot of these safety institutes should be thinking about, is a measurement science behind these models. We need to understand how you actually can measure these models before you try to make policy around what these models can or can't do. And I think it was very telling when Dario from Anthropic went and testified to the Senate last year. He said this exact thing. You can read in his testimony that the actual measurement science around LLMs is unsettled. So if you think about back to what I said very early on, what should the federal government be spending on when it comes to AI research and development? Like, yes, all these private sector companies are spending billions of dollars on these LLMs, but one area where the government can play a role is invest in this early-stage, pre-competitive basic research around the measurement science of LLMs. And I think the entire community will benefit from better science there.

Yeah, absolutely. On kind of the allied free world side of the equation, you have the United States that stood up its AI Safety Institute. They're trying very hard to make it sort of the prominent lead in this space. Next month, there's going to be a summit held in San Francisco that our Commerce Department at the US is going to host, where all the other safety institutes are going to come, and these conversations around cooperation are going to start. So if you think about what are models where you can get sort of the allied free world to work together, you can think of it in some ways as kind of like — I like to think of it as sort of like airplane testing. So if you're the FAA at the United States and you certify safe a particular aircraft that's made by Airbus, for example, there's sort of reciprocity. So other countries around the world agree that that is safe. They don't need to retest the plane again for use by their country's company's airline. And I think the US AI Safety Institute is trying to get into that category of work where you can sort of create these agreements across multiple AI safety institutes around the world, so you don't have to pre-test the same model multiple times by different agencies. So I think that's good, and I think we need to sort of continue that. Now on the other side, on kind of the authoritarian side of the world, you have a CCP that is hell-bent on creating the dominant LLM in the world today, not only for use by its own people, but trying to export that around the world. And I think the terrifying and sort of like very disturbing thing about the approach that the Chinese have taken in developing their models is the regulatory regime around it, particularly around the fact that in order for a large language model to be released for public use in China, it has to actually pass a test that ensures that it conforms with Xi Jinping Thought. So they actually went and they created a test that then they use on these models to ensure that this model is actually communist and authoritarian, and then they want to take that model and they want to export it around the world. So the level of danger around this is something that we need to be very thoughtful of and not casually kind of move along with our lives and believe this isn't a new version of or new component of the digital Belt and Road Initiative.

Absolutely. I mean, the best parallel which I have is kind of the way that the CCP approached the export of Huawei and the export of advanced communication hardware around the world. So they're very smart in understanding that at the end of the day, most developing countries around the world or the Global South do not need the cutting-edge Ferrari piece of technology. They just need something that works and that meets the needs of their leaders. And they went out and they subsidized Huawei around the world. And then we were stuck in this really tough position when I was in government on how do we convince all these governments to essentially rip and replace this technology with safe, trusted Western hardware. And I think what we as the West need to think very carefully about is how do we ensure that this does not happen again in this new wave of AI. And why it's particularly of concern is that the actual ramifications of these Chinese LLMs being the base LLMs used across the Global South is actually pretty scary. So if you think about it, if you're a government anywhere in the world, there is a future in the very short term where you will be using large language models across all of your government services. You will have an LLM that helps people with their health care, their government health care. You will have ones that manage your property records. You will have ones that deal with your taxes and all of this stuff. The way it will typically work is you'll have a base foundation model that is then fine-tuned on top of the specific data of the country. And there is nothing that the Chinese want more than to ensure that their large language models are the ones that are used as the base models that are fine-tuned for all these government use cases. And once that's done, then you have these entrenched LLMs that for years are going to be used in these countries around the world. So I think it's incumbent on the United States, especially because we are the home for all the greatest chips and all the greatest LLMs in the world, to help in the export of these and make them accessible to all these countries that will be targets for the Chinese.