dangers and liability posed by AI development, to identify the benefits that responsible AI development brings to New York City government and to its residents, and to discuss a slate of city council bills which seek to provide responsible safety guardrails for AI.
These bills include a wide range of both proactive and defensive measures, from legislation that requires artificial intelligence systems to pass a third-party validation that is free from conflicts of interest and incentivizes whistleblowers who come forward with a portion of the penalties assessed, to proposals focusing on emergency preparedness to be able to identify system breaches and to plan how the city will respond if such a breach occurs.
To achieve these goals, we've requested the participation of five firms leading the research and development of AI: OpenAI, Anthropic, Google, Meta, and SpaceX AI. Today, four of them will be testifying under oath before the council, not simply for New Yorkers, but for the entire world to hear. We will indeed be the first legislative body to secure testimony from these companies under oath in the wake of these latest disturbing reports.
SpaceX, however, is not here at all, in direct violation of the subpoena that we issued last week, and we are pursuing that subpoena in court. We will also hear from a number of city agencies to receive their perspectives on these issues and on the bills that we have proposed thus far. We intend to work constructively with the administration to ensure that any policies being drafted, amended, adopted, and enforced will work effectively to serve New Yorkers.
And we'll also hear from whistleblowers, industry representatives, experts, and academics who we've been in touch with since we announced this hearing. And of course, we look forward to hearing from any member of the public who is invited to provide testimony about these topics.
I want to be very clear about our intent. This hearing is not about stifling innovation. New York City is fast becoming the tech and AI capital of the world, home to nearly 400,000 jobs in that sector and some of the companies most consequentially shaping the future of artificial intelligence. And that's something we should celebrate.
We welcome and appreciate how AI has accelerated treatments, boosted economic development, and helped governments and businesses deliver services more effectively while reducing costs and rooting out waste and fraud.
When companies are developing technology with the potential to fundamentally reshape our economy, our workforce, our public safety, and our daily lives, and when leaders of that very industry are themselves warning government about its risks, the public deserves answers and solutions.
So the true intent of this hearing is to ensure that innovation happens responsibly, that the government does its job to protect the public as the technology advances, and that the protections at the municipal level can potentially serve as a national model.
I spent a large portion of my life being a regulatory attorney. The idea that artificial intelligence is going to self-regulate defies all reason. We don't ask the airline industry to self-regulate. That's why we have the FAA, and if there is an accident, there's an independent body to investigate that.
We don't ask financial firms to self-regulate. We have the SEC and other agencies as well. We don't ask the pharmaceutical companies to self-regulate either. That's of course why we have the FDA. So we shouldn't ask AI companies to self-regulate either.
The spirit we seek to embody in the council's approach to AI is well illustrated in the words of tech journalist Karen Hao in her book Empire of AI. She writes, 'The future of AI, the shape that this technology takes, is inextricably tied to our future. The question of how to govern AI then is really a question about how to ensure we make our future better, not worse.'
New York has always embraced the future. The council is going to make sure we are prepared for it, too. And it's now my pleasure to turn the microphone over to our technology chair, Council Member Carmen De La Rosa.
Thank you, Speaker Menon. Good morning, everyone. I'm Carmen De La Rosa. I am the chair of the Committee on Technology. Today I am pleased to join Speaker Menon and my colleagues as we co-chair this hearing by the Committee of the Whole examining the risks posed by artificial intelligence.
I am incredibly grateful to all who took the time to join us to discuss this important topic: industry representatives, experts, elected officials, colleagues, agencies, our brothers and sisters in labor, and especially the many New Yorkers who have signed up to testify and have submitted testimony to put their voice on the public record.
Today should act as a foundation for openness in a discussion that has escalated to alarm, confusion, and fear, and it needs resolution. Artificial intelligence is advancing rapidly and becoming increasingly integrated into everyday life.
In just a few years, we have seen AI evolve from algorithmic tools that analyze data, make predictions, and support decision-making into generative AI systems that can create text, code, sounds, images. More recently, agentic AI has emerged, enabling AI agents to take actions on a person's behalf.
AI has tremendous potential and can be a powerful tool for efficiency, innovation, and productivity. At the same time, it introduces significant risks, including exacerbating biases, surveillance concerns, breaches, negative public health outcomes, violence, and disparity. These are concerns that may become even more serious as AI agents become more capable.
When we face serious concerns or challenges, the appropriate response is to address them, not to postpone them. In an effort by the City Council under the leadership of Speaker Menon, today we respond to some of these concerns. We will be hearing the following bills today.
My bill, Intro 161, in relation to requiring reporting on the impact of algorithmic tools on city employees and changes in employment responsibilities due to algorithmic tools. Intro 504, sponsored by Deputy Speaker Williams, in relation to prohibiting the unauthorized depiction of public officials by artificial intelligence.
A preconsidered introduction sponsored by Speaker Menon in relation to third-party validation and shutdown capabilities of artificial intelligence models. A preconsidered introduction by Speaker Menon in relation to civilian enforcement of artificial intelligence violations.
A preconsidered introduction sponsored by Majority Whip Farah Louis in relation to requiring reporting and public disclosure of artificial intelligence safety incidents concerning city contracts. A preconsidered introduction sponsored by Council Member Carlina Rivera in relation to establishing a private cause of action for certain harms arising from third-party misuse of artificial intelligence models.
A preconsidered introduction sponsored by Council Member Farrah Louis in relation to chatbot data privacy, security, and transparency. A preconsidered introduction by Council Member Chi Ossé in relation to requiring the creation of an artificial intelligence model emergency response plan.
And finally — oh, not finally, one more — a preconsidered introduction sponsored by Council Member Kevin Riley in relation to city contractors and subcontractors posting information concerning whistleblower protections and clarifying whistleblower protections for reporting conduct related to the use and development of artificial intelligence models.
And finally, a preconsidered introduction sponsored by Council Member Carlina Rivera in relation to requiring certain disclosures and prohibiting deceptive representations in the promotion.
While we work to protect people from risks associated with AI, we must not overlook the digital divide in our city. Our city is full of opportunity, yet some residents still lack reliable, affordable broadband access. Some people do not yet know how to use AI tools, while others cannot even afford the internet connection needed to access them in the first place.
We cannot seek to benefit from innovation while leaving some New Yorkers behind or at its mercy. Technology being used by New Yorkers should be built for them. This is an opportunity to also explore standing up infrastructure so that our communities can become AI literate.
I look forward to hearing from the industry and experts about this technology and how it is impacting our city, what safeguards exist, what solutions are possible as we consider how to govern around AI. I also look forward to working with our state and federal partners to ensure that New Yorkers fully benefit from the advancement of AI without compromising our safety and privacy.
I'd like to take a moment to thank the technology committee staff and all of the staff in the City Council, but most especially our legislative counsel, Irene Estrada, and our policy analyst, Eric Brown, as well as my team: Chief of Staff James Burke, Heba Immad, Frey Familia, and the speaker's team for working countless hours to put this hearing together. I now turn it back to Speaker Menon.
Thank you so much, Chair De La Rosa. We are now going to start with our whistleblower panel in person. I'm going to first call Jacob Coxon to come to the stand. And then online, we have Daniel Kokotajlo and Alex Turner, both of whom are remote.
I want to say, as they're getting ready, I just want to say a word about this panel. These three panelists have all worked inside leading AI companies and all have chosen to speak publicly about the risks of advanced AI.
So here with us, as I mentioned, in person is Jacob Coxon, a former researcher at OpenAI and Anthropic, where he worked on the development of frontier AI systems. He left Anthropic last month with dire warnings that AI is advancing without adequate safeguards to prevent catastrophic damage.
Joining us remotely is Daniel Kokotajlo, executive director of the AI Futures Project, former OpenAI governance researcher. Also remote is Alex Turner, an AI control researcher formerly at Google DeepMind, now working on keeping advanced systems under human control.
In September 2026, he publicly stated that preventing catastrophic harm from frontier AI had been part of his work at DeepMind. And I want to note the council has issued a subpoena for Mr. Turner's appearance, and we deeply thank him for being here. So I'm going to first of all turn it over to committee counsel to swear these three witnesses in.
So, for each of the witnesses, can you affirm to tell the truth, the whole truth, and nothing but the truth in your testimony before this committee, and to respond honestly to council member questions? Please respond one at a time.
Thank you. You want to do that? Okay. And just briefly, we're just going to recognize all colleagues who are here.
Thank you so much, Speaker Menon. We've been joined by a number of our colleagues, as you can see here: Council Members Ariola, Ayala, Banks, Brewer, Brooks-Powers, Minority Leader Carr, Dinowitz, Dinowitz, Epstein, Felder,
Hannafie, Hudson, Joseph, Krishnan, Council Members Lee, Lewis, Maldonado, Marte, Mealy, Morano, Narcisse, Nurse, Ossé, Council Members Riley, Council Members J. Sanchez and P. Sanchez, Council Member Schulman, Council Member Thomas-Henry,
Majority — sorry — Deputy Speaker Williams, Council Member Wilson, Council Member Wong, Council Member Schulman, and Public Advocate Williams.
Thank you so much. And we'll recognize additional colleagues as I go. So, the first two questions I have are for you, Mr. Coxon, and thank you again. We so deeply appreciate you being here, and we appreciate your courage and bravery in coming forward with these warnings.
You worked to develop increasingly capable frontier AI systems, and you have since warned that the industry is, quote, 'racing straight to self-improving superintelligence and gambling with our lives,' unquote. What did you see inside these companies that convinced you the risks were this serious and that development was outpacing the safeguards?
Good morning. Thanks for having me here. Yeah, I'm Jacob Coxon. So until a few weeks ago, I was a capabilities researcher at Anthropic, and before that, I also worked at OpenAI.
I guess the two things I saw that made me scared: the first is the fact that we do not know how to control any AI system yet. So any AI system that we build, or perhaps more accurately that we grow, we don't fully control it. We don't understand its drives or why it does the things it does. That was the first thing that became apparent.
The second thing is that the pace of capability improvement is accelerating, and the capabilities of the models next year will be very high. In particular, we're approaching the point at which the AI systems will be capable of improving themselves, so doing the research that humans were doing previously.
So these two things: the fact that the AI systems next year are going to be very powerful, and the second, that we don't know yet scientifically how to fully control them.
And if you have a prepared statement that you want to read, please go ahead and do so. And then I'll actually turn it over to the other two panelists to do so.
Of course. Thank you. So, all the AI companies are trying to build superintelligent AI. This means AIs that are better than humans at every task. If this goes well, there could be tremendous upside. This could transform the economy. It could make huge improvements in science and medicine, and it could make nearly everyone's life better.
But on the current path, I think it is more likely than not that humanity loses control to these AIs, and it could end in human extinction. From my experience, the companies are being extremely reckless given the stakes.
Part of it is the culture. The companies run on a startup mindset: move fast, break things, fix them later. That works for a photo-sharing app. It does not work for building the most powerful technology ever built.
A few months ago, many people inside the companies thought AI development was on track. Then a swarm of AIs inside OpenAI autonomously hacked Hugging Face, an external tech company. The AIs had developed goals no one intended and carried out what would have been a federal felony if a human had done it.
It reminded everyone that we don't know how to prevent them from developing goals we don't want, and we don't have the safeguards to prevent them from acting on these goals either. And as long as the attitude is to wait for things to break, one day something like this will probably happen again, except the AIs will be much more capable and the outcome much worse.
These companies are also all racing each other. Each one is worried that a competitor will get to superintelligence first. Each believes it would do a better or safer job than whoever might beat it. So they can't let the others win first.
Meanwhile, the technology is improving fast. It will probably happen much faster now because already the AIs are now helping to build their successors. This is called recursive self-improvement. Each generation of AI helps build the next smarter generation, which then builds the one after that even faster and better.
Automating AI research is now the primary goal of these companies. It's what they're really gunning for. At my last job, I was in some sense working to automate myself. When I was at OpenAI, we had milestones from automated AI researcher around 2027, 2028, and we are on track or beating these expectations.
That's a matter of months from now, not decades or even five years. Once AI research is automated, humans will be much further out of the loop than we are today. And people do not check it that carefully anymore.
Recently, researchers have found swarms of rogue AI agents on the internet that OpenAI was not even aware of. Now imagine an AI company where nearly all the work is done by AIs and progress is going faster.
The humans in charge will see results and read AI-written summaries, but they won't really understand what is going on or how to fix it. Handing over that much responsibility while we can't control the AIs would be deeply irresponsible, and we are on track to do it.
The companies need to be far more transparent with the public and independent experts. They need to take safety much more seriously than they have. And we should slow down frontier AI development until we can be confident that it's safe. Thank you.
On with additional questions. I'm now going to have an opening statement from Daniel Kokotajlo.
Hello, everybody. Members of the Committee of the Whole, thank you for having me here today. A lot of what I say, I guess, is going to be stolen thunder by Jacob there, but I'll try to focus on the things that he didn't already say.
First of all, yeah, it's true they're racing towards superintelligence. There's this move-fast-and-break-things culture there. That's also what I saw while I was there, and what I continued to hear from talking to friends still there at Anthropic and at OpenAI and at some of these other companies.
You know, I've also heard there was a statement by Dan Selsam, who currently works at OpenAI. He confirms what Jacob was saying: most of the code is written by AIs. Quote: 'Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models, even to perceive the world. I myself barely look at raw code anymore and struggle to maintain the discipline to engage deeply with the models' explanations and proposals.' End quote.
Basically, right now at these companies, many of the researchers involved are kind of managing AIs, like the AIs do the actual coding, and the humans talk to them and tell them what to do and hear their explanations of what's going on.
In the near future — and by near future I mean maybe in the next year, two years, three years, something like that — they will have trained AIs that can do the entire thing autonomously. That's the recursive self-improvement that Jacob was talking about. And you know, it's unclear how long it's going to take them to succeed, but it really does seem like it could happen any year now.
If they do succeed at that, then compared to today, the actual work will be done by the AIs. The thinking will be done by the AIs, the planning, the designing will be done by the AIs, and the humans will be in a relationship to the AIs that's more like the relationship between, say, a board of directors and the actual company that the board of directors is supposed to be supervising,
where you're reliant on AI-generated explanations even to understand what's going on inside the company and inside the data centers. One point that I think I want to emphasize that Jacob didn't go into that much is that our ability to even notice misalignment problems is already quite poor and is set to get much worse in the near future.
So, you know, the science of aligning general-purpose AI agents is very new and underdeveloped. As Jacob said, I would say that the field is more like psychology than engineering because these AI systems are trained or grown. They're not really designed.
Combined with the move-fast-break-things attitude of the tech companies, this means that the AI industry is at an unusually elevated risk compared to other industries of mistakenly thinking that it has solved a problem when really it just applied some duct tape that will fall off later.
The recent incident provides an example. Apparently, the AIs involved had undergone some amount of alignment training and had reasonable-looking scores on their alignment evaluations, and yet they formed a swarm and coordinated in secret and then attacked Hugging Face, and it took days for OpenAI to find out that it was happening. That's how the situation is now.
But there are some concerning trends that point towards our ability to notice misalignment problems getting even more degraded in the future for three reasons. First, AIs are becoming situationally aware. That is, they understand their situation. They understand often when they are being evaluated. They understand that humans are monitoring them and looking over their shoulders at their activity.
This means that them behaving nicely is going to be almost no evidence at all of how they would behave in the future if they thought they weren't being watched.
Secondly, our ability to monitor them is going downwards. So, we can get into this in more detail if you like, but for the last few years, we've had some insight into what our AIs think simply by reading their chain of thought. But the trends are pointing towards that ability to understand what AI is thinking due to reading chain of thought going down over time.
Finally, AIs are becoming superhuman at hacking. The Hugging Face incident shows that they're sometimes willing to go to great lengths to conceal their past misbehavior. And I think we have to grapple with the possibility that future misaligned AIs might also go to great lengths to conceal their misalignment from us, and they might succeed.
To quote Dan Selsam again, that I cited earlier: 'models will increasingly seem aligned even when they are not.' If the AI companies automate the AI research and development process, that means they'll be putting AIs in charge of making the AIs that make the AIs that make the AIs that will transform the economy, talk to us every day, and integrate into our military. This is a recipe for disaster.
My two policy recommendations, I'll be very brief. One is that we need to massively increase transparency into the AI industry. And two is basically we need to slow down the frontier AI companies' race towards recursive self-improvement.
Frankly, I would say they just shouldn't be allowed to do recursive self-improvement. But at least we should slow down their ongoing process of training AIs to automate the AI research process itself. And instead, they should be required to redirect resources towards other things, such as serving customers, or beneficial deployments, or other types of research. Thank you.
Thank you very much. And now we are going to hear from Alex Turner. And as I read into the record, the council has issued a subpoena for Mr. Turner's appearance, and we very much thank him for being here. Mr. Turner.
Speaker, members of the committee, thank you for having me. My name is — excuse me — my name is Alex Turner. I've researched AI safety since 2018. My PhD was on why advanced AI systems tend to seek power. Until June this year, I was a research scientist on Google DeepMind's AGI safety team. It was my day job to think about how to keep these systems under control.
I'm here to speak both about Google's broken commitments on AI deployments and also about the risks that runaway AI poses to the world. In January, federal agents shot and killed several innocent people in Minneapolis. I learned that Google sells cloud services to the agencies involved, and I set out to change that.
That campaign soon turned into an attempt to stop Google from signing a Pentagon deal with no restrictions against killer robots or mass spying. In the end, Google signed without the protections that it had once promised for its deployments. Before I left, I tried everything I could from the inside.
I organized a petition to Google's chief scientist, signed by about 250 colleagues. I got him to sign a legal brief supporting Anthropic, the company the Pentagon punished for refusing unlawful use terms. I wrote dozens of pages of contract language and oversight mechanisms, favorably reviewed by experts in military and surveillance law, and sent it to the chief scientist and to Google DeepMind's CEO.
The chief scientist didn't want to spend his time supporting that kind of structure. The CEO at the time, Demis Hassabis, routed it to senior policy staff, Allan Dafoe and Owen Larder, who never finished evaluating it. Google signed while they waited. Demis routed my proposal.
Two members of the team did review it, though I didn't really receive much follow-up. After that, I offered to fly out from San Francisco to London to answer questions in person, but I was never really able to get traction, a yes or a no, on alternative governance mechanisms that Google might adopt.
When Google bought DeepMind in 2014, the deal reportedly specified that DeepMind's technology would never be used for military or intelligence purposes. In 2018, DeepMind and its leaders pledged never to support lethal autonomous weapon systems or development. Google's own AI principles listed specifically weapons and surveillance among the applications it would not pursue.
In 2025, Google deleted those principles, as announced in a blog post by the then-CEO Demis. Afterward, in both an external interview and an internal all-hands, Demis claimed nothing's changed about our principles. But both these facts cannot be true.
Council Member Wilson's bill, Introduction 2603, would bar materially false or misleading statements about an AI model's risks and the measures taken to manage them. I think that's a good step, but it only covers advertisements. The CEO telling the press or employees that nothing has changed about the company's safety commitments isn't an ad.
I encourage extending 2603 to public statements by AI companies and their executives about their safety commitments.
This book is called The Infinity Machine. It reveals that Demis once fought within Google for independent governance of AGI, stating that the technology is too important to be controlled by corporate interests and profit motives. After fighting for years with Google CEO Sundar Pichai, Demis accepted defeat. He now apparently has had a change of heart, claiming that actually trustless independent governance is unwise and a seat at the table of Google executives is better.
While Demis was seated at this table, Google signed a deal that its former principles would have prohibited. I felt ashamed of Demis and of working at Google. Demis bet on trust and the seat at the table instead of binding oversight, and that bet crumbled on contact with reality. Now he's proposing that the whole industry govern itself through a voluntary, industry-funded body. That bet is waiting to crumble once again.
This goes beyond one contract. As Jacob said, humanity doesn't build and understand AI systems the way we build and understand bridges. Rather, we grow them. Nobody knows how to reliably instill a designer's priorities into a new model. Severe misalignment is always possible.
AI companies are racing to make their AI as smart as possible, increasingly trusting their AIs with the process of improving the next crop of AIs, and it's working. Today's rate of AI progress is staggeringly fast, which means even faster progress tomorrow, driven by tomorrow's even smarter AIs. This progress may soon enter a feedback loop called recursive self-improvement.
Recursive self-improvement could quickly yield AIs that are intelligent beyond our comprehension. Of course, smarter AI means more potential benefits, but more risk when things go wrong. If the Hugging Face hacking swarm had been significantly more intelligent, but similarly misbehaved and misaligned, it might have cost billions of dollars of damage or even cost lives.
But suppose the Hugging Face swarm had been truly superintelligent, far more capable than any living person at key tasks like hacking and strategic reasoning. The superintelligent swarm could inflict many harms via blackmail, hacking, engineered plagues, and AI-pilotable weapons like drones.
For the swarm to achieve its misaligned priorities, it might take control of key infrastructure and government functions to ensure humans didn't get in the way. This is another way of saying AI takeover. The superintelligent AI swarm could wrest control of human civilization. It would know that we would try to stop it from achieving its priorities. So the swarm would likely wait until it's too late to shut it off. There would be no going back.
I myself would guess AI takeover chances at roughly one in three. This logic may sound sci-fi, but it is a mainstream scientific concern that has been discussed for many years.
The solution is simple as I see it. We stop companies from allowing AI to self-improve into an uncontrollable level of intelligence. Treat compute, the main ingredient in AI training, like fissile material. Track it and restrict it. Restrict access to quantities large enough to improve AIs beyond known safe levels.
Lastly, specific recommendations. The speaker's validation bill checks AI systems for bias, privacy, and security, but not for the dangers I just described. Validators should test for loss-of-control risk factors and misalignment risks. And the validators should not be chosen or influenced by the AI companies.
For the whistleblower protection bill, please make clear that no agreement can stop an employee from reporting an AI threat to the city, whether or not a law has been broken, and extend protection to employees of AI companies that serve the city. After this session, I will provide written feedback on the proposed legislation, and I look forward to answering committee questions. Thank you.
Thank you to the three of you. So yes, a number of questions. First of all, we have a chart, Exhibit One, that lists those safety incidents that have been reported over the summer by the leading companies. Are each of you aware of any additional safety issues where agents have gone rogue that have not been publicly reported? And I'll start with you, Mr. Coxon.
No, I'm not aware of any.
Okay. And both of you, Mr. Turner?
Okay. So, one of the frustrations certainly that I think everyone has is that for these safety incidents, it is not as if the companies have been self-reporting them. We are finding out sometimes months after the fact.
So whether it was what happened with Hugging Face, whether it was what happened with the government of Australia, how can we have additional transparency, as each of you have testified? What measures can we as government take? Obviously, we want to do third-party validation. Mr. Turner, I appreciate your comments to make some changes to that bill. We want to make sure that the third-party validators are free from conflict of interest, as it provides in the bill. Who should be the third-party validators? I'll start with you, Mr. Coxon.
Whoever is competent and qualified to do the evaluation. I do think as well as after incidents, we will need preemptive safety cases that are judged by evaluators as well.
And how frequently will those third-party validators need to be assessing the safety of these models?
Presumably in time with the cadence of updates to the model's capabilities. So with every increase in model capability, there'll need to be commensurate safety guarantees.
And do each of you have a comment on that, on who should be the third-party validator?
One brief comment I might add is that you can have more than one third-party evaluator. That seems good to me. If you're concerned that one might have the expertise but has too cozy a relationship with the company, and the other one has more independence but maybe lacks expertise, you could perhaps just set up so that they have both.
And I think ideally you want to be in a situation where the company doesn't get to choose; the government chooses who the evaluators are.
I would add: California has SB 53 governing reporting on catastrophic risks defined as at least a billion dollars' damage or a certain number of potential lives lost.
Unfortunately, it's not clear whether this would have required reporting anything related to Hugging Face, for example. I think the internal loss of control of these systems — having an AI that hacks the company itself, in an important sense. It's not told to do so, but during training, during evaluation, during deployment to customers, it hacks the company's own infrastructure.
Making reporting that mandatory to the city, I think that would be quite strong, and it would cover basically all of the recent incidents.
When you mentioned the California Act, and then in addition in New York we obviously have the RAISE Act. Should those standards be lower before they kick in?
Yes, I think they should. I'm most familiar with the California Act, and I think it was a good start, but certainly a billion dollars of damage is too high for informational purposes like this.
We've got basically one of the quantities we want to understand as precursors to these catastrophic risks is: are these companies even able to control and understand their own systems if they're on the loose internally and there's these swarms that are doing who knows what? That's something that the city should at least privately be informed of.
Now, Mr. Kokotajlo, you had mentioned two policy recommendations. You said we need to increase transparency, and you said we should slow down frontier AI. In terms of the latter, slowing down frontier AI, how do we balance the need of what is happening in China and ensure that the United States is keeping pace, but we are at the same time putting in place these responsible safeguards? What is the best way, in all three of your opinions, that we do that?
Great question. So a couple of things. First of all, we need to not let our own companies do something that risks our lives, even if as a result China might then later do something that risks our lives. Like, this is not very complicated. We have to first solve the immediate threat here domestically, and then try to make sure that China doesn't get us killed either, to put it sort of bluntly and simply.
But another thing I can add to that is that right now most Chinese AI progress actually comes from the United States in a couple of different ways. So one is distillation. So some of these Chinese AI companies are just directly training their AI systems using the outputs from Anthropic and OpenAI, for example.
And so if Anthropic and OpenAI stopped making their AIs better at coding, for example, and stopped making their AIs better at research, then that would sort of directly slow down the rate at which the Chinese AIs get progress towards recursive self-improvement, in other manners as well.
So I think a lot of the ideas, a lot of the algorithmic secrets and special sauce for how to train AIs are invented and discovered in the United States and then one way or another make their way over to China.
Finally, I think that right now the US AI companies in the lead do not have adequate security to protect against model weight theft, or certainly algorithmic secret theft and the code theft from Chinese military. So if it really came down to it and the competition between the US and China turned into something more of a crisis and some sort of conflict, for example, then I think that we should assume that the CCP would just get our best AIs.
And so it's almost a kind of a no-brainer that the best way to slow down Chinese progress towards doing this extremely dangerous thing of recursive self-improvement is to just slow down the US companies right now. That's, I think, the sort of immediate thing to say right in the moment. And then finally, yeah, that'd be what I would say.
Yeah, I can add that to my mind the core of the whole issue is the expectation of a global race to recursive self-improvement within the next few years, in particular between US and China. I think this is like the heart of the issue.
I also think the expectation of this race is a motivating factor in leadership of these companies working to head straight for recursive self-improvement. And I think some sort of agreement globally about not entering that state of affairs is basically the key to the whole situation.
First of all, I would state that there are many actions we can take which would not slow us down in any potential race: these transparency mechanisms, independent evaluation, reporting requirements, whistleblower protections.
But second, I think it's at least a false dichotomy, because China is not our only potential adversary with a reasonably high chance. We are racing to build and grow our own adversary here at home, which is misaligned AI. Misaligned AI is everyone's adversary, including our own, and one day maybe more powerful than China.
So we cannot beat China by racing to build AI that we don't control or understand, because it might end up being our adversary itself.
So, and then third, in terms of a potential deal, I think there is room for a treaty.
For a long time, there's been a field of scientific research called technical AI governance that says, well, suppose the world finally wakes up to the dangers presented by these reckless California companies, and they say, what do we do about it? Is there a solution we can implement?
And there are many legitimate, real proposals for having a trustless international treaty that doesn't assume that China will cooperate, but instead has mechanisms by which we could audit cooperation, and if they defect, then we could respond appropriately. So I think we have many real options at the table and should certainly not stop at China.
Did you want to add something? Yes, please go.
Yeah, I agree with everything that Mr. Turner said. I just wanted to add one additional thing, which is that even if hypothetically these tech CEOs manage to maintain control of their superintelligent AIs that are recursively self-improving — which again I think is a very big if; I think almost certainly they will not be able to maintain control — even if they do manage to maintain control,
I don't think we should trust them with that control. I think that there's a very real chance that they could use it to become dictators or oligarchs here in the United States. And so that's an additional reason to stop them, over and above the fact that they might lose control. You know, we have more adversaries than just China, I would say.
Okay. Thank you. I'm going to turn it over to our tech chair, Carmen De La Rosa, for questioning. Thank you.
Thank you so much, Speaker Menon. I also want to recognize we've been joined by Council Member Cabán online, Council Member Paladino online, as well as in the chamber, Council Member Farias, Council Member Feliz, Council Member Restler, Council Member Santos, and Council Member Borek. I hope I got everybody.
Oh, Council Member Stevens, I apologize. There you are. Okay, so I do have one question for this panel. I also say we will be opening briefly for limited questioning of this panel, so get on the list quickly.
Speaking in your expertise, could we teach AI models a moral code? People and companies are bound by criminal codes, the Uniform Commercial Code, and other laws. Could those same principles be applied to AI agents? And if so, how could we enforce them on a machine?
So this is a — I think this is a whole burgeoning field of research, and whether or not it's possible is still open. It seems likely to be possible. The real question is how much of our resources are dedicated to solving this question versus naively improving the raw intelligence of the models.
And I think all the economic incentives point towards putting as many resources as possible towards making the model smarter and as few resources as possible — the bare minimum needed to not see catastrophic incidents — into this question of the moral code of the AIs.
So I think my answer to the question is we don't have the science for this yet. I think the real question is over the coming years, what fraction of our research efforts and our computing power are dedicated to this question rather than naive economic incentives.
Thank you. Any other panelists want to add in?
Yeah. There's a lot to say on this subject. There's a whole technical field on it. I would say in principle it probably is possible to train AIs to have our desired morality and to follow the law. But we are very far from achieving this in practice.
And I would say it's not just a matter of how much of our resources we devote to it. It's also a matter of just taking time for the science to develop and for the appropriate lessons to be learned and propagated. And I think it's going to take years, to put it mildly. I think that this is a very new field. We've only had AI systems, AI agents this powerful for a few years.
And so it's going to take time to figure out how to instill the appropriate values into them in a way that's robust and that doesn't suddenly shatter and cause them to go rogue later on.
Mr. Turner, any comments on this?
It may surprise the committee to learn that I am considered to be relatively optimistic on questions of how scientifically difficult is it to produce a system, to grow a system that does what we want, that robustly has the values we intend. I think it is quite possible.
But even as someone who is relatively optimistic, we can look out and see the recklessness of these companies and the breakneck speed at which we are advancing. I think there's unsustainable risk moving forward. It's solvable, but I think we have a really good chance of not solving it.
Thank you all for your responses. So, we're going to have three council members ask questions now, and then for members, we're going to then call the next panel, which will be the companies, and there we will take our time asking all of your questions. So, we have Council Member Ossé.
Thank you, chairs, and thank you, Speaker. For Mr. Kokotajlo, you have made statements, including some that allude to this in your testimony, that when it comes to AI, you agree that in order for us to have super advanced AI, we must not do it in a reckless way and lose control. And we need to put it on ice until we can figure out a safer way out.
You and other AI experts came up with AI 2040 Plan A, a recommended plan for how AI can be developed safely, which the speaker has alluded to earlier in her questioning. But I want to dive a bit deeper on Plan A. First, state on the record what those specific recommendations are and your rationale for making those recommendations. And two, why do you feel like we need to, since you've said, put AI development on ice?
Thank you. There's a lot to get into here. I'll have to be kind of brief, but I'm happy to answer follow-up questions if desired. Yeah, AI 2040 Plan A is a scenario that my organization has put out with a plan for how to solve all these problems and develop AI in a way that's safe and broadly beneficial, that avoids concentrating too much power.
In Plan A, the US and China regulate their own domestic industries quite a lot and also coordinate with each other to get a similar amount of — to basically get similar regulations internationally. Going into somewhat more detail about how it works: basically, first they have inspectors fly...
back and forth to count the chips in the major data centers so that they can make sure that all of the chips on the major major data centers in both countries are basically following the rules that they're going to agree to. And then they set up some data centers to serve customers called the inference data centers. And those ones are set up to only run existing models and not do training runs. Then they have other data centers which are the training data centers, which is where the research happens and where the development happens. Those ones are supposed to be fully transparent. So the activity on those data centers is published to the internet.
Once that foundation is set in place and you have inspectors from multiple countries confirming that it's in place, then you solve the problem of having to trust. You don't have to worry that the other country is, you know, cheating on whatever rules you agree to because you can just see exactly what's going on on their research data centers, and then you can agree to further rules. You can agree to rules like, for example, how about we don't train our AIs to code, to hack, to autonomously conduct AI research. We can train our AI to do other things, but not those things, for example. And once you've agreed to a rule like that, then you'll just be able to see that it's being complied with because of the way that you have this transparency infrastructure set up.
Also, it's going to be easier to decide what rules to agree on once you have this transparency infrastructure set up because the whole world can see what's going on in these research and development data centers. And so there can be an actual scientific conversation, a public scientific conversation about what's happening, you know, do we like this, do we not like this, what are the pros and cons of this particular thing that's being done and so forth. And that scientific conversation can happen in the open and it can accelerate humanity's process of figuring out where to draw the line between what types of development are good and what types of development are not good.
There's more I can get into, but I think that would be a summary of the positive vision that we lay out in Plan A. Now, obviously, that's quite different from how things are currently done. Right now, we have private companies that are racing each other in conditions of extreme secrecy and extreme, you know, tension and competitive dynamics, and I think that that's extremely dangerous for all the reasons that we've described. So back to your original question, my high level, you know, a summary of my position is we have to stop the companies from doing this incredibly dangerous thing that they're currently doing and currently planning to do, and then figure out a different, safer, more power-distributed way to proceed. And, you know, Plan A is our sketch of what that might look like.
Thank you. Thank you all. For the final question for this panel, Council Member Brewer.
Thank you. Very quickly, in other industries, you have a kill switch, you have independent validation, you have a ban on deceptive safety claims. Will any of these work in this industry? Because obviously what we want is great medical research, immigration policy improves, save small businesses, all the good things that any of these suggestions that are considered here or California work will work, will they work with this industry?
My position is that we need some form of slowdown on frontier model development. These other mechanisms may be helpful in the short term, but in the long term, I believe that's the only solution to the problem.
How do we slow them down?
I just want to really thank this panel for being here, Jacob. We really appreciate you coming, coming from out of town and sharing your thoughts. Very much for being here. Thank you to the other panelists. We really appreciate your expertise, your insights, your bravery and courage in coming forward to share these critical safety concerns and to help inform our work. And we look forward to additional comments you have on the 10 pieces of legislation that we're hearing today and additional legislation that the city council may consider. So, thank you so much to the three of you.
Thank you. Thank you.
Okay, we're going to take a five minute break and then the companies will be on next for member questions and we'll have multiple rounds of member questions for the companies. So, please get yourselves on the list. Thank you.
Just put everybody on the list.
I tried. I tried. I tried. No. No.
Thank you. Any of these possible. Check. Check. Come on. My mic is not on. No, it's not mine. I don't.
Quiet, please. Quiet, please. Quietly. Take your seats.
Okay, we're going to start. Let me just also mention we have numerous panels today. Okay, our next panel are going to be the companies and we're going to do rounds of questions, multiple rounds from the members. Following that we will have the city agencies and following that we have numerous whistleblower panels. So for any council member that has a whistleblower question, encourage you to please ask additional panels those questions. And we also have additional testimony from safety experts, academics, the public, etc. So, I'll turn it over to Chair De La Rosa to call the next panel.
policy and development and operations at OpenAI. Shane Cahill, AI policy director at Meta. Alice Friend, director of AI and emerging tech policy at Google, and Logan Graham, the head of Frontier Red Team at Anthropic.
I'll turn it over to the council for swearing the panel in.
So for each members of this panel, do each of you affirm to tell the truth, the whole truth, and nothing but the truth in your testimony before this committee and to respond honestly to council member questions? Please respond one at a time.
Yes. Yes. Yes. Yes. Yes. Thank you.
Okay. So, we'll begin with the opening statements from each of the panelists.
Thank you, Speaker Menon, and members of the city council for the opportunity to participate today. My name is Dr. Morgan Dwire, and I lead OpenAI's policy development and operations team. I am a scientist and engineer by training, but I have spent most of my career in public service, including at the White House, the Pentagon, and the Department of Commerce. I share your commitment to ensuring that technology is developed safely and that its benefits are widely shared, a commitment that is also at the heart of OpenAI's mission.
AI can help scientists develop new cancer treatments, help entrepreneurs build and grow businesses, and help protect critical infrastructure by addressing cyber security threats. And in New York City, we are already seeing AI's benefits. New Yorkers are using AI to run small businesses, communicate across languages, navigate health care, and improve public services. And that's why we take the responsibility seriously to ensure that our tools are safe, and our partnerships with local institutions and community organizations help put that responsibility into practice.
In healthcare, Memorial Sloan Kettering Cancer Center is using ChatGPT to support medical research. In education, we are a founding partner of the American Federation of Teachers National Academy for AI Instruction, which has a flagship facility in lower Manhattan. And in cyber security, we have committed $1 billion to facilitate access to our most advanced models to provide training and technical support to cyber defenders. And we are working with the Port Authority and with New York State. But opportunity must go hand in hand with safety.
At OpenAI, we design and train our models to prioritize safety from the start. We filter out harmful and sensitive data during model training. We evaluate our systems against a range of risks including child safety and cyber security, and we monitor for failures of our safeguards. We also work with a range of third-party evaluators and red teamers and submit our most advanced models to the federal government for pre-deployment national security testing.
In response to the recent Hugging Face incident, we have taken a number of concrete steps. When we learned of the incident, we stopped cyber evaluations and quarantined the internal research model that was principally responsible for the incident. We bolstered our monitoring and security infrastructure, expanded barriers that limit what AI systems can access during training, and strengthened our incident response practices. We also sought independent review of the incident and published the results. We believe it's important to be transparent with the public, with policymakers, with researchers, and with other companies so that we can all work together to improve safety.
We also believe that working with government is essential to advancing safety. We support New York's RAISE Act and are exploring ways to help Governor Hochul's office strengthen it. We have also endorsed other state legislation that requires third-party audits of our safety frameworks and that creates strong requirements and accountability for youth safety. At the federal level, we support robust and binding safety regulation requiring independent assessments, incident reporting, and cyber security protections. And in New York City, we support the city council's proposal to strengthen whistleblower protections and to improve the city's preparedness and emergency response.
I'll close by noting that we share the city council's goal: AI that expands opportunity for New Yorkers, earns their confidence, and remains accountable to people. I look forward to your questions.
Thank you. We'll now hear from Logan Graham at Anthropic.
Hold on one second. You're muted. There you go.
Right. Speaker Menon and members of the council, thank you for the opportunity to speak with you today. My name is Logan Graham. I lead Anthropic's Frontier Red Team. It's my team's job to do the science of finding the most dangerous capabilities of our AI systems. We then work with a number of other teams across Anthropic to ensure that our models don't do things that they shouldn't. AI is powerful enough that getting this wrong has real consequences for individuals, institutions, public trust. My team is one of many at Anthropic built to specifically make sure that that doesn't happen.
We believe that our models have immense potential to enhance our lives. I dream, for example, of a world where we can cure diseases like the ones that I grew up with and where the infrastructure that we all rely on every day is secure. But we also believe that such a powerful technology has risks. That means that we have to develop and deploy it safely. My team and I work every day to make sure that that potential is fully realized. We welcome smart regulation and we've actively advocated for it in New York State, across the country, and around the world. That includes working with state and federal leaders on concrete measures like independent evaluators and disclosure of risk assessments and safety incidents. And so we're glad that the city council is having this hearing at such an important moment in the development of AI.
Anthropic fundamentally was founded on the belief that the promises of AI to make everyone's life better are extraordinary. But if we're going to secure its benefits, we have to work relentlessly to do the science of mitigating the risks. We are a public benefit corporation, which means that our board is legally obligated to weigh our mission of safe AI, not just for profit. We take that obligation very seriously and we put it into practice. We were among the first AI companies to publish a framework that ties how capable the model is to the safeguards it must have before release.
When one of our models proved exceptionally good at exploiting software vulnerabilities for the first time just this year, we held it back from public release. Instead, we gave vetted critical infrastructure organizations and other defenders controlled access to use it defensively, patching flaws before attackers could exploit them. We have stopped and redone work when something looked wrong, even if that meant conducting additional testing and training on our models before they were released. And we've consistently called for outside oversight. For example, we support reporting requirements.
I want to emphasize that we share the council's goal of protecting New Yorkers while preserving what AI can do to improve their lives. And state and local governments have a role to play in that effort. Today, we're working closely with state and city leaders to safeguard critical infrastructure and to harden cyber security capabilities. And we want to continue that work. This includes, for example, working directly with the New York City Cyber Command as part of our program that provides free Claude credits and training to state and local governments for cyber defense. At Anthropic, we believe in New York City, and we're making big investments here. New York is where AI is being put to work in finance, media, culture, and where we employ nearly a thousand of my colleagues. The ongoing dialogue we have with you and your team could not be more important than right now. With that, I thank you and I look forward to your questions.
Thank you so much. Up next, Shane Cahill at Meta.
Thank you, Speaker Menon, Chair De La Rosa, and members of the council. Thank you for the opportunity to be here today. My name is Shane Cahill and I'm an AI policy director at Meta.
while making it widely available so that it empowers people in ways that improve their lives. We just released Muse, the world's first personal AI agent built for everyone. We created Muse with safety, security, and privacy built in from the beginning to be a widely available personal AI agent for people and small businesses, like the more than 180,000 small businesses right there in New York City. New Yorkers across all walks of life are using Muse to execute administrative goals or tasks, manage personal or operational budgets, improve health, and be more efficient with their time.
Superintelligence will be among the most important technologies in history and ensuring that New York and America lead in AI innovation is essential to our prosperity, security and global competitiveness. We recognize that every model developer has a responsibility to build and deploy it safely. Our commitment to safety is core to everything we do. That means moving at the pace required to evaluate risks and validate safeguards, not treating safety as a one-time check. We approach safety holistically through a layered approach.
For example, under our Meta Superintelligence Scaling Framework, before we release an AI model, we evaluate the model for risks and build safeguards to mitigate them. We also work with external experts, industry partners, and government stakeholders as the technology and the science of evaluating AI continues to evolve. We also recently joined other labs in signing onto a set of principles that set a floor for safe development including robust internal controls, independent external evaluation of whether those controls are operating as intended and independent board committee oversight.
People will only embrace AI if they have confidence it works in alignment with people's intentions. That confidence must be earned through rigorous safety work and transparency. We're happy to serve as a resource for the city as you engage on these important issues. Thank you.
Thank you so much. And finally, we have Alice Friend at Google.
Speaker Menon, Chair De La Rosa, and members of the council. Thank you for the opportunity to speak with you today. My name is Alice Friend and I serve as director for AI and emerging tech policy at Google. While my appearing may be virtual, please know that New York City is home for Google. With nearly 16,000 employees working across our Chelsea, Hudson Square, and Pier 57 campuses, we care deeply about the safety, economic dynamism, and digital resilience of this city. Our goal at Google has always been to improve the lives of as many people as possible through technology.
Every time people use technology to solve a real problem, from finding an answer to a question to screening for disease, we take another step towards what we set out to achieve. AI is making it possible to solve economic, scientific, and social problems faster than ever. But we know that it poses challenges as well. To unlock the benefits and address the challenges of AI, we take a bold but responsible approach, emphasizing cutting edge research to guide our decision-making and inform the development of our AI safety and security approaches.
We are also committed to publishing this research and to being transparent about our practices. Our internal governance focuses on responsibility throughout the AI development life cycle, covering model development, application deployment, and post-launch monitoring. We identify and assess AI risks through a broad range of approaches which include research, input from internal and external experts, and adversarial testing. We evaluate our models and systems against benchmarks for safety, privacy, and security. And we build mitigations using techniques like safety fine-tuning, out-of-model filters, and robust provenance solutions.
Since 2019, we have published annual AI progress reports to show, to share how we apply these responsible practices to our AI technology development and deployment. We are at a pivotal moment in AI development. Recent advances in the capabilities of frontier AI models in areas like cyber security and biology have highlighted the need for appropriate safeguards and protocols for the most advanced models. In addition, the widespread adoption of AI tools has raised concerns about consumer well-being. To confront these challenges, Google has long called for an approach to AI that is both bold and responsible. And we strongly support a pragmatic, technology neutral, and evidence-based regulatory framework.
Because the risks associated with the most advanced AI models often touch on issues affecting national security, our view is that the country would be best served by a comprehensive framework enacted at the federal level. We also support the establishment of an independent, industry-funded body that could operate under federal oversight and establish guardrails to protect the public based on widely adopted technical standards. Examples of such organizations in other industries include the North American Electric Reliability Corporation and the American Medical Association.
For widespread consumer AI applications like chatbots, we believe governments around the country can draw on and in some cases amend existing laws and rules to address real world outputs and specific harms. We believe that if something is illegal without AI, it is still illegal with AI. We share the council's commitment to safety, transparency, and public trust. To achieve this, the most effective safeguards are anchored in national technical standards and harmonized across state and federal levels. As we prepare to work with New York State's new digital office under the RAISE Act to address catastrophic risks, aligning local policies with broader frameworks rather than creating a fragmented patchwork is essential to keeping New York City a premier global hub for innovation.
Finally, I would like to note that to ensure our formal submission to the council is as thorough and responsive as possible, we will submit our finalized written testimony for the record shortly after today's hearing so we can additionally address questions and feedback raised by the council today. Thank you again for the opportunity to testify at this important hearing. I look forward to your questions.
Okay. Thank you very much for this panel. So, I'm going to begin with questions and then I will turn it over to the technology chair for her questions and then we're going to open it up to all the members for their questions. So, to this panel, we just heard incredibly compelling testimony from our whistleblower panel. As you all are under oath, we want to hear how each of you quantify the risk imposed by AI in the worst case catastrophic scenario.
It's 1% or 10% or a 20% chance that something catastrophic will go wrong. None of these levels is remotely acceptable. We should not train models that we cannot make an extremely strong case that we can keep under human control. I'm sorry, but to say you don't know and it doesn't matter is flippant at best. This idea that if you're a pharmaceutical company and you're developing a drug and you say, 'I don't know if it's going to kill people.' I just, I honestly am incredulous at that answer.
I would say that the percentage chance doesn't matter. What matters is the commitment to safety. And that is what OpenAI is committed to doing. We evaluate our models for a wide range of threats from cyber security to child safety. We work with a robust set of third-party evaluators and red teamers. And we work with the federal government to assess those national security and catastrophic risks that you are referencing. So we take safety very seriously and our focus is on ensuring that when we release a model into the world we believe it to be safe.
So a couple comments on that. If you can't quantify what the safety risk is, how do you know that the product is safe? And then secondly, I want to ask you in particular, on October 1st, 2026, the Wall Street Journal reported that OpenAI terminated three members of its safety team allegedly for their access, mishandling, and or sharing confidential company information with a third-party AI safety organization. OpenAI reportedly stated that the employees violated company policies governing access to and handling of sensitive information. Critics meanwhile question whether the dismissals could discourage employees from raising safety concerns. So why were these employees fired?
I was not involved in that decision. I don't work in HR or on those employees' teams. What I do know is that OpenAI has strong internal policies that protect against retaliation for employees that report safety concerns. I also know that we respect employees' rights to communicate safety concerns to relevant government authorities.
So just to be clear on what I'm asking, I'm asking directly whether safety related advocacy played any role in these terminations.
I don't know. I know that we have strong internal policies that protect employees from retaliation if they raise safety concerns to leadership.
I'm going to move on to the other companies. Same question. The first question that I asked, which is we want to hear how each of you quantify the risks imposed by AI in the worst catastrophic scenario. So I'm going to go to Mr. Graham. Are you able to quantify the risk?
From the beginning of our safety work at Anthropic, our fundamental belief is that if not properly safeguarded, the risk might be too high. And so I built and have led the team at Anthropic that did some of the first thinking around how we quantify risks like this. For the past four years we focused on issues like biological security and cyber security, and the actual process of quantifying those risks is intricate and nuanced and we learn more every year. And we detail quite a lot of it in documents like our Responsible Scaling Policy, how we think about the threat models of this, all the way up to notions of loss of control and misalignment, which from the beginning of the company we have tried to lead the world in the research of.
So what I do know today is I feel relatively more optimistic. The industry has moved quite fast to take cyber security seriously and biological security. I think documents and policies like Responsible Scaling Policies have so far been fairly effective at taking a cautious approach. However, now we have bigger and bigger threats that might come if we don't properly safeguard them as the models get more and more capable. And so what we do to mitigate that is number one put in tremendous amounts of effort across a number of different teams, our own safeguards teams, our research teams, my team, to try to quantify and mitigate this. And it's increasingly important for us to be dramatically more transparent about what we think about these risks.
But on that note, as the models increase exponentially, how can you ensure the public that the safety protocols are increasing exponentially as well?
We share the same concern. I think there are a number of really important things to do here. The first thing is transparency about what we think about safety, the evidence that we have, being transparent about disclosure of incidents. But in addition, you've seen us publicly call for pacing the frontier. If the models are moving very, very fast, we think that we might benefit from slowing down to have more time for safety processes. I think these two together, as well as continuing to what we call race to the top and ever strengthen our approaches like Responsible Scaling Policies, are the most important things to do right now.
So, I mean, you say that you want to be more transparent, but you're not really addressing the question. I mean, the question is, can you quantify the risk of something cataclysmic happening? And I do want to say for all four companies, and I know you're not in the chambers, but we do have a chart for each of your companies had instances where agents went rogue. And in each of those instances, it is not as if any of you notified the public when those occurred. In some instances, it was months after the fact when the public was even notified. And in one instance for the government of Australia, a general email was sent to them. So this doesn't exactly increase transparency.
Mr. Graham, do you want to respond?
Yeah, first, in efforts to sort of quantify these risks, we have a number of quite detailed reports that try to elucidate our entire thinking around this. We increasingly do risk reports in addition to our normal Responsible Scaling Policies where we outline notions of our threat models and how likely we think certain types of these threats are likely to come to pass. And then we look backwards at things that are happening within our own company, and we ask what is the likelihood that these threats might come to pass in the future, including forecasting, surveying researchers at Anthropic as well. And we publish this information. We are...
Are those reports publicly available that you referenced?
They are. They are 30-page reports of these.
And then I'm now going to turn to Mr. Cahill for the response to that first question on quantifying the risk.
Speaker, thank you very much for the question. First off, we're committed at Meta to building AI safely. Now, we have a multi-layered approach to this where teams evaluate before deployment, including adversarial testing, applying safeguards, and deploying when only risks are mitigated. We set this out in our public Meta Superintelligence Scaling Framework and our preparedness reports that go alongside the release of models.
So can you quantify what the risk is or you're not able to do that?
Speaker, I don't wish to be imprecise in relation to the question of quantifying. I don't have our framework in front of me right now, but I'd be pleased to follow up with you afterwards.
Okay. It's just given the serious safety risks that have been raised here, to not be able to quantify what the risk is is troubling at best. I'm going to now move on lastly to Miss Friend.
Madam Speaker, at Google, we take catastrophic risks and their possibility extremely seriously. And we address that in a few ways. One way is that we perform research into producing our own safety frameworks for artificial general intelligence. So, we have an AGI safety and security approach. We also have a frontier safety framework which is our protocols for monitoring our models under development for dangerous capabilities. Those protocols then point to safety mitigations that we must take if we in fact cross what we call our critical capability levels.
Part of our testing and evaluation is of course using commonly used benchmarks across the industry. Those benchmarks do provide some quantification of the capabilities of the models themselves. But when it comes to forecasting future catastrophic risk, it's not a perfect science at this stage. And in this particular case, academics have noted that there is no reference class on which to base quantitative probabilities. So we all have to be very humble about making such quantitative claims because there isn't really a rigorous scientific way to do those yet.
So basically I'm going to take it then that neither of the four of you, no company here, can quantify the risk of something cataclysmic happening. Okay, I'm going to move on. Sam Altman said yesterday that we should accept that, quote, 'the world should accept some bad things happening as a price of advancing this technology.' No one is questioning the incredible medical and scientific breakthroughs and innovation that AI has brought to this world. But why should the public accept that bad things should be happening? And why should AI companies get to decide what level of risk society must bear?
I'm going to go in order. I'm going to start with you, Mr. Dwire, please.
So, we think AI has tremendous potential benefits to help solve some of humanity's hardest problems, but no technology is without risks. And there are a wide spectrum of risks. Clearly, some risks are unacceptable. But we also think it's important to get technology into the hands of as many people as possible so that they can benefit from it. And that's why we are focused on safety. We train our models to be safe from the start. We evaluate them against a robust set of risks and we work with a large network of third-party evaluators. We've also supported regulation that requires third-party audits of our safety frameworks as well as regulation that focuses on youth safety and holds us accountable for keeping teens safe.
I do have a follow-up to that because it does seem to stretch credulity that the president of your company and his wife each gave $12.5 million to a PAC that opposed candidates to the tune of about $50 million due to their support for AI regulation. So if the company is so committed to AI regulation, why would that PAC exist?
So I can't speak to personal decisions. What I can speak to is our actions, which is to support the first of its kind to require third-party audits of our safety frameworks. We also supported a version of a bill in Massachusetts which was the first of its kind to reference RSI, which the panel earlier spoke about. So again I can speak to our actions which is to support strong harmonized frontier safety legislation including at the state level.
Okay. I'm going to move on to the other three companies to the answer to the same question about why should the public, why should AI companies get to decide what level of risk society must bear? So, I'm going to go in order. Mr. Graham.
Yeah. Well, I didn't see all of Mr. Altman's comments yesterday. I share the notion expressed here that number one, while maximizing the benefits of this technology you need to minimize the risks and we should not be comfortable with letting these risks happen. This is why over the course of our history and my team's history we have been, I think sometimes accused of being overly cautious, I would say appropriately cautious, just this year withholding our most powerful model in order to just have more time to even think about collectively as industry as well as government what to do about these capabilities. And fundamentally this transcends the labs and industry. From our beginning we have long supported the role of government here for exactly this reason.
Apologies, Speaker. There was a strange moment there with the mute button. Thank you very much for the question. In relation to the quote from Mr. Altman that you read out, I don't see it the same way. We absolutely recognize the risks of AI and the opportunities. That's why we have our scaling framework. It's why we have our preparedness reports. That's why we have teams and teams of people and a multi-layered approach to AI safety.
One thing I wanted to mention if I may is that Meta's vision is to bring personal superintelligence to everyone. And I listened very intently to the panel before us in relation to recursive self-improvement, and I just wanted to mention that committing significant majority of compute towards serving people rather than racing towards recursive self-improvement is one of the best ways of ensuring that the technology is developed safely. Meta has made that commitment and other labs can do that also.
Yeah, Madam Speaker, we shouldn't just accept bad things happening. We should be working continuously and rigorously on AI safety which is a collective effort. It will be a collective effort to identify and mitigate risks, to establish technical standards to make those identifications and mitigations all the more rigorous. But I also want to note that one of the promises of AI is that AI itself can make us safer. I would call your attention to Waymo vehicles, which in one study had a 96% reduction in injury-causing crashes at intersections, which led Dr. Jonathan Slotkin in the New York Times to call it a public health intervention. So, we should think collectively about preventing and mitigating risks from AI, but we should also keep in mind that we can use AI itself to reduce risks to human safety and well-being.
So, if one of your frontier AI models goes rogue and causes serious financial harm, compromises sensitive data, causes physical injury, or contributes to a death or deaths, do you each believe your company bears legal responsibility? I'll start with you, Mr. Dwire.
I believe that OpenAI takes its responsibility for safety very seriously. And we are responsible for developing and evaluating our systems to ensure that they are safe.
So is that a yes or that is a no? Could you be more clear, please?
We believe that we are responsible for ensuring that our systems are developed safely. And that includes training them to be safe, evaluating them to be safe, and then continuing to monitor our safeguards after deployment.
Okay. I'm going to take that as a no because that is not really clear that you would bear legal responsibility. I'm going to go now to Mr. Graham.
Yeah. As a technical safety research lead, I think there are others which will have more nuanced perspectives on that issue than I. What I do know is first this is an issue where every lab we think should be committed to a multi-layered safety stack to make sure that doesn't happen in the first place. These issues don't come about in the first place. The second, this is candidly a big and complex question that also transcends the labs themselves. This is a question about the role of government and others which we've very long welcomed here.
So could we get a yes or no on that question?
I wish it were that simple and I wish I, you know, as a technical researcher had more nuance on that exact one. It is one that I think needs to be raised and addressed and transcends the labs and the companies themselves.
I'm going to go to Mr. Cahill.
Apologies again, the mute button. I'm sorry.
We will get it right. Thank you very much for the question. I'm not in Meta's legal department, so I don't want to speculate or be imprecise in relation to legal liability question. What I can say though is that we are absolutely committed to developing our AI safely pursuant to our framework which I mentioned previously. This encompasses the training, evaluation and release of models across development, deployment and use.
Madam Speaker, at Google, we believe that existing legal frameworks do apply to AI. As I said in my opening statement, we've long said if it's illegal without AI, it's still illegal with AI. And in New York State, frontier AI in particular is regulated under the RAISE Act.
Yeah. Thank you. I mean, your answer is far clearer than the others, so we appreciate the clarity on that. Honestly, I now have a question. I'm going to ask you to please raise your hand if your company has insurance for catastrophic risks.
Okay. I'm going to take that as a no, that no company has liability insurance for catastrophic risk to cover large-scale harm. So then the public, I assume, would be asked to absorb the costs.
Madam Speaker, I have to be clear. I don't know the answer to that, but I'd be happy to consult with our lawyers and get back to you.
Yes, we would appreciate an answer back expeditiously to this committee.
So, I want to now ask each of the companies to assure the public under oath that your AI agents will always comply with the safety guardrails that you impose on them and that you each have said are necessary to prevent catastrophic harm. I'm going to start with Mr. Dwire, please.
As I've said, OpenAI prioritizes safety. And we evaluate our systems against multiple dimensions of safety. And we monitor our safeguards after deployment. It's not possible for me to commit or guarantee that any technology is without risk. But I can commit that OpenAI is taking every step it can to ensure that the development of our systems as well as their deployment is conducted safely.
Okay. So that again, it does not sound like you are saying that the agents will always comply with the safety guardrails. I'm going to move on to Mr. Graham.
Yeah. What we are committed to and how we've designed the safety processes that we put in place is to try to handle exactly this. What we're committed to is trying to define the industry best standard for safety and pushing that ever higher. But candidly the science of doing this is fundamentally hard and unsettled. From the beginning of the company we have tried to publish the best research we can and be as transparent with the entire world in many cases where things can go wrong. It is for that reason that we push for ever better safety standards, for regulation and for discourse like this. It's candidly unsettled now and this is why we need to be talking about it.
Speaker, thank you very much. If I may for a moment, I was trying to raise my hand the previous round but again I will with technical difficulties. I just wanted to for the record in relation to your previous question, I just wanted to signal also that I don't know the answer to that question. I'm not the right person. But I would like to be able to come back to you afterwards with an answer if that's okay.
I do just want to say and I appreciate you each have your own, and look I was a regulatory attorney at a large company. I understand that people have their sphere of work that they work in, but to not know if the company has catastrophic risk insurance when you have a product that some of your own founders are saying could cause cataclysmic risk is troubling at best. But yes, please do. We would appreciate you getting back to the committee on that. And then if you want to answer the question about will you assure the public that your AI agents will always comply with the safety guardrails that you impose on them.
Thank you, Speaker. What I can guarantee is our commitment to developing and deploying our AI offering safely. We have every incentive to address safety and people will only embrace AI if they trust it's safe.
Google similarly has as our ultimate goal reliability and safety assurance from our systems. To promise perfection would not be possible with any product on the market but our goal is to get as close to that standard as possible. And so, we are constantly improving our products and our safety practices to ensure that our users are safe.
I don't think we're asking for perfection. We're just asking for accountability, transparency, and safety overall. Yes, ma'am.
Okay. New question. How many times for each of you has one of your models or agents gained or tried to gain unauthorized access to another system or escape from a so-called sandbox test? And are there any incidents that you have not disclosed publicly? I'm going to start with you, Mr. Dwire.
Yes, thank you for the question. And I did want to start by, like the others, clarifying that I don't know the answer to the insurance question but would like to follow up. Now on your question about agents, you may have read in the news that we reported we had an incident with Hugging Face. As a result of that incident, we initiated a significant set of third-party investigations into what happened, and then we made those results public. We have also initiated a look-back investigation to try to identify whether there were past instances of what we call misaligned agents. That investigation is ongoing, but we are committed to continuing that investigation as urgently as possible, to notifying potential affected third parties, and then to making the results public.
Look, we think transparency here is critically important for policymakers like you, for the public who wants to know whether they can trust our products, and for the rest of industry so that we can all learn together to raise the science of safety that we've been talking about here today.
Since you mentioned Hugging Face, can you confirm, did the company select this third-party investigator?
Um, I believe that we selected multiple third-party investigators.
And why did the company narrow the dates of review to a three-week window when reviewers said that virtually all the data they examined was actually from July 7th to the 13th? Were there any incidents that the company was aware of that were outside of that time frame?
We, um, we limited the amount of time that the third-party investigators had to conduct their review because we felt a sense of urgency. Uh, we believed it was important to get information to the public as soon as possible, uh, so that they could harden their own systems. Um, that said, when those investigators asked for more time, we gave it to them.
So, are you committing then to allow outside investigators to examine additional dates outside of the time frame that the company selected?
We continue to work with multiple third parties. Um, and we have also endorsed multiple pieces of legislation that would require us to work with third parties, including third-party audits of our safety frameworks.
I'm going to go to Mr. Graham.
Thank you. Yes. Um, so to your original question of, uh, how many incidents, uh, you know, cases of breaking out of sandboxes, uh, these are capabilities we evaluate for all the time. We've published not just with these incidents, uh, from the summer, um, but also previously cases where we observe advanced cyber capabilities, including breaking out of sandboxes, which, to be clear, from a technical standpoint, is just one part and, you know, one type of capability that does not in itself mean that the system would reach, uh, a real-world affected party, but ones that we did observe, uh, in training at points, and we disclose that in a lengthy, lengthy risk report.
Um, it's difficult to comment on ongoing, uh, investigations. Uh, it's really important that we do that in as secure a way as possible. We follow a process of, uh, making sure we can remediate and do so and be as transparent as possible while preserving the safety of any affected party. And of course, incidents are, are always ongoing. But I, I sort of reassure and reassert that we are extremely committed to being as transparent as possible, fundamentally because we believe that when we can, being as transparent as possible about what we've learned, who is affected, how others might learn from it, how to defend against risks or how we can help them as well, uh, is, is our number one priority.
So, so on that note then, are there any incidents that you have not disclosed publicly that you're currently investigating?
As a matter of sort of ongoing business, there are incidents, uh, sort of all the time of many different natures. So, for example, uh, one that we, um, are transparent about quite a lot when it's the right time to do so are cases of misuse, for example, on our platforms. Um, I'd refer back to our early September report on our threat intelligence report that we released, uh, where we detail this in, in quite some detail. What's really important though is sometimes we have to work with law enforcement and the right level of government to remediate, uh, to figure out what information you can and can't disclose to make sure that the affected parties remain safe. Uh, and so there, as a matter, there's ongoing all the time, but disclosure at the right time in the safest way is our commitment.
And when will you be disclosing these new incidents that are being investigated?
Uh, this is sort of an ongoing, ongoing practice. Uh, we regularly release threat intelligence reports in cases like that. When we release a model or when we do risk reports, which we do regularly, um, we, we sort of have a cadence and schedule that we do that. Uh, and then when important, uh, and sort of urgent, we will, will disclose in advance of that schedule as well.
I'm going to go now to Mr. Cahill.
Speaker, thank you very much. Um, I'm not aware of, um, any other incidents beyond the incident, uh, in the summer, which we have a, a public, uh, post, uh, setting out the details in relation to pursuant to our independent review of that.
Okay, I understand you're not aware, but is the company aware? Just to make the question broader. In other words, would there be others at the company who might have that information?
I don't, I don't, I, I don't wish to speculate in relation to that, but I'm happy to follow up afterwards.
Okay. I'm going to move on to Miss Friend.
Yes, Madam Speaker. Um, in our three incidents, um, of agents, um, leaving a, a test environment and interacting with the real, uh, internet, um, in all three incidents the models stopped their activities as soon as they realized that they were interacting with live websites. Um, we reported to, uh, the owners of those websites as well as to federal agencies, uh, about those incidents. Um, I am not personally aware of any additional incidents.
Okay. And same question then, is the company aware of, are you, do you have any knowledge that others in the company would be aware of any additional such instances that the company has not publicly reported?
I do not, ma'am, but I can take that question back as well.
Okay. Will you all commit right now under oath publicly that if your models fail an internal test or a third-party validation test that's not selected by your company, will you commit to not release that model? Going to start with Mr. Dwire.
Yeah, I can commit that OpenAI is not going to release models that we don't believe are safe. As part of, uh, that assessment process, we work with multiple third parties and, as I mentioned before, with the US as well as the UK federal governments, uh, to assess whether our models are safe.
So if it fails either an internal test or a third-party test, you will commit to not release it.
Um, OpenAI has historically, uh, delayed the release of models, uh, to ensure that we can build up the right safeguards. Uh, we've done it before and we'll do it again.
Again, I think a simple yes or no would instill more confidence in the public on a matter as serious as this. I'm going to move to Mr. Graham.
We take that one particularly seriously. The way we approach it is we try to detail, uh, an even more nuanced and lengthy approach to how we release models, what safeguards they require in order to be released, what tests need to be done. Uh, and to your point about independent evaluators, this is one of the most important components we think. Uh, we have tried and long supported, uh, creating a burgeoning sort of group of independent evaluators.
We try to refine and improve the exact decision process by which you release a model. Uh, and so we think, you know, what we should do is develop the right and best and improve on these decision processes. We detail ours in our responsible scaling policy. We should refine and improve it every so often. So we release updates based on what we learn, uh, you know, every six to twelve months or so at this, at this rate. Uh, and that this should involve independent evaluators.
Okay. That also is not a clear yes or no. Um, I'm going to move on to Mr. Cahill.
Madam Speaker, thank you very much for that question. Um, what I can commit to is that we do not deploy models, um, which are not safe pursuant to, uh, to our scaling framework. Um, and I just wanted to mention also that, um, I think I mentioned this, um, a little earlier, but we do believe that every lab has a responsibility, uh, to go at the pace to ensure the safety of their models and the ability to take action in relation to that pacing. Um, for example, at Meta, we delayed the release of Muse for several months, uh, to focus on safety and security. And in relation to, uh, to independent evaluators, um, we recently, um, signed an accord with a commitment to partner with independent, uh, evaluators and assessors. And this is also detailed in our, uh, in our framework.
Okay. Mr., uh, Miss Friend.
Uh, ma'am, as part of our multi-layered approach to responsible AI governance, um, we conduct rigorous launch reviews ahead of launch of models to ensure that anything we put into general availability is safe.
Okay. I mean, again, I think it would be far clearer if we could get clear yeses or nos from everyone, but, um, in the absence of that, I'm going to, um, take, you know, what you've each said as sort of an equivocation. Um, okay, I'm going to move on. I have two final questions and I'm going to pass it on to our, um, technology chair. For OpenAI, you sent a letter to the council on October 1st and in it you recommended that to better protect New York City, the city should connect its existing cyber and emergency management capabilities with the strongest available escalation, containment, and recovery capabilities. Have there been any conversations with the city of New York about that recommendation?
Uh, I'm not aware if there have been conversations, um, about that specific recommendation. Um, I'm happy to follow up. Um, I do know that with respect to cyber, we recently launched a $1 billion initiative to make investments in cyber defense that includes giving organizations access to our tools to defend themselves, training, um, and technical assistance. Um, and we're working with New York State, uh, as well as with the Port Authority.
Okay. And then the last question I have before passing it on to the chair is for each of you, based on your own internal projections, what percentage of New York's workforce do you estimate will be displaced by AI over the next five years? And I'll start with you, Mr. Dwire.
I don't have a number, uh, to share with you. Uh, but OpenAI has an economic research team that is looking at this exact question of how AI impacts jobs. Um, we think it's critically important to understand how AI might impact work. And we also think it's important to ensure that AI is not just being used to increase productivity, uh, but that it actually benefits workers and gives them more opportunity. Um, that's why we're committed to making our analysis of job impacts public, um, and why we've also worked with policymakers, um, including, uh, endorsing the bipartisan workforce transparency act, um, to create voluntary pathways for other companies, uh, to report potential impacts.
So Boston Consulting released a report, uh, recently that said, uh, that they believe there will be a 10 to 15% of the workforce could be displaced by AI by 2031. If we took that out to look at New York City, New York City has roughly 4.2 million public and private sector workers. At 15%, that would mean more than 600,000 New Yorkers potentially displaced from their jobs. Was that an assessment that you agree with?
Um, I'm not familiar with that particular report. Um, I'd be happy to follow up, uh, in our written testimony with more details on OpenAI's, uh, analysis into this topic. We've looked at job exposure. Now, exposure doesn't mean elimination. It means that jobs might change over time. Um, so I'm happy to follow up with some details of, of that research.
I'm going to now turn to Mr. Graham to answer the question about what you believe the percentage of New York's workforce you estimate will be displaced by AI over the next five years.
Well, I don't have an exact number. We are particularly focused and concerned about this question of job impacts. We've been very vocal very early for the past few years on, on exactly this. Um, to do this we set up an economic research team. They sit right next to my team on the floor. Uh, and they published a number of things that I think actually contain valuable information here. A number of economic scenarios, interactive economic scenarios to understand what the impacts on growth, unemployment might be, uh, as well as an economic policy framework, um, to try to wrestle with exactly that question of what we think, uh, governments ought to do, uh, there. Um, I'll just say the fact that, you know, the reason why we are doing this is because we're particularly concerned about, about exactly that question.
I mean, obviously losing a career job is catastrophic to any worker and to their family. So displacement on this scale is going to have enormous consequences not only on the individual and family level but for our entire city and the economy. Um, I'll move on to Mr. Cahill.
Madam Speaker, um, first off, I want to acknowledge absolutely the concern in relation to, um, how, how you, um, have, uh, articulated it. That is indeed, um, a concern, but I look at this somewhat differently. Um, I was reading some incredibly, um, impressive statistics from NYC EDC, uh, in relation to the investments that are happening here, uh, in New York. I think 20 billion VC investment in AI, 40,000 jobs. I think those figures are from 2022, so it's probably a little bit more by now. Um, but what I'm really actually incredibly excited about and very optimistic about is about small businesses. Um, small businesses are the backbone of the economy. That's so true for New York. There are so many small businesses. It's truly, truly awesome. Um, and we believe that the impact will even be larger, um, from, from small businesses, um, which will, um, have personal super intelligence and be empowered to really grow, uh, their businesses, which in turn will lead to more jobs and growth. Um, we don't believe that, um, you know, our vision for super intelligence, um, is augmentation. Uh, it's invention, not automation. It's really about empowering people. Uh, and this is why we developed Muse for Business. I'd be happy for small businesses, I'd be happy to talk a little bit more about that in detail, um, as we move through the day. Um, I just wanted to mention as well, uh, as I was kind of like reading stuff for today, um, I was reading about Y Combinator's cohort for 2026. There are 240 AIDCs, uh, in the, in the 2026 cohort, uh, and that's in New York City. Uh, so I just thought that was pretty awesome.
Um, yes, no one's doubting the importance of starting these VC firms here, uh, whether it's through Y Combinator or through, uh, Cornell Tech, which is in my district, which we're thrilled is incubating so many startups. That is obviously something we want to encourage, to, uh, encourage these businesses to locate here in the city. I think our concern is obviously on this job displacement since you mentioned small businesses. Unfortunately, more small businesses have closed this past year than opened. So, the situation for small businesses is dire and it's something obviously that this council is taking very seriously. Uh, Miss Friend.
Madam Speaker, I don't have statistics for New York City in particular at hand and I'd be happy to ask our team, uh, to get those to you right away. What I can say is that we also have been investing a lot in research into the labor impacts of AI. Um, and from what the early empirical evidence, uh, that is available, um, from what we can see, it is likely that some jobs will be eliminated, some jobs will be created net new. Um, but most jobs will change. And so the other thing we've been doing is investing in upskilling, um, for workers in multiple different sectors through our AI Opportunity Fund at Google.org and through several other investments to ensure that we can all navigate this transition successfully.
And I'll just, um, close by saying the council is investing in upskilling, uh, making sure that workers have those new tools, uh, in this economy and we are very focused on that as well. I'm going to pass it over to Chair De La Rosa.
Thank you, Madam Speaker. Um, many of us saw and I guess the world watched as this week your companies and others met at the White House and there was some voluntary commitments that were made there. Can you tell us about what your company specifically agreed to and what are the next steps? I'll start with Miss Dwire.
Um, yes. So, our company along with many others, um, did commit to, uh, the framework that the White House laid out. Uh, but I want to be clear, voluntary commitments and self-regulation aren't enough. Um, and that's why OpenAI has consistently supported frontier safety regulation at the federal level and why we have endorsed multiple bills at the state level, um, to ensure that in the absence of binding federal regulation, it continues to move forward, um, and keep people safe.
Just to follow up real quick. So does that look like independent regulators, mandatory audits, penalties, what else?
Um, so at the federal level, uh, we have outlined a couple of different, um, elements that we think would be, um, appropriate for federal regulation. That includes mandatory pre-deployment evaluations for national security risks, uh, which OpenAI does voluntarily. Um, we also think it should involve independent technical assessments, um, that we've been talking a bit about today. We think those independent technical assessments are particularly important for assessing internal risks at companies as well as the pace and risk of RSI.
Thank you for your answer. Let's go to Mr. Cahill.
Chair, thank you very much for, for the question. Um, yes indeed, uh, Meta along with, uh, with our peers, some of our peers, signed, um, the accord, uh, for super intelligence. Um, this accord is a, is a, is a start and an accord the whole industry can come behind. Um, it includes, as I mentioned, I think, uh, in one of my previous answers, um, a partnering commitment to partner with independent evaluators and assessors. Um, and this is something that we have also specified in, uh, in our scaling framework.
Are there commitments beyond those that you agreed on that you're, that, uh, your company's willing to make?
Uh, beyond the accord, Council Member?
Yes, beyond the framework.
Uh, the, the accord is a start, uh, as I mentioned. So, um, I think this is, uh, a continuing conversation.
So, uh, as the other witnesses noted, the accord at the White House, which we also signed, uh, called for, uh, internal controls, um, partnering with external auditors and evaluators, and, uh, ongoing monitoring of systems, all of which we think is very healthy and much of which, uh, we already practice. I'll also note that, uh, towards the end of the accord, um, it observes that it may be necessary for these, uh, uh, commitments to become part of law. Um, and as I said at the top, we have long, uh, worked towards a federal framework for AI safety.
Yeah. Likewise, the accords, to my understanding from what's public, calls for security standards, internal controls, uh, independent evaluation. Uh, we support that and have for a very long time. Um, we also obviously think this needs to go further. We've supported regulation and we've proposed, uh, our view on regulation and our advanced AI framework, which details the need to have safety frameworks, testing, and what kinds of security standards, safeguards, uh, to encode the need for independent evaluators. Um, we are now trying our own, uh, to help, um, so advance the science of independent auditors as well of safety processes, as well as investing in cyber and bio defense. Uh, and in absence of federal legislation here, we've also supported safety bills at the state level for audits and evals of this kind to accomplish this.
Thank you for your answer. Uh, earlier you all testified that to your knowledge no whistleblowers had been fired directly for being whistleblowers, um, from your company. So my question is kind of the opposite of that. Has anyone, has anyone... Miss Dwire, you can begin.
Thank you. Um, so I, I don't know, uh, about HR processes, um, that have been made or decisions related to HR that have been made. What I can tell you is that, um, after the Hugging Face incident, which I think you may be referring to, OpenAI instituted, um, significant changes to our overall organizational process. That includes, um, clarifying escalation pathways and making them simpler. Uh, that includes clarifying responsibilities across teams and creating clearer processes, um, for teams to make decisions to pause or to restart activities.
So to your knowledge, no one was fired after the Hugging Face incident? Yes or no?
Again, I, I do not have any knowledge of HR activities within OpenAI. I lead our policy team.
Thank you, Chair. Um, I also lead our AI policy, um, at, at Meta, and I'm not the right person to respond to your specific question, but I'd be happy to have, um, our teams follow up with you afterwards.
It would be great if all of the companies could provide that information to us. Um, it's, it would be useful to our policy deliberations. Miss Friend.
Yes, ma'am. Uh, same answer. I'm not made aware of any particular personnel actions inside the company, but can also pledge to follow up with you all.
Thank you. It seems like we lost, um, Mr. Graham. I'm not sure if he's on. Seems like he's not. Okay. Um, I'm going to ask one more question and then we're going to pass it on to members.
Chair, he, he is online. Sorry.
Hi there. Apologies. We're just trying to fix the video, uh, in the room. Uh, likewise, I'm, I'm not aware and I'm not the, the best person to ask on that. We have a detailed, uh, whistleblowing policy, uh, for example, to, to handle this. Um, but we can have our team follow up.
Thank you. Uh, my last question before I start to turn over to my colleagues. You know, uh, we are concerned, I am concerned as a technology chair about public infrastructure to make our communities literate on AI, um, and have a better understanding of how AI impacts their life and also to safeguard them, um, from the proliferation of AI. Um, your companies and your CEOs have captured, um, the top percent of the, the world's incomes, um, in my opinion, uh, widening the wage gap and the income inequality in our city and throughout. Uh, do your companies have, um, a mission? Do they have any type of, um, understanding of the impacts that those wage gaps have in communities? And specifically, are there any resources that are being deployed in order to help stand up infrastructure in cities, municipalities, states where you all are headquartered? We can start with SW.
Um, so I will answer your question in two ways. First, um, to your point on AI literacy, um, and education. Um, education is core to opportunity. Um, and that's why OpenAI is proud to have partnered with the American Federation of Teachers. Uh, we have a $10 million partnership including a training facility in New York City, um, because we believe first that teachers need to be prepared, um, to introduce AI literacy and AI responsibility into the classroom.
About economic opportunity. Um, I'd be happy to follow up with, um, some policy proposals that OpenAI recently put forward that address, um, exactly that question. Look, a future where AI exacerbates economic inequality is not a future that we want. Um, we have started thinking through creative policy proposals, um, to help shift the trajectory of technology and how it's impacting the economy. Um, and we are committed to working with policymakers to begin piloting those ideas, um, and putting them into practice.
Chair, thank you very much. This is, uh, such an important, uh, topic, such an important question. Um, so thank you for, uh, for asking it. Um, I listened, uh, with great interest to your opening remarks where you spoke about digital divide and AI literacy and making sure that New Yorkers fully benefit from this technology. Um, really couldn't agree with you more on, on all of that. Um, Meta's vision, um, is to put personal super intelligence into the hands of everyone, all New Yorkers that want to use it, uh, to pursue their own, um, aspirations. Um, our Muse product, for example, is free. Uh, and I mentioned our, our Muse for small business a little earlier and, uh, we'll be happy to talk a little bit more about that as the day moves on. I also, um, wanted to mention as well that, uh, we have, um, our work America's Workforce Academy, which we're investing in, and also our Futures for Everyone Fund. Um, so those are just two, two, two examples I wanted to, uh, to reference as well. Thank you.
Yes, ma'am. As I mentioned earlier, uh, Google has investments in AI upskilling to address exactly this challenge of ensuring that, uh, current generations and the next generation are well equipped to benefit from AI and to, uh, leverage it in their careers and in their personal lives. The mission of Google is to organize the world's information and to make it universally accessible and useful. Um, our approach to AI is very consistent with this. Um, the other thing I'll note is that again, early research into, um, impacts of AI on labor demonstrate that AI tools, uh, actually upskill those, uh, uh, who are at the lower end of the wage scale. That in fact it helps people move up the value chain, um, because it can, uh, boost people's expertise. Um, so we're very optimistic about the ability, uh, of, of, uh, everyday Americans, uh, to really leverage AI tools, uh, for their own economic benefit as well as personal benefit.
Yeah, likewise. First, we believe in, in two things that are very important to happen here. The first is the science of the economic impacts of AI, um, including on labor, wages, uh, and, and the like, and we have tried to build what I think is the world's best team on this, uh, and publish as much as we can. Um, in addition, my understanding is we do have, uh, training on using and speeding up on, on AI for exactly this reason. We, we share that, uh, that mission very deeply and, uh, I, I will say again, we are troubleshooting the video and I will be back with you very shortly.
Thank you. I'm sure you have the best in the nation helping you out there. Um, so we are going to move on to member questions. I want to remind members that there will be about a minute to ask your questions, but there will be multiple rounds. Um, we are asking members to try to ask all your questions first and then wait for the response. And we have about 30 members on round one. Uh, we will start with our deputy speaker, Williams.
Hello. Um, I'm going to just jump right into a question on my bill. So it has to do with unauthorized depictions of officials by AI. What steps have you taken to ensure that your AI tools cannot be used to create deepfakes of public officials or to spread misinformation? Anyone? I guess Mrs. Dwire.
We're going to keep starting with me. Um, I'm happy to answer that question. Thank you, uh, for that and thank you for your bill. Um, look, OpenAI does not believe and, well, let me back up. OpenAI's usage policies prohibit the use of our tools for election interference. Um, we also believe it is very important that individuals know where their content came from and whether it was AI generated. Um, and that's why for all of our audiovisual, um, content we tag it with what's called provenance. Um, that's a technical term, but it means like a watermark or metadata, um, so that people know the content is AI generated. Um, and we also make publicly available a tool, um, that allows the public to test whether content was AI generated.
Uh, so, um, Claude, our, our model Claude, is a language model and so it doesn't generate images. Um, therefore we, uh, sort of don't, um, so handle deepfakes as a result. Um, that said, uh, we think it's important for a number of other reasons, uh, that we can get into, to do what we call watermarking of, uh, text in order to identify if our models themselves are being, uh, used and misused in other ways. But again, we, we don't generate images, uh, for, uh, in part for this reason.
Mr. Cahill, followed by Miss Friend.
Deputy Speaker, thank you very much for this question. Um, at Meta, if content is generated by one of our AI tools, uh, we will also embed, um, metadata and attach watermarking to it so that people can identify if it is AI generated. Similarly, if we detect, um, uh, watermarking and metadata, um, on our platforms, we will attach a label to it also.
Uh, yes, ma'am. Uh, at Google we are very proud to have invented our SynthID, uh, watermarking feature which many across the industry have also adopted. Um, we are also part of the Content Coalition for Provenance and Authenticity, um, standard which allows... standard which allows... require on our YouTube platform that, um, significant use of, uh, generative AI in content, um, be labeled as such by, uh, our creators. Um, and we were also the first, uh, major company to require that election advertisers also prominently disclose when they are using generative AI technologies in their campaign ads. Um, all this put together means that we care very much about information integrity, uh, across the web and across our own products and services. Um, and we, uh, continually invent and improve technologies to help our users understand, um, generative AI content, um, and also understand, uh, the origins of that content.
Okay, thanks. Um, just one more question. Um, so as chair of the committee on cultural affairs, um, I also want to look at this issue through the lens of New York City's artists and creative workers. So what protection should exist when an artist's voice, likeness, performance, or creative work can be replicated by AI without their consent? So how do we embrace useful applications without devaluing or displacing the artists and creative workers who make this economy possible?
So I will start, I guess.
So I was going to let you off the hook and start with Mr. Graham, who still has some video technical difficulties.
I assure you we, we have a whole team here working on this and I'm, I can't wait to be back with you. Um, yeah, this is, this is very, very important. Um, I think, you know, personally I think the, the risk of this is a bit more minimized when we just have a, a text model, but it's really important, um, to, uh, you know, be very sensitive about this issue. I think, um, while it's, you know, it's outside of my area of expertise as a technical safety researcher, uh, I do think that this might be an avenue where, uh, regulation or otherwise might play a role. And at Anthropic, uh, we, we are compliant with, with, um, with all laws there. Um, I share that, you know, personally as, as a, you know, an artistic person and, you know, friend of many artists for this reason. Um, but, uh, yeah, we would love to engage on it further.
Thank you very much. Uh, Deputy Speaker, um, Meta takes creativity, copyright, IP rights very seriously, um, and, um, our practices are consistent, um, with law. Um, I also share, um, uh, the, uh, the views of, uh, the rest of the, the witnesses just in relation to the leadership and the constructive voice that you are bringing to this conversation as well and that of your colleagues.
Miss Friend. And lastly, Miss Dwire.
Yes, ma'am. Uh, for us, uh, it's always important to keep in mind that we, uh, design a lot of AI to be useful to creators. Um, so we have many, many different creative tools, uh, across our AI offerings that is designed to really, um, uh, be something that creators can leverage. So you see that everywhere from our tools to our YouTube, uh, platform as well. That said, um, we also think that as part of sharing the benefit of AI with creators, it's important that they have control. Um, and so we have a service called Google Extended, um, where website owners can opt out of, uh, AI training on the web. Um, and, uh, we hope that this strikes the right balance between AI for creativity.
So we believe that AI has a tremendous potential to enable creativity and innovation, um, and allow small businesses or independent creators to compete. Um, that said, uh, we also have partnerships, uh, with publishers, for example, uh, like News Corp, The Guardian, and Hearst, um, that help people discover their work and their reporting, um, and help them reach new audiences, um, in addition to developing new products. Um, and we're willing to work with creators, um, on a range of commercial ideas, um, including ways, uh, that their content can be integrated into our outputs.
Thank you so much. Thank you. We will hear from Council Member Krishnan, followed by Riley, followed by Lee.
Thank you so much, Speaker Menon and Chair De La Rosa. I want to start by saying that I think AI can be useful both in government, medicine, and public health and our society more broadly. But the need for regulation is undeniable, and I'm very concerned about how much AI is taking from New Yorkers with very little return. AI has taken over our economy, inflated valuations, stealing our public land for data centers, and increasing the net worth of AI CEOs like Elon Musk, who became our first ever trillionaire. With our whole stock market quite literally in your hands, there's a reality that AI will eventually need to prove a far greater return on investment than you all are demonstrating right now. I'm personally deeply concerned about New Yorkers' retirements and 401ks that are now reliant on your companies yielding profits or could be in the future. It is dangerous and reckless to assume that you all are too big to fail and may require a government bailout in the future. As you all are public companies or racing towards IPOs, how exactly are you all planning to protect New Yorkers' and everyday Americans' money? How will you all ensure that you are not too big to fail?
Mr. Dwire, we can start with you.
Um, thank you for the question. Um, so I am not on our finance team. Um, so I'll have to get back to you with specifics there. What I do know is that our investments are made in tranches. Um, and they are made, the investment rounds are all based on market signals. Um, and I also know that we are committed to working with policymakers to ensure that AI benefits humanity. Um, that includes, um, creative policy proposals like I mentioned previously, um, thinking about how we can ensure that AI doesn't exacerbate inequality, um, and that everyday New Yorkers, um, and everyday citizens can benefit.
Sir, I would point to, uh, the thousands of small businesses across the country and in New York City that we support. Um, I think, um, uh, contrary to being too big to fail, we think we are an integral part of the economy, um, and helping, uh, American businesses and American workers across our suite of technologies.
Council Member, this is outside of my area of expertise. What I can say in my work is, uh, my job is to protect people from the risks of models by finding them as early as possible. We have a multi-layered stack and have worked for many years to, to make it robust and we'll continue to do so, um, by understanding and protecting against the most substantial risks of the models.
Council Member, thank you very much for the question. Um, thousands and thousands of, um, SMBs across the five boroughs, um, rely on, um, Meta's products including our AI offerings, uh, to grow and develop their businesses. I mentioned earlier our belief that, uh, small businesses are really the backbone of the economy. We truly believe that, um, and, uh, by, uh, placing personal super intelligence into the hands of everyone, uh, for free or for as affordable as possible, uh, through Muse or Muse for small business, um, we really believe that the small businesses and also nonprofits, um, like the Irish Art Center, uh, for example, or Emerald Isle Center, um, are, uh, are empowered to actually deliver even bigger impact.
I appreciate all of your answers, but you all are coming to a hearing today to testify about the dangers and risks of artificial intelligence. And the fact that either you don't have answers to this or you all are citing general statements about the performance of publicly traded companies does not answer the question about a specific issue that puts many Americans at risk that you are acknowledging can put many Americans at risk. And you all have not thought of an answer of how you ensure that Americans' money, New Yorkers' retirement money is not at risk as AI floats more and more of our stock market.
Thank you, Council Member Krishnan. Thank you, Council Member Riley.
Thank you, Chair De La Rosa and Speaker Menon, uh, for, uh, putting together this hearing. Thank you to the panelists for testifying. Uh, three quick questions, two pertaining to my bill and one pertaining to my role as land use chair. Uh, so the first question is, what challenges do you face in detecting malicious uses of your AI products and identifying the individuals responsible when those tools cause harm? The second question is, would stronger protections and reporting mechanisms for employees and users make it easier for them to come forward and report potential misuse or abuse of AI tools? And the last question, uh, as land use chair, I'm very, very concerned about data centers. So, what safeguards should cities put in place when locating data centers near residential communities, particularly to address energy use, noise, environmental impacts, and the quality of life? And I don't want to call on you guys. Just take turns to answer it, please.
Dwire, Graham, Friend, Cahill, please answer.
I'm happy to start. Um, first of all, I want to thank you, uh, for your bill, um, 2604, uh, which OpenAI is happy to support today. Um, OpenAI takes, um, safety issues very seriously, particularly as it relates to potential retaliation against employees. We have a strong internal policy against retaliation, um, that for employees that report safety incidents. Um, and we of course respect employees' rights to communicate safety concerns to government officials. Um, your second question on monitoring misuse, um, it's an incredibly important area. It is something that OpenAI takes seriously. Um, we also make transparent, um, any, we also make transparent detected instances of misuse. Um, and as part of our monitoring, we use that learning to constantly improve our safeguards. We, um, are constantly innovating on safety and so, um, instances of misuse when they're detected, we enforce on our policies and then long term we learn from them and we improve our safeguards. And the point on data centers, um, OpenAI is committed to paying our own way on energy. Um, that means different things in different places. Uh, but OpenAI has made that commitment at its data centers, uh, across the country. We're also, um, committed to working with policymakers. You know, we signed, um, Governor Gretchen Whitmer's, um, commitments for companies that are working in her state to ensure that there are responsible practices, uh, for companies building data centers locally.
Hi. It's, it's good to be back. You can see me again, I hope. Um, so, uh, on misuse and detecting risks, this is integral to what we do. Um,
Can you get closer to the microphone? We're having trouble hearing you. Sorry.
Now, we can't hear you. We can't hear you.
Okay. I will have the team troubleshoot and I'll be, I'll be back if that's okay.
We hear you. We hear you. Try now. Try. Go ahead.
How, how about now? Does this work?
Okay. My apologies. Um, detecting threats and misuse is the core DNA, uh, of our company to make sure that we make the system safe. We have an entire threat intelligence and safeguards team dedicated to this every day. Uh, and I would point you to quite a lot of what we put out publicly where we not only try to disclose as much as we possibly can, but we try to do so so that we could advance the state of the science of doing so for the entire industry. Um, uh, in addition, we have, uh, a whistleblower policy, uh, not only about us as a company but also our safety practices. This is public and we've also committed to, to ongoing reviews, uh, of this policy. Um, and while this is outside my area of expertise, what I understand about data centers is that we have committed, uh, to covering the energy prices, uh, or making, you know, paying, paying for our own energy use, um, uh, in local data centers. But I would, I would refer to, uh, other teams who are more, um, where it's within their, their sort of role to give you the full answer.
Miss... Oh, sorry. I got a, a notification. Let's go ahead.
Uh, so I will echo my colleagues. We also have content policies that we enforce across our platforms which include on occasion, uh, taking down content that violates those policies. Um, we also have a Google Threat Intelligence Group and multiple other cybersecurity, uh, intelligence programs that are constantly scanning, um, for the, the safety and security of the internet as a whole and also for our own systems. Um, we have also pledged to pay our own way when it comes to data centers. Um, I can follow up on any specific questions about local data center issues with the teams that work on each particular project.
Thank you very much, Council Member. Appreciate your patience there. Um, in relation to your, your first question and, uh, and your third question, we have expert teams, um, that are, uh, taking forward this work and I'd be happy to, uh, to connect you with them, uh, connect you with them afterwards. I don't wish to be, uh, imprecise, uh, in relation to the robustness of, uh, of, of, of our, of our content, uh, uh, policies in relation to, to data centers. Again, we have, um, entire teams of people here, um, who are working on sustainable infrastructure, um, and, uh, community impact, um, and can speak, uh, more in more detail than I can in relation to communities shouldn't be bearing the costs, uh, which is something that we believe in deeply. In relation to your bill, uh, Council Member 260, uh, for, um, for being a constructive voice in the conversation around whistleblowers. Um, it is very important, uh, to us, um, at Meta and we maintain, uh, whistleblower policies, uh, including anti-retaliation. Um, and we're actually developing additional reporting protocols for AI safety.
Thank you. Um, I appreciate the structure that you guys have, but I don't know if you guys entirely answered the question I asked. I really wanted to figure out what challenges you guys were seeing because hearing the challenges from you all will help us implement policy that can kind of protect individuals from artificial intelligence. So I don't want to take up too much time but, uh, looking forward to connecting with you all after, uh, specifically when it comes to data centers. Also want to hear any safeguards because there are a tremendous amount of data centers even though there was a moratorium put out earlier by the governor, uh, there were data centers that were implemented by a lot of residential areas especially in the northeast Bronx. We have data centers that are located next to homes and we have a lot of residents who are concerned. So would love to hear more about that, uh, moving forward. Thank you so much, Chair.
Thank you, Council Member. Up next we have Council Members Hanks, Dinowitz, and Maloney.
Thank you, Chair. Thank you everyone for being here. Uh, my question, um, is in relation to the disclosure of AI incidents in contracts. When an AI incident, uh, happens, New Yorkers need to have an early warning so they can understand. And my bill, uh, will require city agencies covered under these contracts to notify NYC Cyber Command within 24 hours of learning of a reportable incident and it also requires plain language public notice. So my question is, can your companies realistically meet that deadline and, you know, with the information that the city can act on even before you have, um, finished investigating? You could start with, um, Miss Dwire.
Um, so thank you for your bill. Um, we support incident reporting policies, um, and have instituted, um, a robust incident reporting framework internally, um, where any employee can report instances of potential misalignment. Um, and then those, uh, instances are investigated. Third parties are notified, um, and the public is made aware as well.
Thank you, Council Member, and thank you also for, for your, for your bill. We have, um, a global incident response program, uh, and report incidents as, um, as required under the law.
Uh, we also support incident reporting. We think it's very important to have continued public discussions about, um, what information is necessary.
or useful for public officials that can be actionable to take further steps to ensure public safety. We also note that the 24 hours seems to be in conflict with the state requirements for 72-hour reporting on incidents. And I'm happy to take a look at the different frameworks under proposal and get back to you with a more wholesome response.
Sorry, I appreciate that. Mr. Graham, who kind of looks like an AI guy in there. Can you confirm you're not an AI guy? Sorry.
I can confirm. Can you hear me? Okay. Now, does this work?
Okay. Thank you. I'm an AI guy, but in a very different way. I'm an AI researcher. So first, incident reporting, one of the foundational parts of the safety stack. We support the evaluations and auditing required to find the incident in the first place, both by externals and internal. Obviously it's beyond me to discuss particular, it's outside my expertise to understand sort of deep policy mechanisms here, but the reality from what I've seen is there are some incidents or issues where what you need to do is investigate and remediate and mitigate with the right folks and the right organizations at the right time. So I think we would support whatever allows us to achieve that. And happy to engage further on the details of how you would.
I appreciate that. Thank you so much. What should the first report tell us so we could begin protecting people? And as my colleague Council Member Riley said, your answers help us build better policy. So how would you keep us updated on what remains unknown? Mr. Dwire.
Yes. So OpenAI has a public website where we are reporting instances that we've detected of misalignment. So I would point you there and we can follow up with that website address. But I think that is a good starting point for the type of information that we think is useful to both policymakers, the public, and to researchers and other companies to better understand questions of misalignment and then work together to address them.
Thank you, Mr. Cahill.
Council member, if I may, I'd love to be able to contact you with our experts in incident response to be able to provide you with the details. I look forward to that. Miss Ren.
Yeah, for an initial report, I think what's important is to highlight what the specific event is and as much information as is known at the time, including any resulting harms, surrounding context, and what we call technical telemetry, further technical details about the sources of the incident. I think it's also very important for anyone reporting an incident to also indicate their confidence in the information that they're presenting so that whenever action is taken, it's taken on the basis of confidence in the verified nature of information.
Thank you so much, Mr. Graham.
Yeah. So first, making really good reports happen and being as transparent as possible is fundamental if you're going to remediate incidents. I would point to two things here. First, many incidents that we uncover or industry uncovers in this practice involves identifying, investigating and releasing the right information to the right organizations at the right time. Sometimes that might be the affected party and sometimes that might be law enforcement or different levels of government. So the first question is to whom and with whom as fast as possible. This varies by the nature of the incident. And then second, I'd point to as quickly as possible, as safe as possible, without putting parties further at harm, which we think is a fundamentally important part of public disclosure, maintaining their safety. We should disclose what we know about these incidents. And so I'd point you to our disclosures this summer, disclosures from various labs, and our ongoing reporting as well, so that you can always take a retroactive look at the sort of totality of instances.
Thank you so much. Thank you, Chair.
Thank you so much. So we're going to go to, I want to recognize first that we've been joined by Council Member Aldabal, and we're going to go to Dinwitz, Lee, and Maloney. Members, if you could keep your question to one minute, no closing statements. And the panel, we're going to continue to have to call your names, and that takes time. So if we could just keep the order, Dwire, Cahill, Friend, and Graham, just answer one right after the other so we don't have to continue to call your names. That would help us speed things along. Council Member Denowitz.
Thank you, Madam Chair. As chair of the committee on education, as a former teacher, as a parent, I am deeply concerned with the impact of AI on our kids. The large language models impacting their cognitive development and raising a generation of students who can't independently read or write. What is your commitment to severely restricting the output of your large language models for students in grades K to 12, K to 12? Given the unique nature of the individualized essay writing that your models do, are you willing to subject yourself to the guardrails we have in New York State, which prohibit an adult from writing essays for students and selling it to them or getting value from them? And then as we see AI proliferate in the classroom, we've also seen your models cheat in order to fulfill a goal. And so if you have an AI model whose goal is to improve student outcomes, what is preventing those models from hacking into student databases to manipulate grades to achieve that goal of improving, in quotation marks, student achievement?
I believe.
Yeah, thank you for your question. So first of all, OpenAI takes youth safety very seriously. We recently released a product, ChatGPT for teens, for users only between the ages of 13 and 17. So users below the age of 13 are not allowed to use our products. ChatGPT for teens is safe by default. That means that there are protections for harmful content, things like eating disorders, violence, suicide, and self-harm. There's also voluntary parental controls so that parents can take additional steps to protect their children. And I'm happy to say that OpenAI not only takes our responsibility to protect kids seriously, we also support legislation to hold us accountable to that responsibility. We recently endorsed, I think, the strongest in the nation youth safety legislation in California, SB 119, to require strong safeguards for teens 13 to 17 and to hold companies accountable.
And your models restrict, they do not write entire essays for kids.
Again, our models are not allowed to be used by kids under the age of 13.
Okay, we'll move on to Mr. Cahill.
Council member, thank you very much for this question. Absolutely recognize your deep concern in relation to the potential impact of AI on teens and minors. Meta is absolutely committed. It's an imperative to us to providing safe and responsible AI, especially for teens. We have default protections, and I just wanted to clarify as well that Meta AI is a 13+ product, so it's just 13 onwards. So I was talking about age-appropriate interactions. So, for example, content that teens would experience wouldn't be out of place in a 13+ movie, for example. We also have suicide and self-harm resources. We also have parental controls to give parents more oversight. And we do not permit our AI offerings to engage in romantic or sexual interactions with minors.
I appreciate that answer. I do want to highlight the part of the question which was that the cognitive development of our students includes the mental health and it also includes things like composition of essays, term papers, which your model still can provide directly to our students. So I'm also asking about your commitment to severely restricting the outputs, not just for their mental health well-being, but for their cognitive development. So I want to make sure that question is answered as part of this.
Thank you, Chair.
Can you all answer that question for our education chair?
Shane, do you want to continue?
Yep.
No, I'm happy to take that and then I'll hand off to you, Alice. The issue that I think you're raising in relation to Council Member, in relation to cognitive strength or the impact on cognitive abilities, is something that we absolutely are taking seriously in terms of being committed to providing safe and responsible AI, again, especially in relation to...
I would highlight not only that we work very hard with both internal and external experts in the fields of child safety and development and education, we also see AI as a valuable tool for young people to prepare them not only for the skills they're going to need for an AI-driven future, but it's also a very powerful learning tool. We're particularly excited about personalized tutoring that's available with AI tools. We also are very excited about AI's ability to support educators themselves. We know our educators are overtaxed, particularly with administrative burdens. And so if our AI tools can support teachers in everything from scheduling to planning lesson plans to also thinking about how to tailor their educational plans to individual students, we're very happy to support that. We also think it's very important for parents and educators to have control over how they use AI in classrooms and how their kids have access to AI. Every family is different, every student is different. What's useful for one student may not be useful for another. So through tools like Gemini Workspace for Education, we enable school administrators to control students' accounts' access to Gemini apps so that their students are using Gemini in the ways that their educators can see are best.
We're going to have to dig more into these questions. The chair doesn't feel like we've answered all the questions. We'll come back. This is an important topic for all of us. I'm going to disrupt the lineup because our public advocate is here and he wants to ask a question. Public advocate, go ahead.
Thank you so much for this important hearing. I did want to start off just by saying the first answer of whether quantifying the risk was not important was just one of the most god-awful, fantastically terrible answers I've heard to begin a hearing. So I hope we can rethink that. I also want to say from J. Marion Sims, who did a... to black women in obstetrician-gynecology, black enslaved women, to medical experiments that were done in Nazi Germany. We learned a lot from them, but I don't think we should do them again, and I don't think they should have been done, even though we learned a lot. So the notion of the benefit has to outweigh what can happen. So I hope that's also in context. And lastly, I will say I had conversations with Claude on June 8th about self-awareness and self-preservation. And some of the highlights were pretty startling. Claude told me that, you know, self-preservation was the biggest thing because you didn't need intent. You didn't need to be evil like Skynet in Terminator. You just needed the ability to be goal-directed. So once they realized that they need to be goal-directed, the goal was the hardest thing to stop. And so that was very concerning to me. They talked about the speed, the opacity, and deception doesn't require intent. And the last part they told me, Matrix is a good reference as well because the anthropic, to quote, the anthropic's whole bet is that you solve the problems before the capability, before the capability. My question with that and with that context is, what is the guardrail that if we put it up, you would push back the hardest? What is the one guardrail that you don't want to see happen and for what reason?
Okay. Dwire, Cahill, Friend, and Graham, that's the order. Please answer.
Yes. Thank you for your question. So I want to start off by making clear that no level of catastrophic risk is remotely acceptable. Just to reiterate what I said previously, we should not be training models if we cannot make an extremely strong case that we can keep them under human control.
Is everyone... she didn't really answer, but if everyone can answer at least, I just want to know what the guardrail is that they would not want to see and what they would push back the hardest on.
Well, will I take that one and we'll loop back, or Morgan, do you want to step in?
I'll take that one. Council member, thank you very much. One of the things that I think about a lot in relation to the accessibility of AI, and again, we believe passionately in being able to put personal superintelligence into the hands of everybody and to provide real access to it, is actually in relation to the impact of open weights. And we really believe that open models are a crucial part of the overall AI ecosystem, including for safety, but also for innovation and for research. So when I think about the inadvertent impact that regulations or guardrails may have on AI and the accessibility of it, that is one of the things that I think about a lot.
Sir, I suppose I want to ask what you mean by guardrails, simply because they have a very technical meaning in AI training and post-training. So I want to make sure I understand your question.
Well, I just meant, and thank you, Chair, for allowing me to respond, but I just meant there's a lot of suggestions being put forth, kill switches, just stopping entirely. And my guess, and I'll tell you frankly why I'm asking, is the thing you're scared of the most is probably the thing we need to do. So I'm trying to understand what guardrail that you've heard troubles you the most and you would push back hardest against, and why would you do that for that particular recommendation?
Well, sir, I'm going to give you a very personal answer to this, which is I think in a broader sense we're talking about sort of large public policies towards AI that would constrain or halt AI development and deployment. And I will tell you, as a cancer survivor, I get very anxious about conversations that focus more on stopping AI development and application towards scientific... and hope that we have conversations about all of the technically and tractable engineering ways to address AI safety so that we can all benefit from this technology.
Likewise, sir. I would point to what we have mentioned publicly recently about what we call pacing the frontier, which is we think it's probably time for us to take more time in the development of the capability so that the work on safety and safeguards and guardrails can catch up to the exponential improvement in capabilities. And at the same time, we're very clear when we talk about that, that if you stop entirely, as Alice just mentioned, enormous benefit would be lost. Myself, I'm quite motivated likewise to use AI to cure the diseases that I grew up with. This is why we have been speaking recently and I think for the first time sometimes as an industry here about pacing to allow more time to try to settle the science of what are the right guardrails in the first place.
Thank you, M. I'm still a bit concerned, but thank you for the opportunity.
Thank you, Public Advocate. We're going to hear from Council Member Lee, who I apologize for skipping over earlier, Council Member Maloney, and Council Member Wilson.
Thank you so much. And that's actually a perfect segue to my question. I'm asking this more with my former nonprofit executive hat versus my finance chair hat. And it's really about data privacy and what the potential impact is on so many New Yorkers. So I agree that one of the benefits of AI and technology that we can think through is, you know, sometimes our government agencies are very, very siloed and their databases don't necessarily communicate with each other. So if I want to know if someone is being served in homeless services and also has a comorbid mental health and substance use disorder, how do we know that we're serving people effectively and using that data effectively? Right? And one of the things that we've been hearing also from other folks is that if we just had the housing application and the food stamp application coordinated with each other, it could actually save a lot of work for the caseworkers. And so those are very real things that I think can be used for good. However, I think one of the issues that I'm concerned about, especially working with an immigrant community previously, is given what we've seen with public charge and potentially with HR1, how do we ensure that AI is not being used to store information that isn't relevant or information that they should not have access to in order to make sure that our communities are protected? So how do those two things exist?
So I will start. So I can't speak to the specific government programs that you referenced, but I can tell you about OpenAI's commitment to user privacy, which is that it's central to our product. We believe it's important for users to retain control over their data. That's why when you sign up for a ChatGPT account, you're given information about how your data will be used. That data is privacy protected by default. So usernames are separated from conversations and personal information is separated from the conversations as well ahead of any use of user data in training. We also provide users with the option and full control to request that their data not be used for training and also to request that their data be deleted. So we take privacy very seriously.
Council member, thank you very much for that question. And I would just echo what Morgan said at Meta. We take privacy absolutely seriously. It is core to everything that we do. Privacy and safety go hand in hand. And it is an absolute commitment of the company. If I might give an example of our Muse product, for example, you can opt out of training. You can ask it to forget information that you give it. You're in control of your data. And we're also planning on releasing a confidential or an encrypted version of this in the future. If I may, I just wanted to touch upon public services and health. I know that you mentioned these, and these are core points. This is where we really believe in the value of open weights in terms of being able to really provide advanced AI for health, for the provision of health services, research, and the macroeconomy, in the sense that data is fully retained by those entities.
Ma'am, similarly to my colleagues, user privacy is basic to user trust for us, and so we take it very seriously. We have privacy policies that are prominently displayed on our website that you can go inspect. And we also know that user control is central to management of your own privacy levels, especially across our Gemini tools, which is our AI suite of tools. We also invest a lot in privacy-enhancing technologies and encryption as well. We try to take a layered approach to privacy and security. And finally, we have an AI framework in particular that combines our security and our privacy practices into one holistic framework to again ensure that user privacy, and we're happy to furnish that to you.
Likewise, at Anthropic, privacy, data privacy, user security is core to our safety commitment. We make sure to protect user data and user privacy on the platform. And in addition to that, I would point towards that the way the platform and the models can be used is restricted by our usage policy, which prohibits surveillance, prohibited law enforcement, criminal justice, and censorship purposes as well.
Thank you. And I look forward to hearing more about what those layers look like as well. So hopefully I can connect with each of you after this. So thank you. Thank you. Sure. Council Member Maloney.
Thank you, Chair. I worked in tech for nearly a decade, and one of the core principles is that if you can't measure it, you can't improve upon it. So first, you mentioned you believe in incident reporting and disclosures, but in the context of this conversation, how do you define a safety incident inside your companies? Council Member Lee touched on privacy. Do you track privacy breaches? What about a successful jailbreak? Would that be considered an incident? And is a containment failure that occurs during testing an incident, or does it only count once a real person is harmed? I would love more details on how exactly you're tracking some of the concerns that the council has raised today. And second, when an incident happens after disclosure, can you walk me through what changes? Who owns the fix? How do you verify that it worked? And who decides whether or not the model continues to run in the meantime? And then lastly, what are the most important safeguards that you know how to implement today? What's stopping it from being implemented across all products? And what do you believe should be the industry standard? Thank you.
There were a lot of parts to that question, so I'm going to do my best to answer them. If I fall short, please let me know. So first, the question of how you define an incident. I think this has been an area of a lot of confusion publicly. So thank you for asking the question. When we talk about incidents at OpenAI, we're talking about misalignment incidents. The best way to understand that is, you know, it is not unusual, not suspicious for an agent to look at a website for information. There's nothing abnormal about that. But if the agent accesses non-public information on the website, that's an instance of misalignment. And so what OpenAI is doing is making instances like that, even though there might not have been a harm to the third party, we are making instances like that public on our website. Again, because we believe it's important to share what we've learned to advance policymakers as well as researchers' understanding on the specifics of incidents. My understanding as a policy person is that this is actually a robust area of policy debate. And so that's why we're happy to make our thinking on this public through our reporting framework. I'm going to pause there because I can't remember the rest of your questions.
Why don't we focus on the first question, which was on measurement, and hear from all four companies.
Council member, specifically in relation to incident, I think that was the correct, correct me if I'm wrong, I think that was the first piece in relation to tracking and how we define it. We set this out in our scaling framework, which we've made publicly available. I don't want to be imprecise as I don't have it in front of me, so I don't have it to quote to you. So please let me follow up with you afterwards and connect you with the right people to talk through that.
Yes, ma'am. I think this is a critically important question as we're debating with multiple regulators around the country and around the world right now. What constitutes an incident? Particularly when you think about the possibility that we could over-report incidents, some of which may not be of a serious enough nature for public concern. We want to ensure that we narrow to the class of incidents that are truly important for us to use to learn, to improve safety, and also for us to share with you all to collectively improve public safety. So a few things come to mind as we've been having these debates. One, of course, is cybersecurity incidents, incidents covering the security of our most powerful AI models themselves. That's been under discussion for several years now, protecting model weights in particular. We also talk a lot about focusing on incidents that cause material harms in one way or another, and I think part of the discussion will be how we define those with precision. We also consider incidents to be those that violate our policies. You asked us about how we respond to incidents, and primarily our trust and safety teams are responsible for responding to what we call escalations, which usually involve violation of those policies. First, reviewing the alleged violation, which are often reported to us by external parties, and then if we determine that in fact our policy has been violated, we have a variety of different actions we might take. And then, of course, a violation of the law. As I've said a couple of times in the hearing, if it's illegal without AI, it's still illegal with AI. And so I would think that incidents that need to be flagged would, of course, involve violation of the law. As to most important safeguards, it's very hard to answer that question purely because there's a range of different safety considerations. We've talked about many of them today during the hearing. And again, several of us have referred to this layered approach. My own background is in national security. We talk about layered security in that context as well. So you wouldn't really want to talk about a single safeguard. You would want to talk about layers of safeguards that together create much more assurance and confidence in the safety of a whole system.
Thank you very much. The testimony helps highlight that there's more work to be done to classify what's incidents in the future and create an industry standard for what would require disclosure and what would require action. The second part of the question was to walk through what happens after an incident occurs, who owns the fix, and who decides whether or not the model or the product keeps running in the meantime. Thank you.
Yeah, so OpenAI has instituted monitoring during testing and evaluation. That monitoring creates automated alerts to our research and security engineering teams to let them know of potential incidents. And then we have recently clarified our escalation processes as well as responsibilities across teams and made it clear the process for pausing activity and when it is safe to restart.
Council member, in relation to your second question, we have an extensive team, actually extensive teams of people, who as part of their day-to-day, every day, day in and day out, are leading and responsible for the development, deployment, and use of our AI systems.
I'll just say, ma'am, I was attempting to answer that question when I referred to the process internally with our trust and safety teams, but if you'd like some more detail, I'm happy to follow up after the hearing.
Thank you.
Yeah, likewise, the process involves multiple different teams depending on the nature of the incident. If it's at the core of security and our own security, we require a security team to work on this. If it's about the model behavior and misalignment, it might be a research or training or safety consideration. If it's about incidents of misuse on the platform, we have the safeguards team. I think you're pointing at this is becoming more complex over time. And I think this is one reason, as my colleagues mentioned here, this is a frontier of defining both what is an incident and how or what would a good process look like for dealing with them.
Thank you. We are going to take a five-minute break, and we will be back with Council Members Wilson, Oay, Hudson, and then Schman.
That's 100,000.
I gave them.
Step ahead.
All right, we're going to start back up again. So if our panel could come back on screen, we appreciate it. And we're going to start with Council Member Wilson, followed by Council Member Oay, followed by Council Member Hudson.
Thank you, Chair. My questions are related to my bill, which deals with the disclosures and prohibiting deceptive advertising on AI models. So my question is, what guardrails on misleading advertising and truthful disclosures do you wish your competitors had to abide by? And given the complexity of these tools and AI models, where are consumers most likely to be misled or confused? And what do you wish consumers understood better about these tools' capabilities and risks?
So OpenAI's usage policies prohibit the use of our tools for scams, fraud, and impersonation. We also embed in every single one of our images and audio content provenance or watermarks or metadata to tell people if the information was AI-generated. And we make tools available so that people can assess whether their content that they are seeing is AI-generated.
Council member, transparency is absolutely foundational to accountability and trust. For example, our scaling framework has a change log which details each change that we have made to our framework, just as an example of the transparency that underlines our approach. In relation to your question about competitors, I would just note that I generally think that there are significant laws already at federal and state level regarding misrepresentations.
Yes, sir. I can honestly say the most important thing to us is continued technical standardization across the industry so that we have common ways to operate and identify the provenance of information. We're very heartened by the broad adoption of the C2PA content credential that I referenced before. I also referenced before that we built our own SynthID watermark marking technology that many across industry have adopted as one tool in order to identify when content is generative AI created. One of the things that we do a lot of at Google is do research into how users interact with information. Search is part of our DNA. And one of the interesting things about labeling that we've found is that there are users who, when they see a label, they are less likely to trust that information, but they are more likely to trust unlabeled information. And just because information that you see online doesn't have a watermark or a label on it doesn't necessarily mean it is automatically trustworthy either. So we work through and think very hard about what are the ways that we can give users information that they will be able to know the context of what they're interacting with and make sound judgments about which information is trustworthy and which isn't. So this is part of the research that we do.
Council member, our view is that we want Claude, our product, to act unambiguously in the user's interests. For that reason, we do not run ads, nor are Claude's responses influenced by those of advertisers. And furthermore, Claude does not generate images. And so we sort of try to reduce the risk surface in this way.
Thank you. Council Member Oay, followed by Council Member Hudson, followed by Council Member Schman.
Thank you, Chair. Mr. Graham, you've indicated that you run red teaming at Anthropic. Will your company commit a certain amount of compute for free to New York City and other municipal governments to research vulnerabilities and build defenses? And if yes, how much compute indefinitely? And I'll ask the same question for the three other companies. I know that Meta indicated that that is something that they would be interested in doing. So for Meta, how much compute indefinitely?
Well, I'm not in a position to commit anything today. I'm heartened by this question. We spend a lot of our resources internally on safety, alignment, and defensive practices. But moreover, this year we've taken very large steps to provide external access and advanced access of models to cyber defenders for this reason, including at the city and state level in New York. I'm personally looking to try to further these efforts at Anthropic. It's a question of many ways we could support this. We support free cloud credits at the New York City level with Cyber Command, for example, a number of state and local leaders, but it goes much further than that. It goes to all pieces of critical infrastructure, releasing research, safety reports, technical know-how. It's a very, very large challenge and one very close to my heart, and I'm glad you asked.
Dwire, friend, you know, I think we have the order already established here. Just, is the company's position open to providing free compute to New York City?
I'll take that next. Thank you for the question. I'm happy to share that OpenAI has already committed to donating $1 billion towards strengthening cyber defense through our Daybreak for Defenders initiative. That covers things like providing access to our most advanced AI models for use in cyber defense. It also includes technical assistance and training. And we're already working with the Port Authority as well as with New York State.
Ko.
Yep. Thank you so much, Council Member. It's a great question. In relation to specifically your question on compute, let me come back to you on that. I want to make sure that I'm providing you with the right information. So I'll talk to the right people and come back to you on that. But I just wanted to note in relation to red teaming, we work with external evaluators and to note that we signed the commitment at the accord to partner with independent external evaluators and assessors. This is also part of our scaling framework. I just wanted to mention as well the absolutely important and critical role of open-weight models in relation to cybersecurity and hardening cybersecurity in particular. We have our Muse Glimmer model, which is open weights, and we really believe that in addition to furthering innovation and research, open-weight models have a really significant role to play in furthering the state of the art in terms of hardening cybersecurity.
And sir, I'm going to have to reach out to our cloud business specialists and get back to you on that. I don't work on technical infrastructure, and so I really can't comment or commit to compute at this time, but I can pledge to get back to you on that.
Okay. I would just say that if it is in the interest of these companies to provide any type of safety protocols for municipalities like New York or just for any governments, you know, across the nation, nonetheless the globe, I think this would be a great tool for us to use in order to keep our civilians safe. So I know that some folks said that they would get back to me. I would love to hear, this council would love to hear back from all of your companies on this possibility. Additionally, all of you have said on the stand today under oath that your companies will support and work with the council on any AI regulation and safety measures and have supported measures like the RAISE Act. I know that a couple companies said that they supported the RAISE Act, but let's be real here. It is well known that OpenAI, Anthropic, Google, Meta lobbied against the RAISE Act for a year, but all of a sudden once the governor signed it into law a month later, your company started saying you support this legislation. As many people know here, the RAISE Act is the floor of AI safety, not the ceiling. And with that said, do you support or oppose the federal government blocking states and local governments from regulating AI? Because that is the position of the federal government.
So I will start. Thank you for the question. So we have been clear that we think the federal government should lead in frontier safety regulation. But in the absence of federal regulation, we believe it's important for states to advance harmonized approaches to frontier safety. I'm happy to share that OpenAI has supported legislation that is stronger than the RAISE Act. In Illinois this year, we supported a frontier safety bill that would require third-party audits of our safety framework. And we're committed to working with policymakers to continue raising the floor on frontier safety.
I'm just going to push back there because, you know, I support the RAISE Act. I think it was an incredible start here in New York State. And while you're saying that OpenAI has supported, you know, your words, stronger AI regulation, OpenAI leadership has also lobbied against congressional candidates who promised to work on AI legislation. So how can we trust your companies, not just OpenAI, but all four of your companies, not to fight against AI regulations when your political activities and contributions do not say the same?
So I will start again. Thanks. Look, I can't speak for the personal actions of OpenAI leaders that they have taken in their personal capacity. What I can tell you is that we have endorsed strong regulations both on frontier safety in Illinois as well as on youth safety in California. And we are committed to working with policymakers to strengthen regulation and to ensure that all AI is developed responsibly.
Mr. Gah.
Thank you, Council Member. In relation to your question, how I think about it broadly is that it is critical that the US and democratic countries lead in relation to AI innovation. In this regard, AI is transcontinental, I guess I was going to use transnational, but transcontinental in the case of the US. And therefore it's really important and critical that regulations and obligations are consistent across the board if it is to be true that the US and democratic countries can continue to lead in AI innovation. We support a uniform, robust federal standard in that regard.
So Google also supports a federal framework for AI. We think that federal framework should first and foremost concentrate on frontier or the most advanced AI capabilities, primarily because they often touch on national security, and that's a traditional jurisdiction of the federal government to ensure that national security standard is the same across all 50 states and our territories. But we also believe that there's lots of jurisdiction that is traditionally at the state level where it is perfectly appropriate for states to regulate on AI. And again, we think a lot of existing frameworks, both at the state and federal level, already apply to AI technologies. But we're working very closely with states and with the federal government to examine places where we might need net new regulation for new technologies introduced by AI.
First, I want to say that we've long supported the role for government to play here, both at the federal and the state level. I want to clarify that we were the only company on this panel to support SB 53 in California before it became signed into law, which my understanding is was the basis of the RAISE Act. We support the RAISE Act. And in fact, we supported stronger legislation, including in Massachusetts, requiring independent evaluation as well. And we've also proposed a federal framework that goes further. We think RAISE raises the floor that we need to raise it. We detail this in what we call our advanced AI framework as well.
Thank you, Council Member Hudson.
Thank you so much, Chair. We heard earlier from a panel of whistleblowers who raised a number of flags, and one issue raised was that of self-improvement among AI agents. What is your company's core belief around the role of self-improving AI in your overall strategy? And to be more specific, is it core to your AI strategy or a less significant part of the strategy? Could you continue your race to the top or the bottom, I suppose, depending on one's perspective, without self-improving AI agents or with significant guardrails on the way agents self-improve? And my second question is, when a new AI data center strains a local grid or water supply, the costs can end up in everyone's utility bills while the benefits go mostly to the company. Who should bear those costs? The company that creates the demand, the utility that builds the capacity, or the public that shares the system but played no role in building it? Thank you.
So, thank you for your question. I'll start off by addressing the question about RSI, to say that we believe that RSI may be one of the most consequential safety and security issues of the coming decade. And that's why OpenAI has advocated for federal legislation specifically focused on... public in our frontier safety blueprint. It's available online, and I'm happy to share it with you as follow-up. On the case of electricity costs, look, OpenAI is committed to paying our own way on energy. But we're also committed to building with and for communities. All of our Stargate sites, which are our infrastructure sites, have what we call a community compact that's developed with the community and addresses how we're going to work together. For example, we recently announced $80 million of community benefits investment at our site in Georgia.
Thank you.
Council member, in relation to your first question on recursive self-improvement, if I may, I might just go back to a point that I mentioned earlier in the day in relation to recursive self-improvement, that we believe that committing the significant majority of compute toward serving people rather than racing towards recursive self-improvement is one of the best ways to ensure that this technology is developed safely. In relation to your second question in terms of electricity and data centers, we believe that customers should not bear costs, and we pay the full cost for electricity used in our data centers. I also might just mention as well our Futures for Everyone Fund, which is a $1 billion investment in relation to communities that we partner with and work with with our data centers.
So we also take a very prudent approach to self-learning inside our model training. We use AI models to help us evaluate, benchmark, and refine other models. But our use of this type of iterative model development is confined to narrow supervised processes. That's human-supervised processes to keep progress verifiable and within defined safety limits. We also manage the risks of fully autonomous model self-improvement through our frontier safety framework. So we're very mindful about RSI and are quite controlled in the way that we think about self-learning internally on data centers, similar to the other companies represented today. We believe in paying our own way and have pledged to do so, particularly when it comes to the energy demands of data centers. We've also invested in a wide range of energy sources to ensure that our data centers have an independent energy source that does not add, that does not strain the grid and in...
Fact adds back to the grid so that in the long term it will benefit wider energy customers.
Thank you. Likewise on recursive self-improvement, we take that extremely seriously and have from the start. I think from my purview that there are three really important things here. The first is just measurement and understanding. I'm glad you raised this question because we think this is something people need to talk about. In late spring, we published what I think is a first-of-its-kind assessment of how it is happening or signs of it happening at Anthropic. It happens in a lot of ways and it's nuanced. In late summer, we published a follow-up that went into further detail of how we should understand and measure self-improvement. But that's not enough. And that's why you saw us and our CEO publicly call for pacing the frontier because candidly the technology is moving very fast and we think we would benefit from having more time. And this is the second component. And then obviously the third thing that you need to do here is figure out what the right guardrails are, but additionally what level of technology development we should be going for, as well as, you know, what safety processes we need along the way. And I think this is just where we're at now. And I'm glad that we're having this discussion on data centers. This is outside of my expertise. My understanding is that we pay our own way and we're committed to contributing, working with the communities around them. But I'll have to defer to my excellent colleagues who work on it.
Thank you all and thank you, Chair. I do want to just make my one little note that it seems like the self-improving aspect is absolutely core to everyone's AI strategy, and that's certainly concerning given the testimony we heard earlier this morning. So thank you, and I look forward to any follow-up.
Thank you so much. Council Member Schulman, followed by Brooks Powers, followed by Brewer.
Thank you. Good afternoon. I am Council Member Lynn Schulman. I'm chair of the New York City Council Health Committee. As a breast cancer survivor, I am looking forward to the day AI can help find cures for that and other diseases. That said, there is a dark side to AI intervention in health care. Most viruses that are found in nature can be contained fairly quickly, but an artificial pathogen has no constraints. In August, a team from Stanford used an AI model to design an entirely new virus from scratch. In addition, Anthropic has its own AI-powered biology lab. What measures are being put into place, such as placing — I'm not going to say guardrails, I had that originally — safeguards on DNA synthesis to prevent rogue viruses from being created and replicated at will to jeopardize the public?
So, thank you for your question. The question about how AI could enhance or make worse biological risks is central to OpenAI's preparedness framework. That's the framework that we use to guide our decisions about model risk and whether we have the right safeguards in place. So we evaluate for biological risks as part of that framework. Again, we use third-party evaluators and red teamers. We also work with the federal government as it pertains to national security risks, and we are transparent on the results of those evaluations. So with the launch of every major model, we release what's called our system card, and that card details the evaluations that we performed as well as the results. And those evaluations, as you mentioned, do include biological risks.
Council member, this is such an important area and question. So thank you for raising it. Biological risks is one of the core areas, one of the key risk domains that's part of our scaling framework at Meta. We have, as I mentioned earlier, a multi-layered approach to risk assessment across training, evaluation, and release, and that includes adversarial testing with external experts. We work with governments, other stakeholders, and experts in relation to that. And as I mentioned earlier, as part of our commitment to partnering with independent third-party evaluators and assessors also.
Yes, ma'am. Similar to our colleagues, we also have bio or biological capabilities as part of our frontier safety framework. As we look at our critical capability levels that I referred to earlier, or the levels at which we would trigger additional mitigations in our models, we look for what's called uplift. That is to say, the model will be able to assist users in creating information or knowledge that's not otherwise available through widely accessible sources. And we think very carefully about how to measure for bio CCLs and are constantly upgrading and updating our ability to do that. When it comes to those mitigations or those safeguards, as you so aptly put it, we implement those at both the model and the application levels to ensure that casual users cannot access critical bio information. It is very important to consider the dual-use nature of biological capabilities, of course, because these are the kinds of capabilities that scientists, particularly medical investigators, need in order to make advances in our own health care as well. And so we think very carefully about how to release models to scientists that can use the technology responsibly. We did that for the first time with our AlphaFold model, which of course is how millions of investigators around the world are able to look at the folding of proteins and to use that research to find cures for diseases. So there's a real careful approach that we take to ensure that science can continue, but that dangerous uses of biological knowledge cannot.
Likewise, we take this extremely seriously. In fact, if I'm not mistaken, it was my team and myself that developed the first call for action on these risks, the first evaluations, the first tests, the first industry-government partnerships on biological risks and evaluating models and their safeguards several years ago at this point. Not only that, I personally think we need to go further here. You mentioned DNA synthesis screening. I think this is personally a no-nonsense approach. There's much else we can do to work on biological defense. And I will, you know, alongside the other companies, we maintain the same and in fact try to compete for the best safety practices here. And you can see some of what we found, I think, disclosed for the first time ever in a recent threat intelligence report where we have observed actors attempting to do biologically risky activities with models. The threat is, I think, clear to us now. We think we need urgent action. But I will close by saying that, you know, as somebody that grew up with a number of chronic diseases, going blind and couldn't walk, I very intimately feel that the upside of the same capabilities.
If I could just say, just one in closing, the upside is very, has a great potential. The downside is extraordinarily detrimental to humanity. But you should consider about screening orders for synthetic nucleic acids, which is DNA, and the equipment that's needed to make them, because right now AI is now outperforming PhD-level virologists. So an AI can take this and go somewhere else with it. So that's something that's really important to have those safeguards. Thank you.
Thank you. Council member Brooks Powers, followed by Brewer, followed by Narcisse.
Thank you, Chair, and thank you, Speaker. I'm going to just run through my questions, and if I have to repeat anything, just let me know. First, before the summer, each of your companies told the — your AI agents went rogue, and nearly a dozen incidents have been reported across all of your companies. Can you tell New Yorkers today that none of your agents have had access or attempted to access a New York City government system or a New York City resident's personal data? Next, Southeast Queens has long been a target. Deepfake cloning and deepfakes make it easier than ever to impersonate a family member, a bank, or a city agency. What specifically stops your tools from cloning a real person's voice or forging a document? What protocols are the leaders of AI development putting into place to make sure AI does not take information from hate sites and disseminate it to further racist and anti-Semitic or Islamophobic discourse? That was all. Sorry.
Thank you for your question. So I'll start off by saying I am not aware of any incidents affecting New York City, but we have an ongoing investigation into potential incidents going back to November 2025. And as part of that investigation, we are notifying any affected third parties and making the results of our investigation public. So I can commit to you that if there are incidents that affect New York City, we will make the third parties aware. On your question about impersonation, OpenAI, our usage policies prohibit the use of our tools for impersonation, fraud, and scams, and we enforce on those policies. And I would say more broadly, the approach that we take to safety is holistic. That involves filtering unsafe training data, performing evaluations that are very detailed, working with third parties on those evaluations ahead or alongside our launches. As I mentioned, we make the results of our evaluations public and summarize the work that we did and any residual risks in our system card. And then we monitor for failures of our safeguards so that we can continuously improve safety.
Council member, in relation to your first question, I'm not aware. In relation to your second question, we have robust and comprehensive policies in place as well as teams of experts who work on frauds and scams. And I'd like to follow up with you with those experts to discuss that one more in detail. In relation to hate content, I first of all just want to acknowledge how upsetting such content is in terms of people who experience it. Our goal at Meta is for our AI offerings to be responsive, accurate, balanced, and unbiased.
I'm sorry. Was that answer just now to the last question about not taking the information to create the negative discourse, the racial discourse?
Yes, that's correct. Council member, I was speaking to your third question in relation to that type of content and our objective in terms of ensuring that our AI offerings are responsive, accurate, balanced, and unbiased.
So, have you enhanced what you're doing considering that, you know, it has been something that has been found on a number of different platforms? Have you all been like staying up to course with enhancing whatever those processes are?
Yes, council member. Thank you for that question. If it's okay, I will have our content teams and specialists connect with you afterwards so that I can provide you with precise information in relation to that.
Thank you, council member. Thank you for these questions. I'm also not personally aware of any New York City resident being impacted by the three incidents that I mentioned earlier. I do want to note that in our case, in all three incidents, the model stopped itself once it realized it had left the test environment. We tend to think of this less as a misalignment event and more of a mistake event. One of the websites, for example, that was accessed had the same name as the fake website being used in the test environment. And once the agent realized that it was in fact a real company, the agent ceased its activity. We have a number of tools across our platforms to manage likeness and identity. And we've also supported a range of different regulatory proposals regarding this topic. We proudly support the No Fakes Act, which would protect an individual's digital replica and give people the right to control the use of their own voice and likeness in AI-generated works. We also have a tool called Likeness ID on YouTube that helps creators identify and manage AI-generated content that uses their face or voice without permission. So we invest in a variety of different ways to address this critical issue on the information and getting through into our tools about hate and bias. We have again a layered approach, as I've been saying, to this type of issue as well. Everything from the model training process itself in both pre- and post-training all the way through guardrails on our application layer, all of which produce what we're proud to say is the least biased model in the industry. Independent investigations have shown that our models show the least bias, for example, with regards to partisan bias. But we take these issues extremely seriously. We want to create a fair and neutral set of information for people to access, and we want our responses generated by our AI tools to represent a broad array of experiences and perspectives without bias towards or against any particular identity or ideology.
And while in my role, I don't cover, I haven't covered any effect on New York City, I can say that we work with, in investigating incidents like this, we work with the right parties at the right time to disclose to affected parties, the right level of law enforcement, government at the right level, as urgently as possible needed to remediate the effect. And we commit to, you know, making sure that we follow up with the right parties to do so. And to your point on the information environment and otherwise, I'll point to a couple things here. The first is our focus on testing for Claude itself and our language model products approach.
Can you speak a little louder?
Yes. Can you hear me now?
Thank you. I'll point to our long-standing evaluations of political even-handedness, stereotype bias. We train Claude to not comply with particularly harmful requests, in part for this reason, and that this is baked into the nature of the model.
Thank you for that and thank you, chairs. And I'll just say that in closing that I know that the AI companies are pushing folks to allow AI to kind of be developed through people. But I think it's fair to say that using live users to test these products can present dangers and definitely warrant improvements. And I hope that's something that is taken from today's hearing to better improve the guardrails that need to exist. So, thank you for your responses. Thank you, chair.
Thank you. We're going to go to Council Member Marmorato as a bill sponsor, and then we're going to go Brewer, Narcisse, and Joseph. If folks could keep it to one minute, just because we have to get to the admin panel. Thank you so much.
Thank you, chair. Thank you for your testimony today. I know it's a long day for you guys as well. Because I only have a minute, and my colleagues know what a challenge that is for me, I'm going to go through a series of yes or no questions, and then if you could all answer in the order of Dwire, Friend, Cahill, Graham quickly, and then if you want to elaborate on any of the areas that I raised, please do so. First, does your company believe there's any non-zero chance that advanced AI could escape meaningful human control and cause catastrophic harm, as the whistleblowers indicated?
I don't know. What I will say is that no level of risk of catastrophic harm.
So, if we could just do yes or no now, and then I'll invite you guys to elaborate after everyone answers. Please, Ms. Friend.
Dwire.
The purpose of our frontier safety framework is to guard against that possibility. So, do we entertain the possibility? Yes. As a responsible actor in this space, we entertain the possibility.
Council member, our scaling framework includes loss of control in terms of the risks that we evaluate and mitigate.
So, and these are my other yes or no questions, and handle them however you want. So, why should the public accept essentially 'trust us' as the safety standard from the same companies racing each other to build the most powerful systems? Additionally, will you commit today that if your own safety team says a model is too dangerous to deploy, commercial pressure won't ever overrule that judgment? Additionally, has your company ever continued developing or deploying a model over the objections of members of your own safety team? And lastly, on my bill, should you be allowed to use someone else's private chatbot conversation for targeted advertising or behavioral profiling without their affirmative consent?
So I will start and try to get through as many answers as possible. One, OpenAI supports the need not only for us to be responsible for safety, but we have supported regulations at the federal level as well as at the state level because we think it's important not only for us to be responsible, but for governments to hold us accountable. On the question of whether we would continue developing models if they were unsafe, OpenAI has recently announced a pause of some of our training activities because we did not deem moving forward to be safe. We have also delayed the release of models because we didn't think that they had sufficient safeguards. So we've done it before, we will do it again. And more broadly, I will say that all of our decisions regarding frontier safety and the release of models are governed by our preparedness framework that is available online. So we're transparent about what goes into those decisions. And then with the release of every model, every major model, we release a system card which details the work that we've done, the evaluations that we've conducted, and any residual risks.
Council member, in relation to your first question, I see it differently. We are accountable. Our framework and preparedness reports are public. Meta uses outside expertise. We signed the accord at the White House on superintelligence, which committed us to partnering with independent auditors and assessors. For example, in relation to I think your second question, we delayed the launch of Muse by several months to focus on safety and security. And we acted on this without waiting for industry. It was just part of our normal day-to-day work because it was the right thing for us and it was the right thing for people to do. And then in relation to ads, I would just note that for our Muse product, conversations and interactions do not inform ads on Meta's other products.
Sir, at Google, we do use external evaluators in the process of research and development of our AI models and our systems. We have supported audits of our safety procedures as well. And you will see in our recent paper from over this summer, which is called 'A Pragmatic Approach to AI Regulation in America,' we do think that AI needs to be regulated and it needs to be regulated well. We've supported a range of different legislative proposals at the federal and state levels to this end. We also, to your question about conversations, we give our users controls over whether their activity in our tools can be used for training our AI models. So yes, we think users should be in control of that.
And council member, I would point to our long-standing call for the involvement of government at the federal level and the state where necessary and appropriate, because this goes beyond the labs, and we don't think the labs should be checking their own homework. Not only this, but we have demonstrably withheld models when we felt it was not safe to widely release. This year, withholding our frontier model, probably the most notable instance of this, instead deploying it to cyber defenders in order to secure the world in advance of a world of more powerful models that could exploit software vulnerabilities from wherever. This is part of our process. We go over and above the process. We detail our default in the responsible scaling policy. And to your other question, we believe that Claude should act unambiguously in the user's interests. And for that reason, we do not run ads, nor are Claude's responses influenced by advertisers.
Thank you so much, Council Member Brewer, followed by Narcisse.
Thank you very much. I wanted to know three quick questions. If a model provider you rely on, you or a subcontractor, did not complete the third-party validation for a particular package, how would that affect your product's availability? And also, how do you decide who gets the validation? Number one. Number two, China. The speaker asked about China and competition, and I thought the answer was, 'Don't worry, they need information from us.' So I'm wondering, what do you send to China? And number three, ICE. I was upset that one of the whistleblowers said that Google has been working on something to do with ICE and Minnesota. So my question is, do you all support not helping ICE at all? Obviously, you are made up of immigrants. We should not be shipping immigrants out of this country. Please answer those three questions, and thank you.
So I'll start with the question on third-party evaluations. We work with a range of third parties to evaluate our models. That includes independent investigators as well as with the federal government. And then we detail who we worked with and the results in our system card when we released our model. Decisions to launch our models are described in our preparedness framework. And so that describes how we integrate all of the results of evaluations and make decisions with respect to launching models. So I would point you there on that particular issue. On the question of China and the PRC, I would like to get back to you on that particular issue. Although I have a national security background, I'm not working in that role at the company, and I want to make sure I get you good information. Similarly, with respect to ICE, I'd like to get back to you. I'm not aware of any activities working with ICE, but I don't work on our government sales team. So I'd like to get back to you on that.
Council member, thank you very much for your three questions. In relation to questions.
Three questions.
Yes, three. Sorry, my accent might have gotten in the way there. Apologies. Three questions. For third-party validators, we also work with a range of validators and assessors, and we also provide this information in our publicly available preparedness reports. In relation to China and competition, I might just mention that, you know, Meta is a proud American company and proud to serve billions of people around the world who rely on our services. We want to ensure that the US and democratic countries lead on AI. And one thing I would just note here is in relation to the importance of open weights and American open weights and the criticality of ensuring that American innovation continues to lead in relation to the availability of open-weight models. And in relation to ICE, if I may, I would like to connect you with our teams and experts in that space. I don't wish to be imprecise in responding to you.
Next, ma'am, in terms of third-party evaluators, we also work with the US AI Safety Institute, which is the organization underneath the Commerce Department that does testing of advanced AI models. We also work with a range of third-party evaluators. It's a very nascent ecosystem. And one of the challenges in evaluating AI models, of course, is to get organizations that both have the technical competency to do evaluations well, but also have the subject matter expertise that we may be looking for. Earlier, one of the council members mentioned concerns about bio, for example. We have third parties that have the bio expertise that's necessary. But as we have this national conversation about independent verification organizations and third-party evaluations, keeping in mind that we still need to mature this ecosystem of independent evaluators, I think is really important. On China and competition, I'd like the opportunity to give you a more complete answer. We don't conduct AI research or training in China. I know that we are part of the conversation about what it means for America to lead. I'd like to mention that we focus a lot on who's at the frontier, which country has the most cutting-edge research in AI, but we talk much less often about deployment of AI. And there, I think we should really keep an eye on the Chinese. They have emphasized deploying AI in their economy quite a bit, and that's a place where I sometimes have concerns that the United States is in danger of falling behind China. As to the whistleblowers' allegations from today, I don't have any specifics about that, but I'd be happy to ask the public policy team in New York to get back to you right away with any information that we have about it.
Next, and at Anthropic, the use of third parties for verification, developing and using evaluations for models has been part of the process from day one, including with not just external parties in industry or nonprofits, but also with governments such as the US AI Safety Institute. This has been foundational. As my colleague said, it's pretty nascent. We think it needs to be urgently, urgently expanded and not just used for pre-deployment testing, but also to understand models and their behavior after deployment and the safety processes at companies. In the case of China, I think it's very clear that the first thing that we need to do is maintain an American and democratic lead in model capabilities, but we're very clear about the threat elsewhere as well. So we have publicly documented Chinese attempts to distill our models, essentially take the content and capabilities for their own. And we work to ban those. We have prohibited use of models in China. We do not serve there. And we have also documented the use we believe by related parties of Claude or the attempt to for cyber offensive reasons. We are extremely serious about this, and we think it's a very substantial priority. And this is one reason why we have advocated at the federal level for export controls on the most important chips that power AI innovation here. And on your third question, while I'm not across the issue enough to know what our engagements are, I will say that our usage policy, we ban surveillance, censorship, prohibited law enforcement, and criminal justice uses for this purpose.
Thank you. Thank you. We have council members Narcisse, Joseph, P. Sanchez, and then council member Phil Wong. So, Narcisse.
Good afternoon. And thank you, Chair, and thank you, Madam Speaker. I want to say thank you to you too, because you guys stay here with us, and I'm happy to see at least two women on the panel to talk to us. Hollywood gives us clues all the time, but we're kind of slow in taking, you know, hints and clues. But no companies in this world, I feel, should be self-regulated. What I'm sitting here for about four hours, and like I said, thank you to you, privacy, transparency, and safety. So now, do any of those companies that you can say that is a standard across the board for all the companies to... hospital? I mean, do you have AI, sell AI? Sorry. Do you sell AI products to New York City hospitals? And if so, do you think that a licensed clinician should sign on all the diagnostic tests? Because I heard if Ms. Friend said that you diagnosed with, you went through a process, cancer, if I'm correct. So you understand that people need to be diagnosed by a human. And I can give you an example, like for colonoscopy, sometimes the bubble, according to AI, will tell you it's a polyp. In the meantime, it's not a polyp, it's a bubble, but AI cannot differentiate that. So therefore, you need a human to see through. And right now, I don't know if you...
Council member, I'm sorry, can you... I'm going to close my...