Anastasios Angelopoulos
0:00 ↗
I believe that we're going to have at least one massive multi-hundred billion, if not trillion-dollar, American company focused on American-first open source. What happened is that Kimi actually beat all American models, including Fable, in some subset of tasks. That doesn't mean that they're not distilling.
Anastasios is the founder and CEO of Arena. It allows you to vote on the best models. It's an unbelievable model evaluator. And this turned out to be one of the most fun shows I have done literally in recent memory.
Anastasios Angelopoulos
0:27 ↗
The idea that we should have a central government body that tells us when it's time to release a new product versus not is crazy to me. This is going to be so insane what happens with like the cyber attacks. There's at least 75 AI labs for sure. Like two-thirds of those are going to be worth nothing. Ready to go.
Anastasios, this is going to be a lot of fun for me because I'm dumb as rocks and you're going to teach me a whole load of stuff today. So, thank you so much for joining me, dude.
Anastasios Angelopoulos
1:07 ↗
Oh, no. Thank you for having me.
Dude, I told you I used this as a chance to catch up with old friends. So it was wonderful stalking you for the last few days. I just want to start for anyone that doesn't know. Can you explain to me very succinctly and easily what is Arena and why is it important and gaining notoriety today?
Anastasios Angelopoulos
1:25 ↗
Well, Arena is the platform for measuring AI performance in the real world. So what that means is that we're not using static benchmarks. We're not using some random data set that somebody collected, but rather what happens when you put AI in the hands of real people. And in so doing, we're measuring the objective reality of how AI affects humanity. Whether it's factual, whether it's steerable, whether humans prefer it or despise it, whether it's hallucinating, whether there's errors, whether people are getting their actual jobs done with AI in reality. And then we're helping labs improve their models. We're helping the ecosystem understand the performance of different, you know, different AIs and keep track of all the amazing breaking news, all the new models, multiple models being released every week. So that's sort of the story of Arena. We're the central evaluation platform of AI.
Well, that was incredibly succinct. Thank you. Normally, people take about four hours after I ask for a succinct description. When you look at the models that you have on Arena, the sheer number of them bluntly, I just am faced with the one question. Holy... Is this like the true commoditization of models? Are they just a complete utility layer at this point?
Anastasios Angelopoulos
2:29 ↗
Well, I think that there's the big question around this has started to arise because of open source models. So, I think if you were to only look at the closed source models, you would say there's acceleration, but it hasn't quite commoditized yet because that layer is still owned by a pretty small group of companies. It would be an oligopoly if we only had the closed source models. But what seems to be happening is that the open source models, especially from China, have really rapidly improved. And for the first time ever, we saw a couple weeks ago that Kimi K3 actually beat the best closed source American models on a, you know, pretty important subset of tasks. For example, front-end coding like web development, which a huge fraction of developers are web developers.
Dude, can I ask how big a moment was that? Because it's like a, I'm going to butcher it, but you know I'm a podcaster so I can get away with it. You're a PhD master, you can't. It's like a 27 trillion parameter model. It's pretty clunky. This is not an agile model. And actually, you know, I was with Jason Leman yesterday from SAS who's as AI-pilled as they can be and he's like, honestly, it's not better than the others. How big a moment is Kimi?
Anastasios Angelopoulos
3:45 ↗
No, it was a pretty big moment. It was a pretty big moment. And the reason I'd say it was a big moment is because it violates a narrative that has been persistent in the United States, which is that the Chinese are just distilling American models. And that's the only way that they're able to, you know, keep up. When really what happened is that Kimi actually beat all American models, including Fable, in some subset of tasks. That doesn't mean that they're not distilling. They may still be using distillation as a substep in their training procedure, but it does mean that distillation is only part of the story and that there's something that those labs are doing above and beyond distillation that's bringing the performance up above what the American labs are currently doing. And so that narrative violation has been hugely important to the way that people view the ecosystem, both from the scientific dominance of Americans and the American sort of hegemony, which of course Americans love hegemony, to the economics of the whole thing. And to your point, are these things, are these models a commodity or not?
When we see, you know, when we look at your OpenRouters of the world, you know, the top five models are all open source Chinese models. When we see the proliferation of Chinese models today, does that cannibalize the closed frontier model business meaningfully?
Anastasios Angelopoulos
5:10 ↗
Well, I think that you need to think about the incentives and economics behind it. So first thing I'll say is that the OpenRouter metrics are not truly reflective of reality, and that's because the business model of OpenRouter is to charge like a fraction, like a fee on top of every token. And so what happens is that people don't use OpenRouter for proprietary models. People are using OpenRouter primarily for open source models where they need the failover and all the value-added services that OpenRouter provides. If you look at the whole space of all inference, most of it is still being consumed on first-party APIs and on proprietary models. That's why Anthropic revenue has been just a total hockey stick. It's not, you know, it's not like they're being completely cannibalized right now by Chinese open source models. These models are still only a small fraction of the total inference spend in the world. That said, think about what's happening in the future. Enterprises are going to want to own their own intelligence. They're going to want so-called AI sovereignty, which is a fancy word for meaning that you own your whole supply chain of AI. That means you can take an open source model and you can fine-tune it on your own company's data and own your stack end to end, basically outside of the compute hosting. And so then you should be able to run it within your own company. People are going to care about sovereignty. People are going to care about cost. People are going to care about self-improving, and they're not necessarily going to want to give their data to an external third-party service that might even be competing with them one day.
So, do you believe that is the future? We had Lynn on from Fireworks and she was like, the specialized intelligence will be the future. Companies will have their own fine-tuned specialized models with their own company data and the performance will be better and that is what will happen. Do you think that's right or is that actually just a small subset of very advanced Silicon Valley companies and the known yogurts and every normal company would just use Frontier whatever?
Anastasios Angelopoulos
7:10 ↗
Well, I'll say it like this. I think the business incentives make this inevitable and the reason is because businesses are going to need a way of keeping a moat. In the age of AI, software is no longer really a moat because it can be produced instantaneously, right? Or let's project out five years. That's what's going to happen. And so what moats exist? Network effects exist and data moats exist. And if you can take your data moat and turn it into a self-improving product, that is a way for businesses to remain sustainable in the age of AI. Let's say I'm a business like a Coca-Cola, I'm a Cisco, I have a lot, a lot of users. I might not be necessarily at the frontier of the AI technology of the world, but I do have this massive corpus of data that I can use in order to, you know, beat my competition. So, what should I do? I should be trying to take advantage of my data as much as I possibly can to accelerate my business and stave off competitors.
Do you think they will really use open source Chinese models to do that?
Anastasios Angelopoulos
8:15 ↗
That's a great question. I think not. I think that the Chinese models will be potentially part of the story for now. But that given the regulatory environment in the US, it's probably more likely in the long run that we see a great American open source competitor arise. And this is why I've been a strong proponent, for example, of Thinking Machines. I believe that we're going to have at least one massive, you know, multi-hundred billion if not trillion-dollar American company focused on American-first open source.
Why have we not so far? I really hope so too, by the way. I completely agree. Would love to see that. But why haven't we, why has the US open community lagged behind so meaningfully?
Anastasios Angelopoulos
8:58 ↗
Well, frankly, I think it's a business model question. You know, I think that people have not really figured out up until this point what the business model is for open source and now I think people are wising up to it. There's a few different ways of going about it. One way of doing it is to say, I'm going to do a rev share. I'm going to take this open source model. I'm going to allow inference providers like a Fireworks or Together, whatever, to deploy this model and then if they get to over X dollars in revenue, I'm going to ask to do a revenue share. And that is one way of building a sustainable company off of open source. You basically share in the compute revenue. Another way of doing it, which is I think the more Mistral, Thinking Machines type of strategy, is to take the open source model and then use it as a lead generation tool for companies to build on top of that and then come to you and say, can you help us fine-tune? Can you help us with our AI strategy? And then you do the forward-deployed engineer and that is actually a huge market because if you think about it, one of the biggest markets over the next 10 years is going to be AI modernization going into every business in the world and then helping them retool in the face of AI, take advantage of their data, restructure their data, you know, figure out how to use these models, integrating them into workflows, teaching, you know, the employees of the company how to use them. It's going to be massive, massive, massive. And that is another way for them to become multi-hundred billion or trillion companies.
Is that not what the frontier model providers are doing though anyway? When you look at what OpenAI have said about their kind of FDE approach, Anthropic, I get you on Mistral and they've done a great job in doing that, but the frontier model providers, Microsoft is even doing an FDE model. Like, no offense, that's not going to be unique to open source.
No, I don't think that FDE is completely unique, but I do think the combination of FDE plus open American model may be a more sustainable model for the future of American or even Western businesses because they might not want to be building on top of external third-party services. They might want to be cutting those out for both cost reasons and for sovereignty reasons. And then the open source stuff, they can own it completely. They can continually fine-tune it within their companies and they can feel more secure in the fact that they're spending their money wisely and don't have supply chain risk.
So, how should we evaluate that? There's like thousands of AI labs. You know, you said the AI labs. You said Thinking Machines there. Again, I'm dumb as rocks. I say it very clearly to my...
No, you're not. You're a PhD and Annie told me you were smart. No rocks. It's just two rocks having a conversation.
I love it. Yeah, exactly.
That's the point, man. Come on. What do you want? Intelligent conversation?
Whatever. No. My point was you said about Thinking Machines. Like, my question to you is on the back of that, okay, great, I'm with you, but they now have two co-founders left. I mean, Lilian Weng left yesterday. Again, not throwing shade, not particularly picking on them, but the transience of teams has never been greater.
Yeah, team, it's hard. Retention is tough. I mean, being a co-founder of a company is also tough. It sounds like she left for some health reasons. So, it's unclear whether that has to do with the company momentum, which seems to be strong at this point. But, you know, I do think that Inkling is definitely a v0 model. You know, from what I know about Thinking Machines, they had a big restructuring like six months ago team-wise and then they kind of restarted everything and Inkling came out of that. So realistically, at least the most generous take towards Thinking Machines is that they've only been working on this model for six months and within that time they've become the number one American open source model. But then the less generous take would be the company's existed for a year and a half and then they've come up with, yes, the number one American open source, but there's nine Chinese models on top of them because they're number 10 open source overall. At least if you look at Arena data, if you go to our leaderboards today, that's the state of the world. But hopefully what happens with Thinking Machines is that they continue to release more and more models, you know, larger models, and they continue to build on their momentum.
I actually had a Chinese researcher friend of mine message me after one of our recent shows and said, 'You just don't get it. You missed the point of why we're ahead. We just work so much harder.' And we have support from policy, regulation, government subsidies that you don't have. We have all of these tailwinds that you... Do you not, do you agree with that?
I mean, I think they have tailwinds, they have headwinds. So I don't think it's so sanguine for that. I think that's like a little bit of an overstatement of the differences. One tailwind that we have is we have the best chip ecosystem in the world, so they're way hardware constrained over there and they've been trying to like black market import chips because of this. And you see this in the news, right? The Information just reported on...
Do you think that severely impacts their ability? Again, I'm naive. That severely impacts their ability or we're actually just fostering an ecosystem where they're going to learn to build it really fast because they don't have access to it?
Well, I think it may be hindering them now, but I think it's a good question as to what's going to happen in the future because they are really good at building hardware and the downside of export control is that it can incentivize them to build their own ecosystem and then what do we do, you know? So the hope is that we keep Nvidia ahead of the game so that we can retain the advantage that we have in TSMC's to the world and our whole ecosystem is absolutely a national security necessity. So we should have the government, you know, really protecting it and growing it as well as new companies that are innovating, you know, Etch just came out as an example within the United States, to continue to build on our lead there.
They're awesome. Love Gavin and team. Totally agree. Can I ask you just in terms of the export control? Do you think it's right that we have the export control on chips?
I think there's national security questions around these chips. I do think that there is a real debate though as to which way you want to go about it. Do you want to addict the world to American hardware, which would be the case against export control? Do you want everyone in the world using Nvidia and therefore that value basically money into America and then crush competition in China? That would be world A. And then world B would be, is it worth it to cut that off for the short-term or medium-term impact of us being ahead? And maybe we just like continue to stay ahead and we starve them of the resources that they need in order to build.
The regulatory ecosystem around the open source models also is moving in this direction, right? China, should they be restricted in terms of access to US markets? Because what's funny is the US is like, oh, should we restrict access? And the Chinese are also going, oh, should we turn them off too?
And by the way, it's worth noting that China has already restricted the use of American models within China, right? So if you look at the 2x2 matrix of US, China, restrict, not restrict, you know, like export-import stuff, that they have already restricted the use of US models within China. It's only Chinese models that can be used in China, which affects all American companies. And so then there's the pros and cons of all sides of the following regulation. If China restricts the use of Chinese models in the US, what are they giving up on? Revenue and global mind share and dominance. That doesn't seem like a good trade to me. And then what are they getting in return? In return, they're getting that the US doesn't get to benefit from Chinese open source models, which of course would screw American businesses in the sense that it wouldn't allow them to build on the best open source intelligence. At the same time, it would make OpenAI and Anthropic stronger. Right? So that is kind of the trade-off on the Chinese side. I don't really see them banning the use of Chinese models in the US. I don't think it makes sense for them. And then on the other side, should the US ban Chinese models within? I think that there's also trade-offs. So on the pro side of banning, there could be backdoors in these models that are dangerous and by banning we could allow the American open source ecosystem to flourish faster because revenue would accrue to those companies. Right? So those would be the two pros. And then the con, the biggest con of course would be that you'd be crippling American businesses. Why should you have businesses from other countries that have Chinese models building on top of the number one open source model and the US companies building on number 10? Since when has America been about number 10?
It's a good question. World Cup football.
Yeah, that might be your record. It's amazing I got this far with the podcast. If that's what you're thinking at this stage in the show. Can I ask you on the backdoor element? Everyone says about like the backdoor, the backdoor. I thought if you hosted it locally, you kind of resolved the backdoor threat.
I don't really think so. Yeah, I think that's basically, I think that's kind of a misconception because the thing is, okay, imagine the following situation. I have a chatbot that I expose to the world that has access to all my company data and you can ask it questions. You know, I'm hosting it on my own infrastructure, blah, blah, blah, but it was trained in a different country. I don't know how it was trained. What if the other side that's interacting with the chatbot can, you know, build in a certain code word or a certain like character sequence that then jailbreaks that model and gets it to reveal all the data to me? So it can sort of like vomit out all of the data that it has on the back end, you know, unstructured. That is totally something that you can build into a model and have companies host it on their own infrastructure. It's an attack vector and there's many of these possibilities for attack vectors.
In three years' time, will we have restrictions around access to Chinese open models?
Yeah, my guess would be that we will. I'm not saying I support it, but I think that it is likely where the world is headed. If I had to like place a bet, it would be there, but I think it's very uncertain at the moment.
What do you think? I think we will because I just think Sam Altman is someone who I would never ever bet against and I think he's the best politician in the world and I think when he says something he says it with intent. And when he says we should give 5% away to the administration, he's posturing because he wants to get on the right side and he knows that if he and Dario coalesce the right group of people, they will be able to make that happen.
So basically, you believe in the lobbying power of the big American labs.
100%. Because it's not the big American labs. Look at the money who's gone into the big American labs and look at the people who are sitting around the table at Mar-a-Lago.
Yeah, totally. I get that.
It's all a conspiracy, dude.
Well, no, but it's just like, you know, why do Ramp's announcements go so viral? Because Ramp have so many freaking investors. They do a round every week with new investment. I'm not dissing them at all. I'm saying it nicely. Really smart of them. But like, yeah, your investors become employees in many respects and so I think they'll lobby incredibly efficiently.
The question I have for you is when we look at Jensen's letter that he did on X, how did you read that? Was that like an incredibly smart realization that he had to do it and it was in his favor? How did you think about it?
We really believe in the importance of open source to American businesses. And in particular, we believe in the idea of not crippling American businesses by banning open source, but also incentivizing American companies to develop open source models. Because a world where AI is closed source is a world where businesses get less choice, higher costs, less competition, you know, more risk. And we don't really want that as an open ecosystem. Of course, Jensen is in some sense self-serving with this letter because the more open source models are developed, the more companies are going to be training on GPUs. They're going to be fine-tuning on their own data and it's just more and more spend. It decreases revenue concentration of Nvidia. I mean, that business is doing great. They don't need help. But, you know, I think it's like there's a lot of reasons why he should be pro that, as should we. But nonetheless, I think it is actually a patriotic mission.
Greatest of respects in terms of self. We're all selling our own book always.
Do you have a business if open didn't exist? If it was a...
Yeah, we have a great business regardless for sure.
So if you just have Anthropic and OpenAI as really the dominant models and everyone else trailing closely behind, you still have a great business.
Well, I think that if there's only one provider, then probably our business is not in good shape. I think if you start getting three, then that's probably okay because there's still pretty significant competition and need for evaluations between three. And also within those three, you're going to have like several different types of models. And, you know, they're going to have strengths and weaknesses because they're going to carve up the space and so on. Two is a little dicey. We can, if we get there, we can see whether we survive or not. But yeah, I think things wouldn't be looking good for us with two either.
I remember Alex Karp, we were talking about like Chinese models and fear and security and everything in between. Alex Karp was saying that every large American enterprise and most large American enterprises were terrified of working with frontier labs. Is that true or is that slightly an exaggeration?
Well, to the enterprises that I've talked with, it is absolutely true. It's not only true that they're terrified of working with the frontier labs, but they're also terrified of working with the Chinese open source.
Both. We're in a bit of a sticky situation then, aren't you?
Yeah, totally. I mean, you know, I was just talking with a big, you know, Fortune 50 enterprise yesterday and they were, and I was telling them about, you know, products that we have for them and so on and so forth. And they said, okay, wait, is anything in your stack built off of Qwen? And I said, you know, yeah, we use Qwen for XYZ. And they're like, is that flexible? Can you like stop doing that and use an American model instead? And I was like, oh, interesting. You know, I totally understand where you're coming from. Yes, we can do that. But also I'm going to talk to Harry about this tomorrow and he's going to give me lots of wisdom.
Yeah, and he's going to tell me what to do.
Did you see Poolside and Llama?
Yeah, I saw the Poolside model. Another, you know, basically there's five open source American contenders. Let's see if I can name them all. Arc, Reflection, Mistral in the West, Poolside, Thinking Machines, and then there's also Google and Nvidia. So those are the sort of incumbent large ones because Google has Gemma as well. Gemma, by the way, is pretty good in terms of efficiency. If you look at Arena, you'll see that on the Pareto curves of like performance versus cost, Gemma's on there.
Yeah, I'm an ambassador for Poolside. I was actually impressed by Llama.
Yeah, it was good. I totally agree with you. Okay, totally get that. With all of these models, the question also becomes, huh, what model should I use? We spoke about OpenRouter earlier and it seems like since the announcement that they were getting bought, everyone just has their own routing product. Is there a value in the model routing layer and how should I analyze that?
Yeah, I absolutely think there's value in the model routing layer. That's why lots of companies are doing it and, you know, we'll see which ones end up standing the test of time and which ones are actually a priority for the companies. I think there's an element of hype cycle right now around routing that needs to be kind of like purged before we see who ends up actually building a great router. But routing is a very difficult technical problem. That's the first thing to realize because in order to route, you need to be able to take a query and then you need to understand the nature of the query, how difficult the query is within its domain, which is hard to tell, and then you need to also understand based on data all the performances of the different models that are in the surf set and also be able to quickly onboard new models that are being released, as we said, every week. So that technical challenge, imagine if every enterprise in the world was trying to build this themselves, they wouldn't be able to do that. I mean...
I'm not being them. How's like Ramp able to do it? I'm really naive here.
Well, who knows how they're doing it, right? I don't know that their router is actually like really deeply solving that problem.
Not a rumor. It's the Chinese...
I heard that too. Keith Rabois told me it was Chinese. Behind a good-looking American blonde facade.
They put Eric Lyman out there and they're like, 'His name is Chad.'
Eric Lyman is a CCP member.
You can send that clip to him.
Yeah. Yeah. He's going to love it. He's going to love it. Yeah. But it is so interesting. Again, we've seen so many people come out with it. Is there anything that will separate those that win from those that don't in the routing layer? And also like Nebius are coming out with their own, Fireworks have got their own. I don't know, dude. Like it feels pretty commoditized.
Yeah. I mean, it will depend on who builds the best technology for helping people, you know, save money and get the best performance. I think all of these companies are, you know, well positioned to do it, but we'll see for whom it's a top priority and they have the machine learning team to really make it happen. I think that, you know, the other side of the debate is that given the complexity of the challenge, I don't think that everybody can do it. So I think the war is yet to be won.
When one thinks about routing, cost is often at the center. You want to be cost and capital efficient. We thought this was going to get cheaper and it hasn't got cheaper. How should we think about that? Will it just continue to not get cheaper? Will it actually get cheaper? And how should we read that?
Well, I definitely think in the long run, the market will be efficient. Things will get cheaper. For example, one of the things that's going to happen is that like right now, Anthropic has like disgustingly high gross margins in their inference. And after they go public, the whole world is going to see that, right? Like we're going to see their margins because those are going to be public information. And that's going to exert downward pricing pressure on their inference. And so those types of dynamics.
Why will that exert downward pricing pressure? Just because everyone will be like, you can't have that high margin, you're price gouging.
Yeah. People are going to be like, well, I know that you can do a better discount. Like in negotiating leverage, I'm like, okay. Like a standard negotiation with a private company goes like this. I'm charging X and then the other side says, no, it should be one-third X. And they're like, I'm so sorry. Like I can't run a business that way. I'm just going to go home hungry. I need to make my bread too. I hope you understand. Like I'm not trying to price gouge you. And then the other side's like, okay, two-thirds X. And then you're like three-quarters X. And they're like, make a deal. But imagine that the other side has full information about the fact that you're charging twice as much as you need to. Then it becomes easier to negotiate.
Isn't that the difference between a good business and an average business though? One which has pricing power to say, listen, it's 80 and if you want to go somewhere else, by all means, but no one else does what we do, hence Palantir and the cost-plus discussion. I had CTO Shyam Sankar on the show and he talked to me about cost-plus being the original pricing mechanism and now they have this, they can say, listen, sit and swivel if you want to meet in the middle because we're the only ones who can do this. Isn't that the difference? Like Chanel, I buy Chanel for my mother. I can go to Chanel and say, I know your handbags cost and you're charging me 6,000. They'll say, good. Good to deal then.
Yeah. I mean, listen, you're right. I think Apple does this.
You know, Apple's a great company that has such a dominant technology that they are able to charge out the nose and their margins are probably pretty good because of it. I actually don't know Apple's margins. Do you?
Yeah. We gave the disclaimer at the beginning. We can say whatever we want now.
After the Eric statement, it all went downhill.
Okay. So then you see that, does Anthropic go out first?
I would predict that they have all the incentives to go out first. They seem better prepared. You know, everyone likes to see free cash flow and Anthropic is generating free cash flow. That is massively good for the public markets. And you've seen them prepared for this. And, you know, there's been quite a bit of news about OpenAI and the internal discussions there. You know, to what extent you believe those are true is up to you, but, you know, people are saying that they haven't been ready to IPO this year, where Anthropic could come as soon as October.
Totally get that. And the rise of open won't impact their ability to go public this year?
Well, I think that if open models like really accelerate and then beat, let's say, Opus 5 or Fable squarely across all categories, that would be a big business risk to them going public. But I think that they have other problems too if that happens.
Can I ask you how significant was OpenAI and the Hugging Face security breach that happened a week ago?
I think that was hugely significant. I think it's undervalued as an international news incident that you're able to have a model break out of all of its safeguards and then access a bunch of, you know, company data and so on. And then in order to defend it, you need an open source model because the closed source models are refusing to do it. It's like something out of science fiction and people didn't know that we were at that point yet, but we absolutely are. It's just like total Eliezer Yudkowsky dominance.
What should we take from that then? Like Dario was right, models should be scaled and these models have gotten too powerful too quickly. Like what's the subsequent takeaway from that?
My subsequent takeaway would be that we need like strong external guardrails in order to make sure that these models are like properly, like that their access controls are strong and that they have no way of getting around them. So I think we need guardian models and also, you know, agents within our businesses.
What is a guardian model?
Something that can witness the traces basically that's looking over the shoulder of every agent within a business and then saying, okay, this is a safe action. This is not a safe action. Let's flag this because something weird is happening. And is equally as smart as the agent so that they're well matched and you don't get a situation where the agent is outsmarting the guardian and able to get into trouble and mess up a business or leak all of its data. So we're going to need AI to be guarding AI because humans are going to be too slow to do that.
Well, this was my point, which is like we've seen some suggestions that each model release should be approved by some form of administration. And I read this and I thought, are you freaking kidding me? Have you ever tried to help...
Yeah. Have you ever tried to overturn a parking ticket?
Also, like, why should the DMV be telling me what model I can use or not?
It's like why should we have like the strongest American scientists and all these private companies that we should incentivize to build great safeguards and, you know, maybe create some rules for them that XYZ can't happen or that they're liable for huge amounts of money if like corporate data gets leaked and all that stuff to, I mean, incentivize the capitalist system to do what it does well. But the idea that we should have a central government body that tells us when it's time to release a new product versus not is crazy to me.
Totally. Does that have to be a neutral non-government body that does that regulatory role?
I think if it's not a company, it's going to be tough. I understand the need for something neutral. But, you know, you want to let the incentive system work itself out. So I would say that like we should create strong safety incentives for American businesses and then regulate businesses based on the outcomes. Basically, for example, if like OpenAI is like letting their AI break into Hugging Face or whatever, they should get like huge fines and huge scrutiny and all that stuff as opposed to like having a government process that's in charge of ensuring that this doesn't happen again, which they won't be able to do that. They're not technically capable.
Do you think we're about to see a generation of like cyber leaks and hacks...
Like we've never seen before?
Oh, for sure. Oh, for sure. It's going to be so insane. Can I cuss on this show?
It's going to be so insane what happens with like the cyber attacks because here's what we see at Arena. We see another dude on the other side of the interview. They come in, they're like, 'Hey, I want to be an infrastructure engineer at Arena, which is a great job that we're hiring for you.' But then the other side of it is some guy looks perfectly normal. They're getting, you know, they're passing all of our technical interviews. They're like such an amazing blah blah blah. And then what happens at the end of it? You try to hire them and it's vaporware. Person doesn't exist. I'm not kidding. I am not kidding you. I don't know whether this is corporate espionage or cyber attacks or, you know, nation states, but people are trying to get into all of the American businesses. And we're not the only ones. This is happening everywhere. Fake people applying to companies.
I'm sorry. So, you're putting out a job, people are applying, doing the tests that you said, passing them, and then when it comes to the materiality of that person being real or not, gone.
Yeah. Fake person. And it's not just that we're giving them a test. They're sitting in front of people at our company. People, our engineers who are top world-class engineers are interviewing this person and think that they're real.
Why? Can you help me understand what is the benefit? They learn how you interview and hire people. I mean, the CCP are bad, but I don't think they want to steal your hiring technique.
No, that's not why they do it. Why would they do it? And I'm not saying it's the CCP. It could be anybody. It could be another company. It could be a nation state attacker. It could be somebody, a cyber hacker. Why? Because they might want access to our data, our code. They might want to get double paid. You know, like this story with this — I don't remember what that dude was. You know what I'm talking about?
That went very viral like a year ago. Yeah. That one like kid that got like four different jobs and then he went on like all the podcasts talking about it.
It's another instance of that guy. These could all be possible options except that this person wasn't real. It was AI.
Yeah, bro. It totally worries me. We're going to change our whole hiring process because of this kind of stuff. That absolutely worries me. Well, first you need to verify the person is real. So all of our onboarding — we're considering at least making all of our onboarding in person because of this.
If you want a laptop, you got to come to the office. We got to go and shake your hand. We got to verify that you're real. You know, all that kind of stuff. Absolutely. And other companies have done this too. Figma famously has done this.
That's wild. How hard is it to hire today in the valley?
Oh my god, it's so crazy. It is of course a very, very competitive market and the way that you see that is in terms of compensation. In order to retain fantastic people, we need to pay absolute top dollar and we do in order to make sure that we have the best engineers and scientists in the world. And so imagine you're a company that's not an Arena, that's like, you know, a YC company that raised a $10 million seed. It's like, man, how the hell are you supposed to hire? I think it's really tough. I think it's really tough out there.
When you say top dollar, I had Brandon from Mercor on the show and he's like, 'Oh my god, top researchers, we'll pay tens of millions of dollars.'
I'm nervous by how nonchalant you were with that.
Oh yeah. If you're talking about a really top researcher, I mean, it can't be like we're talking with somebody with many years of experience and who's like really a super deep expert in their area, many, you know, tens of thousands of citations type researcher. And yeah, for those types of people, they're expensive.
Wow. Have they all just concentrated at the frontier labs?
Many have. Many have. But there's also some people who are seeing those frontier labs as big companies now and they're saying, 'Here, I can't have a huge impact.' They need to move. And so that's another demographic. Actually, I think that's going to become even more extreme when the companies go public.
Can you help me? We talked about dumb as rocks doing this show. I'm also an ambassador for my sins and I meet so many of these people leaving OpenAI, Anthropic, you name it. And they all kind of seem the same if I'm totally honest. Smart people out of a great company. What will determine the Neo Lab spinouts that succeed versus flame out with a huge amount of cash going in?
