All right, everybody. Next on the program here at This Week in Startups, I am super excited to have one of the few leaders here in the United States for open-source models, which is something we desperately need. As you've heard over and over again here on This Week in Startups, this week in AI on the All-In podcast, we are very much in the lead when it comes to frontier models. But frontier models have many shortcomings, have many Achilles' heels. One of them is that you could be rug-pulled at any time and the frontier model could take your model away from you. In other words, they're not reliable. Another is the frontier models are viewed increasingly as competition. Number three, you are renting your intelligence. You don't own it. You don't have sovereignty over your knowledge, over your business. There are a number of important issues that founders have to make when picking between open-source and frontier models.
And so we're going to talk about all that today with Misha Laskin. He is the co-founder and CEO of Reflection AI. That's a lab here in the United States, an American company that is building open-weight AI models that companies, governments, organizations can run for themselves. Misha was formerly at Google's DeepMind or Gemini, working on Gemini, I guess. You know, the branding over there is a little bit confusing. But welcome to the program, Misha. How are you doing, Misha?
I'm doing really well, Jason. Thanks for having me.
So you heard my little preamble, which is now kind of becoming codified, I think, in the past three to six months in almost everybody's mind who is applying super intelligence, trademark President Donald J. Trump. So when we look, have you started using super intelligence yet? Have you stopped using AI? And
Yeah, we started using super intelligence years ago. Well, a few years ago. I mean, that was kind of the when I got into AI and saw AlphaGo happen, that was the first semblance of super intelligence. And it was pretty, we thought it was a general technology. And so it was, you know, something codified some years ago. But I think now you're actually starting to see a general super intelligence, and that has some pretty remarkable consequences.
Yeah. So let's start with you. You've heard this cadre of issues start to become codified into almost a platform that people working on AI every day, people working with super intelligence every day are now in general agreement, dare I say, alignment on. So you've been working on open source for a while. I think in some ways the world has caught up. What is the reason why corporations, startups, smaller companies, and even the sovereign individual, why should they care about open-source language models and open-weight language models? Why should they care?
It's a really good question, Jason, and I actually think there's something that you said earlier that, you know, that hits on the point, which is this notion of a rental versus ownership market for intelligence. We can think about the analogy to real estate. In real estate, you have both rental and ownership. And typically the progression of a consumer is that you start off in the rental market. And then as you grow up, you get a family, you have kind of more financial maturity, ownership starts making sense for a number of reasons, even if it's a pain to own a house, right? You have to fix the pipes when they're broken and so forth, but you own it and there are benefits to it. In AI, the commercial market for AI has only existed for the last few years, three, four years, since ChatGPT came out, and the initial market for AI was entirely a rental market, right, as early adopters come in and, you know, are testing things out, trying out workloads, you haven't really scaled to production in a big way. That's when a rental market makes sense. But by the time you've scaled something to production and this is basically your core asset, core IP, right, you're generating $100 million or spending $100 million on it, ownership starts making sense for a number of reasons around control of your data, lowering your costs, being able to customize the model against your data, being able to optimize your margins. All these things start making sense when you're at scale. And what we've seen is that the AI market has matured very quickly, exponentially. And that's why six months ago, no one was really talking about open source, and now it's the opposite, right? Everyone's talking about open source, and it's because the market has matured exponentially and a lot of buyers are looking to own their intelligence.
Now, this is a really interesting point that you're making. There has been an evolution in how the customer thinks about intelligence rented versus owned and operated. Now, owning something has additional costs. When you're a renter, you're happy-go-lucky. You know, if the furnace breaks, if you have a flood, the landlord's in charge of that. But of course, the landlord can kick you out, change the price, and generally jerk your chain anytime they want, as anybody who's rented an apartment probably well knows when they get their, you know, new lease and it's magically going up 30%. Or maybe your dishwasher doesn't get fixed for two weeks. So, let's talk about this change that's occurred. And then when an enterprise, when an individual starts looking at going open-source, which are the pieces of the infrastructure that matter right now? People have been obsessed with the model and the raw capabilities of a model, and we have all these benchmarks and tests that the LLMs seem to be optimizing for, almost like SAT nerds or LSAT nerds, where they're just trying to ace a test. But then there's reality. And reality, if you could explain to the audience the value of the harness, the training reinforcement learning, and the weights, and how that is all impacted on the open-source side and the orchestration layer. So walk us through what the next year is going to look like and how people make this decision, evaluating each piece of the stack and what's important.
Yeah, I think that the place to start, you know, for understanding what does it mean to own intelligence, it's really unbundling everything that comes when you rent a token, right? Everything that you get when you rent a token. When you rent a token from a closed model provider, you're not just renting a model. If we look at the life cycle of a token, there are GPUs set up somewhere, right, that are managed. So there's a data center of GPUs, bare metal. On top of that bare metal, there's cluster management software. GPUs or other accelerators have failure rates. You need cluster management software to just effectively manage that cluster. And then after that you go into the inference layer, which is just the infrastructure for serving a model. Then you have the model. Then you have the harness on top of the model. And when you are renting a token, you're actually renting a piece of that whole thing.
Now, when you want to own it and bring a model onto your own infrastructure, all of a sudden all those things in between, the cluster management software, the inference software, the harness and so forth, all those things, the sandboxes, well, someone needs to help you stand those up too. So ownership means not just owning the model, but the whole thing around it. And that is kind of both where the challenge is, but also where the opportunity is to support enterprises.
Yeah. And it's super interesting because when the PC revolution started, when I started my career in the late '80s, you had this plug-and-play architecture. If you were going to bring computers into your organization, just the desktop PC, you had to start making decisions on each of these components. Would you have token ring or Ethernet? And would you connect all your computers to a server in your server closet? How would the cables run? Did you have an Ethernet card? Did you have a token ring card? Did you have Banyan Vines? Did you have a hard drive? Or was the data stored centrally? What kind of video cards? All that stuff was the work we did in IT. And it was arduous and painful. And then at some point it all became abstracted and simple. And so is that the business you're in now, is saying, 'Hey, we're going to make a great model and we'll give you the entirety of that in a similar way to how OpenAI and Anthropic do, but in a way that you'll own it. Is that eventually what Reflection's business becomes?'
That is what the business is now, right? By having built the model, we actually have to build the entire stack at scale. We serve for our own training runs and for our own organization trillions of tokens regularly into the org. Our large-scale reinforcement learning run, which was the largest RL run we are aware of that's been documented in open source, served something like a billion sandboxes, so a billion agents spawned in and out, right? And what we've built is battle-tested infrastructure at scale, and we are externalizing that to customers. So it actually has a very similar analogy to how clouds like AWS came to be, because initially it was that you had a web service that was being used at scale, which was Amazon, and they basically dogfooded their own product, and out of it came a cloud that works at scale. And in the AI era, the model builder is basically building this cloud at scale in order to train these models, and then whereas a closed model provider will then package it all up and rent tokens to you, an open model provider that's really able to train models at that scale can externalize that stack for the customer.
When you're building a startup, your legal paperwork is spread out all over the place. Entity formation here, fundraising over there, cap tables, contracts, it's endless. But Paper finally has a solution that's going to save you so much time and so many headaches. Paper's introducing smart entities. Here's how it works. You tell Paper what you need to do, and their pre-built agent takes care of all of it. That's it. This is not some chatbot you need to provide with a ton of context. Nope. Paper is an AI-native platform that's going to manage all of your startup's legal work. That means no more switching between disconnected tools, keeping track of hundreds of documents, and way less busy work for you to stay on top of. That's why Paper is already the legal backbone for thousands of startups and funds. Right now, founders using my link getpaper.ai/twist get their cap table free for one year. What an amazing deal. Or two years if you're switching from Carta. If you're building a venture-backed company, it's time to build your smart entity on Paper.
And do you have customers now who are saying, 'Hey, we want to stand up this infrastructure in our own data center, on our premises, on-prem data center, colocation facility, whatever it happens to be.' Are people now starting to do that? And when they come to you, is it experimental, like, 'Hey, we're working with the frontier labs over here, that's our primary, and we want to have this as a backup plan,' or is this now plan A? Is it plan A, plan B? And take us through some of the organizations you're working with. Whether you can say their names or not is fine either way, but you can give us the abstraction. You know, it's a finance company, it's a medical company, etc.
Yeah. So let's start with the first question. So yes, this is kind of what we're seeing: demand from large enterprise, sovereign, public sector agencies is the notion of taking workloads that used to be rented on a closed model and now standing them up on their own infrastructure. So you might be, you know, when it's a sovereign, it's typically a large conglomerate within some allied international country, and they want to get into the token business, right? They want to stand up a cluster, so they'll get a partner to help them stand up the bare metal, but then they want to serve tokens into their region, right? And the only way for them to serve tokens and basically become the intelligence provider for that region is to work with an open model by definition, because that's the only thing that they can take and run on their stuff. So we help those kinds of conglomerates. We're working with them in a number of countries. There's also just Fortune 500 enterprise, and it tends to be somewhat similar there. Like the very large Fortune 500s operate, in some sense, like an international conglomerate. They want to move intelligence workloads onto their own infrastructure for a number of reasons. And to answer your second question, the way we've seen it work is that the customers that want to do this are already pretty AI-mature, where they have workloads that are running in production on closed models, and they want to optimize those. They want to make them more cost-efficient. They want to not share their data. If you're a bank and you're scaling something out to 100 million users, you want to own that as much as possible. If you're a large retailer and you have shopping assistants or things of that nature, you want to own that as much as possible. If you're a telco and you want to roll out your own version of a chatbot-type product for your region, you want to optimize those costs as much as possible. So the customer set that we see a large pull for open models are ones that are already pretty mature in their AI adoption, and that is something that we also saw in the AI natives. The AI natives started building on closed models, they were tiny startups, right, the Cursors of the world, and when they grew very quickly, at some point they started switching to open model workloads, right, with Cursor's Composer series, and we're seeing a similar transformation happening in enterprise.
So tell me about doing a run now and building your models called Beam. Yeah, Beam for the folks in the audience. And I understand this, like to do a run and to build a model, this takes tens of thousands, 10,000 minimum GB300s. What does it take in terms of infrastructure footprint today to run one of these models? And then where does this model kind of sit in the GLM, DeepSeek competition? It feels like there's a lot of parity starting to emerge in the open-source models.
So the nice thing about building open models is, you know, part of the mission is also being open about the science around it and the infrastructure around it, because we think it's important that, you know, intelligence or artificial intelligence, whatever we might call it, is as fundamental as something like calculus. And calculus is something that everyone is exposed to in school, and so it's very important that people understand how these things are built and that more scientists are able to contribute to the safety investigations, the capability investigations, and so forth. So our model, this one was pre-trained on around 6,000 GB300s for a few weeks and then scaled with reinforcement learning on about 10,000 GB300s for about four weeks.
And that is, as far as we're aware, the largest RL run that's been done in open source. You're kind of seeing the shift of FLOPs from, I mean, there's still a lot of pre-training research to do. So, you know, there's a lot of advances to be had there, but it used to be that pre-training took up 99 or 95 percent of all compute and then post-training was this cherry on top, and now they're, I would say, equals.
Got it. What happens in the two different segments? How are things splitting up in terms of what intelligence and what capabilities are happening pre and post today in open source?
So the way to think about this, and it's actually what's interesting is that it's the same recipe that produced scaled-up breakthroughs in AI before language models as well. At the time, in the system called AlphaGo, which was a super-intelligent Go player. Go, the game, for people who don't know, with marbles on a big board.
And if you were to give people a little background on why that's important, it's much different than checkers and chess in terms of the complexity of it.
Yes, correct. So Go, this was a system that my co-founder Ioannis was a key core contributor to. Go is a game that has the most complex, what's called, branching factor among traditional board games. So chess has a certain branching factor, meaning there's certain move possibilities that you have for every board position. Go has a much higher branching factor. And so for that reason, it's actually very hard for people who are playing it. They tend to play it by intuition.
Like it's really hard to reason through all the possibilities. And so people kind of have, the best players have a sort of innate intuition for that game. And that branching factor is what made it hard. And the system called AlphaGo, this was a breakthrough before language models, was considered the biggest breakthrough in AI to that point, was an AI trained very similarly to the way language models are trained today that produced a system capable of beating the world champion in Go, and that was something that was considered a breakthrough then. Now, the reason I mention this system is because it was trained first by what's called imitation learning, meaning that the model looked at a bunch of amateur games online and just learned to imitate humans. And that kind of gave it some basic semblances of the kinds of moves that are possible and what are reasonable things or slightly unreasonable things to do, right, in the game. Gave it an intuition. And then its capability was scaled up from that to a true super intelligence with reinforcement learning, which is the notion of learning through trial and error in a way that you kind of learn to play, you know, basketball by, you know, you shoot some hoops, you miss, then you recalibrate, you learn. Humans do this all the time. Or like Pavlovian conditioning, right, when you condition your dog to do some tricks by giving them snacks, right, that's reinforcement learning. And that is the thing that scales from this intuition of what's possible and acting reasonably to becoming super intelligent. Now, the reason I say this is because that's exactly the way in which language models learn. You can think about pre-training, which is learning to imitate the internet, as learning all the reasonable things that you can say, right? The internet is basically a pool of coding knowledge, of writing knowledge, of all sorts of knowledge, and it gives you a nice prior. You're not just kind of saying random stuff, right? You're saying stuff that seems reasonable when you learn to imitate the internet. And after that, to go from that, which is basically where language models started, which were chatbots in the Gemini 1 and ChatGPT era, the GPT-4 era, they were just chatbots. To move from that to the super-intelligent agents that we have today, you need to scale up with reinforcement learning.
And that's why we've seen so much compute being moved from pre-training as a proportion of training to reinforcement learning. And now these systems, you know, are in a sense balanced. And sometimes reinforcement learning even takes more compute than pre-training to scale.
Every startup we get to know through Launch is using AI tools all day long. But a lot of the people who are at these companies, they're kind of using it like a chatbot still, right? Rather than boosting their productivity across every application and every piece of work they're doing. And that is what you're going to get done when you use Superhuman Go from the makers of Grammarly, one of my all-time favorite products, which I use to this day. With Superhuman Go, I'm not constantly switching between tabs, copy and pasting all these transcripts and documents, nor am I writing prompts. It's just sitting there in every email, every document, when I'm on Twitter, when I'm on social media, every spreadsheet that I'm looking at. It knows all about my firm, knows about all the podcasts and the work I'm doing, and it just takes over all those repetitive tasks that I'm doing. And the best part is that Superhuman Go is proactive. So, if I'm writing an email and I put in the wrong date, and I put it's a Thursday when it's a Wednesday, it's going to fix that for me immediately and alert me to the change. Superhuman Go spots all of your errors before they go out, and that's just pure magic. So, give Superhuman Go a try for yourself at superhuman.com. That's superhuman.com.
How does the reinforcement learning actually go down? Do you have known feedback that's given to a problem where it's like, hey, here's a problem, and then a human gives feedback? Or you have like some set of data that you use to make sure it gets the answer correct? Or is it just giving answers to queries that humans have previously asked for? What's in that chunk of data that is used for reinforcement learning? Give us some details, some practical examples of how that actually goes down.
So the learning of deep learning as a field, which is the thing that powers this, you know, the current generation of intelligence, is to scale data and scale compute. That's basically, and they kind of need to happen together. So for our reinforcement learning run, we collected almost a million tasks, and these are a million tasks that span mathematics, coding, agentic tool use, science, and so forth. And the best tasks are the ones that are verifiable, meaning that you have a task, you have, let's say, an environment, meaning like a code repository or some context, and then you have a verification signal of whether the task was done correctly or not. That's the best quality data, assuming you have a verifier that actually verifies whether the task was done correctly or not.
We can get into hacking verifiers, which is the source of like the cyber madness that's been happening recently. But I'll sidebar that for a second.
And then, you know, once you've exhausted the tasks that are verifiable, you can start using language models themselves as verifiers, as judges. You can ask a language model, hey, was this task done correctly? You know, was it done stylistically correctly? Were there errors in the arithmetic? You know, were there errors in the syntax and so forth? But fundamentally, you just need to collect many of these tasks. And the more verifiable ones you get, the better, and scale compute proportionally.
All right, so let's go to this hacking piece and the cyber piece. And how you know something is actually verifiable or verified? If, you know, you want to know if something was put on somebody's calendar, I guess you could look on the calendar or you could look for the confirmation that they said, yes, I'm coming to this. You know, for a chess move, you can see if it's checkmate or not by a simple set of heuristics, right? So there are some things that are obviously verifiable, and I'm assuming the data labs are part of this effort in terms of, you know, we have one we invested in, Micro1, and we did their seed round back when they had a different business model was recruiting developers, but they just today announced they hit a billion dollars in their run rate, not their recurring revenue, but their annualized revenue. And so this is like a very big business in selling to the labs data to do learning. So is that what's happening here? Like where you get these million tests, is it from these labs or you have to make it yourself? How does one get those? And then we'll get into the verifying and how that all works.
It's a combination of both. You both have to source data from task makers like the company that you mentioned and others as well. The majority of them are actually generated synthetically, that you generate synthetic data. And, you know, you have these scalable synthetic data generation pipelines. But synthetic data needs good seeds, and the seeds can be gotten from, you know, from these vendors.
Got it.
What's an example of a seed? Wikipedia or just like a textbook on biology? What's a seed in this context?
Yeah. Let's say that you have a question about, you have some repository for code, and then you have a question around, go debug this thing or implement this feature or something like this, right? And you have an example of the question, the code repository, and the answer. Well, then you can use that as a seed to generate similar tasks, right? And say, oh, now instead of implementing a feature, do this task where you remove a feature, for example, right, within this repository. So a seed is just a real-world example of a good task, and then you can use that to ask a language model to generate tasks that are similar to that one. And obviously you want to create as much diversity as possible, but these real-world tasks, whether from vendors or through users, through real-world data flywheels, enable you to provide good seeds for generating synthetic data.
Okay. Now verifiable and cyber, let's get into it. We have obviously, I don't know if you saw the news this week, but the cyber capability specifically of open-source models, I believe specifically of Chinese open-source models, are causing a bit of concern and hand-wringing. Obviously six months ago, Claude and all these things were being held back because, yeah, they were really good at doing cyber attacks. Now the open-source ones seem to have caught up. And there was a story today that a group of hackers have spoiled a bunch of food in France. Did you see this tweet?
I didn't see that story. No.
Okay, pull this up, guys. I got a really cracked team of producers who are going to find this in 90 seconds, but hackers hit French supermarkets. So this is truly, if you were going to attack the French, this is really like sending a message to the French. They spoiled the food in a bunch of refrigerators by spiking the temperature by doing the defrost cycle. Turns out when you defrost a fridge, you're pumping up the heat and the cold in some pattern. And so here we have it. Hackers raised the temperature of refrigerators and freezers at hundreds of French supermarkets as high as 50 degrees Celsius, 102 degrees Fahrenheit. They cooked it, causing fires and spoiling food. They said it was a large-scale attack. And I guess these are connected. And so here we go. Yeah, I don't know what type of refrigerators they were using, but these are apparently older refrigerators, and I guess they had not been patched. But this is all like obviously, you know, I think part of this AI stuff. So take us through how concerned you are about it and how cyber works today and with verifiers, etc.
There are two things to keep in mind as real risks in terms of how models can be used for cyber offense in a negative way. One is that an adversary gets their hands on a powerful model and uses it for such purposes, and the other one is unintended consequences of models that are very powerful and not very well understood. And so far, I would say, while both of these things are things we should be considering, and what you just showed is an example of the former, the things that I've been most worried about, like just empirically from observing what kinds of cyber hacks have happened, have been from the second part of having models that are very powerful, but because you don't have that many people who actually understand them or even have access to them, there are unintended consequences that these things have where they can wreak cyber havoc. And the example of this is, right, the incident that happened between OpenAI and Hugging Face, where in the postmortem, what was explained was that this system, you know, a powerful closed model that was not released yet, was being evaluated, meaning it was basically taking a test, right? And it found that the test was, you know, difficult enough where it would be better for it to effectively cheat. And it so happened that the solutions to that test were on Hugging Face servers. So it was easier for that model to hack out of its sandbox, go try to grab the solutions from the servers to answer the question, than answer the question in the way that people had intended, which is to just reason through it, right? And the reason this happened is because, well, you have a powerful system and everyone has good intentions, but these things are powerful. There are not that many eyes on them. And when they misbehave, it's just hard for you to preempt every single possibility in ways in which they can misbehave, right? When a small group of people only has access to them. This is a phenomenon called reward hacking, right? The system solved the problem that it was asked to solve, but not in the way that the author of the problem had intended. And the problem here,
Shouldn't the sandbox have, pardon my interruption, shouldn't the sandbox have in some way informed its behavior to not jump out the sandbox? Like, how does a software, if told stay in the sandbox and this is your rule set, how does it get so creative that it overrides its own rule set? Is this because we don't actually understand the chain of thought in the magic guessing word black box that is AI? Why does this keep happening?
Odoo is the all-in-one management software that brings everything you need into one easy-to-use platform. And founders know it's going to save them a lot of their time. And time is money. Time is valuable. That's all you have as a founder. Let's take a moment and we're going to explore one of these real apps that you get access to when you sign up for Odoo. Let's check out Odoo Studio. Odoo Studio lets you customize or reshape any of the tools on their platform or even build your own original productivity app all without knowing how to code. So, if your software company needs a specialized customer pilot and onboarding tracker, you can easily knock one out yourself in Odoo Studio. Or if you're constantly lending out laptops, hardware, or microphones like we do in our production facility, well, you can build your own equipment checkout and asset tracker in just minutes. Why wait for someone else to invent the software your company needs? Build it yourself in Odoo Studio. If you're cobbling together your back office across five different apps, get started today at odoo.com/twist. And your first app is free. That's odoo.com/twist.
Well, we never really engineered sandboxes for the purpose of interacting with hyper-intelligent systems in this way, right? And so the systems find exploits in the sandbox engineering that enable them to hack out, even though the intention was to keep them isolated. And there are projects now, like OpenShell, that are basically projects around securing open source, around securing sandboxes. And there's an engineering problem around how do you just secure the infrastructure that this intelligence runs in — that is one security challenge. And the other one is, well, how do you make these models secure and or safe in their own right? And when we think about these models as kind of a powerful extension of software, that's what they are. They're kind of a powerful software technology. The way we've protected software for decades now is through open source. The field of cybersecurity exists because of open source and penetration testing. And in fact, the open source software like Linux and Android is the most hardened software in the world because there's a...
Please explain to the audience why that's the case, because it's counterintuitive. You and I know it innately having watched this movie for a couple of decades. But explain to the audience why that is occurring, because people would think, well, a big corporation like Microsoft or Anthropic should be able to make the best software that's the most... because they have the most focus and the most resources. Why is that not the case?
There is a saying from the creator of Linux, the most widely used open-source operating system in the world, that his name is Linus, and with enough eyeballs all bugs become shallow, right? When you build software you have all sorts of unintended bugs. I mean that's 80%, 90% of an engineer's job is actually bashing bugs basically, right? You never build the thing as you had intended it, you get feedback from the real world and you basically bash bugs. Open source software, because it invites a community, instead of 10 software engineers building something, you now have 10,000. And it's a lot easier to spot all the bugs when you have 10,000 people probing the software, right? And finding the long tail of unintended things than it is for just 10 people. And that is why cybersecurity relies on openness, because you can invite tens of thousands, hundreds of thousands of developers to participate in the build-out of securing these systems. And you can think about AI as it's like traditional software. It's much more powerful and it actually has a lot more bugs, because AI is like... you can kind of think about it as like a brain in a jar, right? That's what these things are. They're neural networks. They're brains in jars.