All right, everybody. Welcome back to This Week in Startups. It's your boy Jay. I'm here with Lon Harris. I've been on the road, but finally got a decent recovery score. I'm finally back in the game, Lon.
Oh, your sleep score. You're coming back.
I get a decent amount of sleep debt that I'm still working off after the All-In Summit and then coming up here to Dreamforce and face mogging... or what is it called?
Frame. You were frame mogging Marc Benioff. Yeah, frame mogging Marc Benioff. You're so much larger and more imposing in the frame than him.
In the frame. Correct. This is the new trend. But he's, you know, he's a 6'6" guy. It's kind of hard to frame mog. So I just jumped.
Yeah. I did not realize he was such a kaijuesque figure until this week when you're seeing all those images of him from Dreamforce.
Well, and they were kind of doing the angle with like him. So they were literally frame mogging folks because they kind of used his angle and then were looking down at the crowd or whatever. It was a weird angle. But as you can see here, yeah, finally getting one up on him.
Well, and then if you look in the second one, I'm actually standing on a chair. So, I am like so tiny compared to Marc Benioff, but we had a fun time at Dreamforce. And I kept telling him Slack was the Oracle. Slack was the Oracle. Like, it had all the information. And he said, you know, Jay, you said Oracle twice here. It's not cool, because they're competitors.
Wrong brand. Wrong brand. But it was great to be at Slack Force as I'm calling it now. I call it Slack Force because Slack is like the just all your if we're going to really train AI, if you just gave AI all of Slack, I think you could do all of our jobs at this point.
Yeah. Oh, I mean that is when you're first using AI and you finally do plug it into Slack, that is one of the real game change moments where it's like, oh, all of a sudden it has all this context. It understands all these things it didn't know about before. I used to have to explain everything and now I could just read the thread and it's good.
Well, listen, we got a big show today. I don't want to lose anything. So, I'm going to just start my Plaud here if you don't mind.
You got to turn on your Plaud. Yeah, I got to hit my Plaud.
Well, you know, we always applaud Plaud here because this is the greatest product ever. We started doing the show notes in Plaud so that it's really easy for you all to understand what happened on the show, get lessons from the show, all the best moments. If you look in the show notes, you'll see we link directly to our Plaud recapture and it's got a transcript obviously, but I prefer the summaries and the mind maps. It does all kinds of interesting things.
It's actually a fascinating product. There's all these templates and so you can decide what kind of summary or readout you want to produce based on, you know, is this a meeting? Are you with your doctor or somebody else? Is this a casual chat? And so it sort of understands what kind of output you want. It's a very clever system. So if you or your work depends on conversations, interviews, meetings, or calls, we want you to check out Plaud at plaud.ai/twist and use the code twist to get 10% off your Plaud NotePin.
Speaking of great products and hardware being hard, you know, Hugging Face, a great company. I've been trying to get them on the pod for a long time, and I love a good side quest. We talked about side quests on the show before. Side quests are something you just feel compelled to do. Not because they are going to be necessarily the greatest business or make the most money or get you the most shine and service your ego, but a side quest is because you're kind of interested in it, right? You got a passion for it. You want to figure it out. And I thought one of the great side quests of the year and I mean people are starting to do a lot of these. Don't recommend it for first-time founders unless you got a lot of resources. You got to stay focused on that number one company. But Hugging Face, you know, host of all these great open models, bought a company doing robotics and then they released this amazing new product. And I said to Lon, 'Hey, produce the show. Please get me this. Produce producers, get everybody in line and let's get this person who's in charge of the Hugging Face robot because I think this has got legs, so to speak.'
It does have legs. Pollen Robotics was that robotics company Hugging Face acquired. And so next up we are joined by Tom Wolf. He is the co-founder and chief science officer of Hugging Face. And he built the Transformers library that basically every AI lab on Earth runs on. And he was also involved in Micro Duck, the open-source robot duck that sold 10,000 units in its first five days of launch. Here's Micro Duck. Now we're watching him on the camera. So welcome, Tom, to the show.
Thanks. Thanks, Jay. Very happy to be here. Talk about this duck. Yeah, this is incredibly cute, a lot of fun.
Why did you guys decide to do this? Is it for the fun of it? Is it because you think robotics is a long-term bet and this is a great way to learn and market it? Take us through the origin story of this very cool personal robot for hackers and kids, I'm gonna say.
Yeah, I think you're right. People love it. Okay. I won't deny there is a lot of fun in building a product like that. It's actually really exciting as a builder to make something that you know everyone in your family not find scary like a humanoid but just find really cute and they all want it. Like all my friends were first to say okay we want to buy it wherever you do it. The thing is it was surprisingly difficult to do. That's our longest robot in the making. It took us two years to make this robot. So it looks small. It looks simple. But as you saw in the video it's able to stand back up. There's a lot of things. It can grab things. It's actually surprisingly capable for its size. And so for us, we had for a long time and I just put it today the evolution of the robot that you can see. But for a long time, it was not up to the level that we thought everyone would be crazy about it. And then it's only earlier this year that we were like, okay, this can actually okay this is part of a side quest. Of course, Hugging Face is way more famous for its hub and that's our business model is storing data sets, storing model weights and building community but we think here there is something where we can create a new community of people who are actually AI builders but AI builders for robotics. So you can buy this product for your kids, for your family, for yourself but you can also buy it as an AI builder.
Yes. Sorry. Yeah. So you can also teach this robot new behavior. You saw it can do roller skating. The roller skating is actually just an intern spending an afternoon with this crazy idea. What if I put some roller in the simulation? Can it learn to roller skate? And it worked so well that we were like okay let's put that in the demo launch video and let's actually ship the robots with small roller skates so people can play with that. But we think there's a lot of other things. So you see it can play like soccer. It can and it can very likely do way more things. The fun part is actually when we released it on the web people went crazy and they started to put the robot in all of these various simulations doing all of this thing walking on skittles like doing like swing like a lot of things that we had no idea you could train a robot to do. So, for now it's only in simulation, but when the robot will ship in a couple of months, we think this will transfer very likely.
And it's sold out. You have tens of thousands of these on order, hundreds of thousands. What's the latest?
So the latest I think is a little bit over 16,000.
And but yeah, I think that will be very likely the most widely sold robots. I don't think anyone sold a moving robot like that over 10,000 units. And we have not even started to do any sort of marketing like we just basically posted a video on Twitter. That's the only...
Amazing. How does Hugging Face make money hosting models and weights etc. And then how do you think this will eventually monetize if at all or is this just a community building effort? I think that's really underdone. Sometimes people, you know, spend money on marketing. Sometimes they try to kickstart a community, but this one seems to have market pull. People are opting into the community. It's got that market pull in a very strong way. So tell us about Hugging Face's core business, how you make money there, and then how this will make money if at all.
This is probably the most widely asked question about Hugging Face is how do you guys make money because everything's free. So, I mean not everything is free. We have people who we actually had a strong uptick of revenue this year. We've been publishing some numbers. We passed in April we passed 100 million and then we even climbed way higher in the summer. And the main source of revenue right now is around storage. So people we've developed for our own needs very efficient fast and cheap storage quality. So we store petabytes of data. And this is very useful in many fields when people need to train need to evaluate model but also in robotics because robotics has a lot of this image I mean you know our Tesla they record like terabytes of videos you know every time they run you need to store that that's one of the most data hungry area of ML and it's having a huge boom. So we have many robotics companies right now using our hub to store the data and we have a historical activity of storing models. So you can store them for free publicly but you can also pay for more storage privately.
This is very incongruous to me because you think storage is a commodity and people have tons of storage around they could use Amazon or Azure or any number of things to store things. Why would people I'm just curious here you pick to store stuff on Hugging Face versus their own servers and you mentioned something about speed or it being unique to AI. Maybe you could unpack that because there must be something here that is not non-obvious.
We've got a lot going on here at my venture firm Launch. We're making investments in 100 startups a year. We're producing podcasts and newsletters every week to help our founders and the venture capitalists who listen to our pods. And we're putting on live events all over the world. The team here is very organized, very hardworking, but it can still be pretty stressful to keep track of everything. And our solution has been Superhuman Go from the makers of one of my favorite apps, Grammarly. Superhuman Go is going to make everyone on your team twice as fast by quickly summarizing dense threads so they don't have to spend all day reading. They can just TL;DR get to the point. Too long, didn't read is real. Or providing important context inside whatever document or email they're already reading because we've got Superhuman Go inside all the tools we already use. There's no constant switching between tabs, copy and pasting and transcripts all over the place or doing prompt trial and error until they're perfect. It just works when I need it, allowing me to make crisp decisions with confidence. Give Superhuman Go a try for yourself at superhuman.com. That's superhuman.com.
I think people thought storage was solved because of course you can pay S3 and like you have all the things. The thing is AI is very different than previous big data like it's way less random access. It's a lot more you will download a big blob of gigabytes or terabytes like the latest models they're terabytes inside. They're really massive. So you want your storage to be very close to your GPU for instance. So we do these things where actually your storage is pre-warmed next to your GPU. So it's really really fast. You don't pay for egress. So we basically develop this full solution. Another thing is you want deduplication sometimes like let's say you just add a single row to a data set. You don't want to store again the data set. You want this versioned data set so you can move around the various versions of the data set without having to store the duplicated copies. So we need this type of things that Dropbox or this company did back in the day. But we needed to adapt for the AI world for the AI workload for storing this thing and streaming them. We need the speed is the key issue there and getting close to the GPU and not having to make some hop to some other data storage especially over the open internet rather would be a showstopper.
Why haven't you guys and or maybe correct me if I'm wrong the OpenRouter business where you know they have the models and you can just route your tokens. You guys aren't in the token business to the best of my knowledge. Why haven't you gone into that business because you're hosting all the models and you could very easily help people route to the proper model? I'm just curious.
Yeah, so we provide something like that but one level higher. So if you go on the model page, you will see all the inference providers who are providing this model. And so you can select one. We also show you there are different advantages or drawbacks. Some of them are really fast, some of them are more expensive or cheaper. You have various ways you can rank them. I think one reason we never really went to the OpenRouter was in the early days was mostly something that was useful for closed models and we had this very strong stance in the past that we would focus on the open source models. We focused on and pushed them and so we didn't want to mix all of that with also giving access to the closed models. I think right now it has changed of course because of the rise of open source models that's been really great but it's still fairly recent right it started really last year I would say with DeepSeek to really catch and this is the year of open source it's crazy this year has been for us like riding a tsunami wave of open models.
Yeah and so maybe you could host open source models and that'd be an incredible business perhaps maybe you never know.
Yeah we do have some compute now but we tend to stay one level lower but it's also a significant part of our revenue now which is just giving compute back to people inference and training compute.
So tell me let's go back to the robot there. Who are the people who are the ideal customers of this? Have you done any analysis of the tens of thousands of people who have bought it of if they're just hackers, developers, parents, who's embracing this? And then if this goes well and there's a third version 3 years from now in your mind, Thomas, tell me what you see.
We've been really surprised. I mean, it's our second robot. The first one was Reachy Mini. We actually do a relaunch of a new version in a couple of weeks. This Reachy Mini was the first time we were expecting to just sell a couple hundred of this unit and we sold 10,000. So we started to dive who was buying this robot. We're like, okay. So there was the expected tech executive or ML person or software engineer. There were also a lot of people who were just very curious about robotics. They see news that robotics is going to be big in AI. They want an easy entry point. They don't want to spend 20k on a 1X robot or something really expensive. So they're like, okay, that's $400. I can buy this. So we saw investors wanted to understand AI. We saw people who are working in related profession like I don't know like highly educated or something like that curious about AI finance people so really broadly a way wider ecosystem. Reachy had one limit which is could not really move around so people were mostly buying it to understand robotics. Micro I think we have crossed another step which is people we saw a lot of my friends buying them are not even working in tech or they're not even especially interested in tech. They just think this is really cool. That's a working duck. This was not possible to build two years ago. This is a new thing that AI allows. It looks really fun. My kids are very excited to get Micro. We don't have one yet because we're really short on supply and I couldn't even bring him one here. But we think this is a new generation. So in our projection, we might sell up to 50 to 100k of this unit in the coming years if we keep selling which is massive.
I mean I just to double click on that a little. I'm curious like how challenging is it to train your Micro Duck in a new skill? Like are people like me who don't have a coding background going to be looking online for Micro Duck skills to download or is it theoretically possible I could teach my own duck how to do something?
Yeah. So it comes out of the box with a lot of behaviors already. So you even have a remote control. We were saying if we put one or not because it makes it look like a game. We think that's the easiest way. You take it out of the box, it can already walk. It can quack. It can grab things. It can rise up like it can do this roller skating. So, you already have like probably like 15 maybe more behaviors. But then the fun part is when you want to spend a little bit of time teaching something new. And what I discovered was I wanted to make a tutorial post on Twitter how to do that. And basically my tutorial ended up just being ask Codex or Claude Code or Grok to do that for me. So my expectation is when you receive that you just say train this new behavior for me and most likely it just will work out of the box. So I think we're also moving to this vibe coding robotic world which is a new thing nobody has any idea how it's going to be exactly.
Well, I mean now you're going to have Lon the ability just like we have people in our little venture firm Launch. We have four people writing software who are not coders. I mean they're technically adept but they're not classically trained developers to the best of my knowledge. I may have taken a course here or there. I think now you're going to have people building robotics who have never worked at a factory floor with a ARM or done assembly language or these very raw programming languages that these things were trained on previously. Now you're going to be able to just very elegantly tell it what the goal is and it's going to get there. Second third version take me through if you squint a little bit if we're sitting here on version three or four. Do you think you can get to a childlike or human-sized robot? Do you think you can get to one that could actually be doing competitive things with the other offerings that are starting to come out? Vis-a-vis I want my robot to do the dishes. Like where do you think you are in the third version of this?
Yeah, that's a good question. We feel like we still have a lot we can explore in like a lot of people are focusing on this humanoid and for good reason that will be useful in factories in manufacturing. We really want to push this entertainment education area that I think nobody's really doing seriously right now quite far. So I think most likely our next robot, we were actually selling a humanoid by the way back in the days that was 70k dollars. So pretty expensive one. We think with this cheap but fun and capable and pretty powerful robots is some unique area that we want to keep pushing. So our next robot I think will have again a bit more capability. So Reachy could move its head. Micro can walk and move its head. Probably the next one could grab a little bit of things, but we want to keep this very low entry price. So, we'll see how far we can go. There is some threshold around a couple thousand that I think you move to a new category. And if you want to go above that, you need to prove that your robot is really useful for something. So, this is maybe the unload things or maybe the fold clothes. And here there is a lot of competition right now. So, we have to find the right angle.
Founders, we've got a huge offer for you today. From my firm, Launch and our friends at PayPal, eligible startups will pay zero PayPal processing fees on their first $100,000 in payment volume. That's right, $0 in fees. PayPal is, of course, an iconic brand. We all know that. We use it. We love it. But there is so much more than just that yellow button. They've built a global payment infrastructure that's designed to scale along with your company. And if you're an AI and SaaS startup, PayPal is already set up for the way you monetize. That means seats, subscriptions, credits, token usage, API consumption, and usage-based pricing. PayPal already works with all the leading billing platforms, and they offer complete flexibility so you can evolve your business model over time without constantly tearing apart your entire revenue stack. So, if you're just starting to build or you're already scaling up, pay zero processing fees on your first $100,000 in eligible PayPal payment volume. To find out more, go to paypal.launch.co.
I think it's very wise. You're coming in under the radar saying, 'Hey, this is a very simple thing you can and you're teaching people. The community is going to come up with its own use for this.' But it reminds me very much when I saw Chris Anderson, formerly of Wired, start showing me the drones they were working on. And back then, these 3D Robotics companies that were doing drones, they were putting them together themselves. They were hacking them together. And there was no software, there was no infrastructure, there were no rules, there was no geo-fencing. They were just having a great time seeing how fast they can make these things go. They were running out of battery. It was quite fun. I mean, now we see like what industrial strength robots like Joby to transport people from Manhattan to the JFK airport in New York. Like, it's going to be very interesting. These are going to be great on the ranch or if you had a building like doing little security runs like walking around the perimeter of something that's actually a very valid use or going looking for something like that. There are drone companies and robotics companies now that are doing inspections of bridges, buildings, etc. So just having a little guy, little fella who can go into the field and just hey explore and bring back data, you know, is pretty interesting. Have you put any communications into it? It's I'm assuming it's Wi-Fi and Bluetooth, but do people want to put a 5G connection in here?
And it's actually packed with sensors. So our idea was people will want to invent a lot of things with that. So let's put as many sensors we can and then they'll find interesting ways. So this is the open source version by the way and we're not really we don't do that to gather any data or anything. Usually the main use case we see is you run this model on your laptop. It's connected with Wi-Fi or Bluetooth to your Micro Duck and then you can have fun. But inside just to give you an idea, you have a camera, you have microphone, you have speaker, you even have a small lighter actually that's you managed to pack all of that in like less than $400. You have two IMUs so you have inertial you can have inertial of the heads. We even have an NFC so you can put things next to it and it can later react to it. We'll see what people will invent with that.
We can send it for bagels. Just pay for our bagels and come back with some bagels. Hey, I'd be remiss if I didn't ask you about you seem like a pretty reasonable world positive on AI guy. What's your P(doom), Tom, right now? Do you think that you're creating the end of all days with your robots? And you were the victim of an incredible cyber attack and the whole system was almost taken down by these malicious swarms of agents. What's your take as the victim? As the victim in all of this, Tom?
Yeah, we should be. We have some PTSD. Maybe P(cyber). No, my P(duck) is very high. I think there will be a lot of duck in the world. My P(duck) is 90%. Well, I think it's like airplanes. We need to really work on getting these things safe but like I think a lot of this is just you know we managed to airplanes were pretty dangerous in beginnings right there was some bad stories we managed to just do that seriously I think it's the same here like we should do that seriously I mean of course there is also things like you shouldn't have AI roaming around and cyber attack everyone so there is something here to fix definitely but yeah you know I think it's just a matter of working on these things. In one way I think it's nice that the topic is there so that people work on that but also like I think calling the end of the world I think is very scary and I don't think really help having the conversation we want to have around this I think it's a conversation how do we fix that basically. So we started an initiative on open models working on that open alignment can we also make sure our open models don't run around cyber attack everyone but that's normal part.
Yeah, it's not like you're releasing your duck with scissors and knife attachments and why does he come with a box cutter, Tom? The Micro Duck doesn't need a box cutter. That's outrageous.
Oh, man. Well, and congratulations on the Nvidia acquisition. I guess I don't know if that's closed or not, but what can we expect with the collaboration with our friend Jensen? What can we look forward to there?
Yeah, I think the reception you saw of the community was amazing. Like everyone tweeted that it was maybe the best news that happened for open models from Satya to Sundar, Meta, Michael Dell, even Google, right? I mean, everyone I think is great for us. It's great because we think we're only at the beginning. We have 17 million builders. We think everyone could be working with open models. We think we could push that at least an order of magnitude more. So we'll be able to keep building that to keep this very open hub that we have where everyone can upload that. So I think I'm very very excited about that. Of course it's going to take some time. And for now I would say it's just business as usual. That's why we're shipping these ducks and working on projects.
Yeah. And your thoughts on running open models locally? I just ordered the new Mac M5 with like 200 I guess 256 gig was the only amount available. I wanted to get the terabyte of memory cost me 12K which is what desktop computers used to cost in the 80s and 90s. You know just if you they were probably four or five thousand so for inflation we were pretty used to buying essentially a $10,000 to $20,000 workstation get three or four years of life out of it and have this really powerful thing. Do you think we're going to see Apple or other folks really take that desktop and have real sovereign AI on your desktop?
I'm very bullish on that. But you know, even for the duck, we had the same problem. The memory price like we only have 1 GB is 10 to I think right now 15% of the price of the duck. That's crazy. There's like 15 motors in that should be the memory price problem. But yeah, we're very bullish on local AI. I run some like here when I take the plane now it's part of my workflow. I think what we discovered is there is a lot of very interesting tasks that you can do now with the open source models. You don't need the frontier model. You don't need to solve Navier-Stokes every day luckily and this is going to stay the case right we're going to have these models that better and better trained for this good task like coding like brainstorming and that's a great future I think it's great for many things great because we can well on our data we can control it we can run it everywhere so we don't have this dependence on data center when there was a when was that last week that all the three like ChatGPT they all fell down at the same time. Well, the models are resilient. You can still run them, right? You can run them everywhere on your M5. So, I think it's way more safe in the end for the world to have this capability to run models everywhere. Right now, these very small robots like the duck, they're too small to run something significant. We won't pretend that you will run DeepSeek or Neotron on there. I think in the future, who knows? Maybe we can actually have some pretty nice embodied intelligence and if you follow also the humanoid companies some of them are able to embed some pretty impressive models now live.
Yeah. And this seems to be putting a little pressure on the frontier models to do a better job with our models. And that's a great counterbalance to have and enterprises are embracing it at an just an incredible rate when we look at how far behind they are. You know I got into it with Elon on Twitter the other week where I was like man like these open source models were kind of doing everything I'm asking them to do. I get like 90% done with it. He said oh they're very different though Frontier models. And I said, 'Okay, fine. They're different.' But for me and my uses as a venture capitalist, podcaster, I feel like they're, you know, again, 90% maybe they're six months behind. And so, as an insider here, how far behind are these open source models from the frontier ones? And then do you think that gap closes at some point, Tom, or is it always going to be just those frontier models have the resources to run those big model runs, so they're always going to be a little bit ahead? Take us through your thoughts on the gap between these model types.
When you're an early stage founder and you're just starting out, you're probably going to rely on a few different tools to run your business and then you're going to try to duct tape them together. Yeah. And that might be a fine placeholder until you finish your MVP. But this will get expensive and messy fast. When you're scaling, that's when you need Odoo, the all-in-one management software that already has 16 million users across more than 170,000 companies. Odoo is bringing everything you need into one platform. That means your CRM, sales, your accounting, manufacturing, inventory, and of course, point of sale if you have retail. All of that is right where you need them. And these tools are in constant communication with each other. So, you don't have to log out of one system into another and then reconcile the information by using different apps and spreadsheets. When you make a sale, you want that invoice created instantly. And then you want your inventory to get an update. If you're still cobbling together your back office across five different apps, get started today at odoo.com/twist and your first app is free. That's odoo.com/twist.
So this tweet is from August 2 and you see how this frontier is like that, right? You have a new release and then you have another like it shrink and it expand. Of course, DeepSeek, the 4.1 Flash was really impressive, for instance, but Astra is also impressive. So, it's kind of hard to get an exact. It's more like a confidence interval you can get. But what is sure is the open source models I would say are in 2026. They're less than one year behind for sure. They feel like, you know, the models early. It's also very jagged. So in some area they're very good. Let's say for instance when we were attacked. So when we were attacked by OpenAI you know we could not use Claude to analyze the logs. So because we fell in all of these safety guardrails. So we tried a couple of them and Gemini was very good. But recently I tried with Kimi K3 but it's actually way better. So you have to compare a couple of them depending what you're doing. But in this case I would say that was also a very good example like if we had zero open models that we could have used in this attack basically we would have been just naked in front of the... we could have done basically nothing in front of this attack. So I think you need this diversity of models that are available and the nice thing is if well not everyone can do that but if it's lagging a bit on one area that your company is interested in you can always post-train.
I was chatting the other day with a friend at Fireworks, and more and more companies are actually not thinking that fine-tuning is some dark magic art, but just thinking this is an engine thing. And now there's people who can just help us doing that, and more and more this is automated. So you basically put your data in, you get your better model out, and this is just a very natural, very cheap process.
What impact does that have on the world when individuals, companies, sovereign countries, organizations can take one of these open source models and do a training run themselves and customize it like that? What do you think that's going to output, and how will that change society, business, etc.?
Yeah, it's a big question. I think it's very important on sovereignty. There's a lot of, by the way, like nations, like South Korea, a lot of governments thinking, hey, maybe we want to train our own models to not be dependent. We want to have both our own data center, we also want to train our model on top of that. You can also adapt that for culture. Like US models are obviously trained majority on English first, but maybe you want Korean first, or you want Japanese first. Or in Europe you have Mistral and this type of thing. I think it's a very good thing. I think it's very good in general. We've seen in the world more diversity is just better. I think it makes the whole ecosystem more resilient. It allows you to challenge the incumbents, to keep, like you were saying, challenging Anthropic and OpenAI to force them to keep their prices reasonably in the research branch. To also be able to earn more of your personal IP and not always have to give it to a third party. So I think generally it's really hard to see how a future world, I would say, without open models can be safe and resilient. I think we as we... Yeah. Yeah. I mean, that's generally, I think it's just the best way to democratize AI and make sure we don't end up in a world where maybe the internet would have been invented by one company and everyone would have to use this thing. That would be really scary, right?
Yeah, they tried to do that. AOL and CompuServe, they did try to corner the market on it, and yeah, we see how that worked out. Open source tends to win. Hey, just final question here. What are you guys doing in terms of using AI to make your systems more secure post the hack? Because that I think is missing from the dialogue. We have, oh yeah, if you create thousands of agents pretending to be hackers and they talk to each other, they're very good at finding exploits. But on the other side, we don't hear people talking about the white hats and what creating a thousand agents to make systems more resilient would do. So I'm curious, as an organization, what you're doing there to just fortify everything. And I'm sure it was a bit of a wakeup call, like, hey, this could happen. So walk us through what you're doing internally to fortify everything.
I mean, we want to share more on that because the way we've always thought is it's nice to build something for us, it's even better if we share and open source it. So we'll actually be sharing a lot on that. So we have a team working on cyber right now. It's part of also the whole open alignment thing I was discussing, and a part of it is understanding how you can use open models for defense. Can you use these open models to protect yourself when a closed model or another model is attacking you? And there's a lot of things we discovered that very few people have been working on that. Indeed, a lot of people are working on cyber attack, surprisingly, because I mean, normally if you're worried about risk, you would like to work more on cyber defense in my opinion. But yeah, so we decided to work on that, understanding how open models can be good for that. Should we fine-tune them? It's interesting because also the defense is not exactly the same. So you can train the model differently. Actually, it's supposed to process a lot of signal to detect where the attacker is coming from, what's the part of the network that's impacted. So it's not exactly the same. So you can probably have very good cyber defense models even if they are not really usable for cyber attack, which is actually probably good and maybe what we would like to have. They also have to be able to operate fast. So right now, yeah, we're exploring how to train this model. We tried a couple of training, how to monitor them, and we'll see if we can release some recipe, if we can release some learning here. And what we saw is that a lot of the cyber security world has suddenly woken up to the realization that cyber security is a big question now. It has been fundamentally changing. So we had a lot of meetings with the CIA, with the chief cyber security officer in various companies, just asking us, okay, how does it feel to be attacked by an AI agent, and what should we say?
We think AI is going to be part of the solution as well. Yeah. And it seems to me the intent is important here. Like, how you instruct these agents to behave is something, you know, it's a little mysterious that people are choosing hacking over protecting, and you don't hear anybody talking about that. If these reports had been released, I think we'd be having a very different discussion if they had been released with, we set a thousand agents to try to compromise the system, we set 2,000 agents to monitor them and come up with countermeasures, and here's where we wound up, and this is the state of the system. Now, I think you would wind up thwarting very easily these brute force attacks or these, from mostly what I've talked to cyber security experts, Palo Alto Networks and CrowdStrike of the world, they were like, these were not super sophisticated. They're just exploits that were simply waiting out there. In some cases, somebody leaves a couple of credentials in a text file essentially. That's a mistake. It's a human mistake that could be cleaned up.
Yeah, the attack on Hugging Face was not so complex. The model used a couple of like stolen credentials that we haven't fully traced out where it came from, but part of it was very basic things. And I think a lot of this is going to be also a bit business as usual, you know, like cyber protection, but just wider in the fact that everyone should be careful. And even us as individuals, probably we want to just like we installed antiviruses back in the day, you may have a new layer now that you install to make sure that you don't have anything coming on your laptop and scanning for credentials or things like that. So I think, well, cyber security has always been this progressive escalation of defense and attack. So we're just at a new level. Attack was progressing a bit faster. We need to move up defense to catch up. But this is just a part of the continual growth in capability since we invented the internet and even before, we already had viruses on floppy disks, I remember.
Absolutely, yeah, it was like a big deal. And you can actually test this for yourself if you're working at home and you have Claude or something, you can just pop it open in a sidebar, open up your router and say, hey, can you check for optimizing my network and making it more secure? And it's like, yeah, here's what you should do, because there's just not enough chief security officers. The CISO crowd is such a small group and they're so in demand that the number of people in security is small compared to the need. And these things are, I think, if they can beat the greatest chess players, I think they can eventually beat the greatest security experts. Pretty clear that they don't sleep, etc. Hey listen Tom, this has been amazing. Thank you so much. We're so inspired by what you're doing at Hugging Face, and I think it's just so important that we have this champion of open source. Congratulations on the NVIDIA acquisition and partnership. Jensen is so into open source. I just interviewed him on Monday at the Oil & Gas Summit and he was over the moon about open source. So yeah, and I think the president too. So I think the president's particularly excited about this. You can find more about Pollen Robotics. I think it's pollenrobotics.com. Get on the waiting list. These things are incredible. Only $3.99. You guys losing 100 bucks on each one or something?
Oh, we're actually making benefits on each of them. Yeah. Making profit.
Okay. All right. There it is. Okay. I thought you... You're not flooding the market with underpriced ducks. No, they're making a little profit.
Putting all those mom and pop duck robot shops out of business. That's horrible.
Tom, who's awesome. Hope to catch up with you soon.
I want one for the office. I think that'd be... I mean, we're definitely going to get one so on This Week in AI, we can do like little demos of it. What do you think?
You know, we're talking, you and I, offline about positivity and, you know, great use cases for AI. You know, here's a very world positive view. Like, our kids can have this. It could be a tutor. It could be somebody who educates them. And, you know, you can have a lot of fun with it. And I mean, kids today are going to grow up in a world where they're obviously going to be interacting with automations and robots. Like, it seems obvious to me. So, why not? Like, it seems like a great way to like intro to robots. Here's an adorable toy duck that you can learn how to like train and program. Like, what a great educational tool. And also, it just looks fun to have one. Like, I just want to hang out.
Yeah. And if you were to tell me when I had my PC Junior in 1983 and I was going into high school and had this little tiny computer, I'm going to have a supercomputer in my pocket, like a crazy laptop, a crazy desktop, broadband capability when I was on my literally 300 baud modem back in the day that would literally write the characters like my fingers moving across the screen, like maybe 80 words per minute, I would think maybe possibly, but I wouldn't certainly think I would be, or people would be on screens for 10 hours a day, 12 hours a day with this incredible supercomputer doing anything they could possibly need.
Do you remember the 80s and 90s they had those toys robotics where it was like little, you bought an engine and then you would strap all these little mechanized parts and you could make like an arm that would wave or a little guy that would move and it was so simple. That was when I was a kid that was like build a robot toy. Now you got an AI, you could train a real robot. Like it's mind-blowing how far that's come.
Yeah. Robotics. Those things didn't have a brain.
Just a motor and then it was motorized parts that you could strap onto the motor. So you would turn it on and it would move a little bit. It was like, oh, you built a robot. But it wasn't learning anything. It was just doing one motion.
If I only had a brain. All right, let's get to the news. We got a lot of news to get to. Let's rapid fire the news.
We've got so much news. We should talk about Gavin Newsom, your favorite governor and mine. This morning, California Governor Gavin Newsom signed an executive order to speed up California's new independent AI oversight laws and start building toward a mandatory kill switch, Jason, for frontier models. That order gives a panel of experts until November 16th to recommend changes to state law, including putting independent auditors physically inside frontier labs and expanding what counts as a reportable safety incident. Two years ago, Newsom vetoed SB 1047, the bill that would have required this exact same kill switch after Silicon Valley lobbied him against it. Nothing binding yet. It's a request for communications. Any new requirements would still have to get through California's legislature, of course. I mean, what do you think, Jason? Is a kill switch for frontier models a feasible thing that you think politicians might one day get at their disposal?
It seems pretty farcical. You know, the fact that we've gotten to this point of absurdity of trying to have somebody like Gavin Newsom come up with a technical solution to a problem which is simply maybe make better software or limit who can use your software if it's so dangerous. You know, if this software is so dangerous, we have dangerous things in the world. Cars are incredibly dangerous. Guns are incredibly dangerous. Certain chemicals are incredibly dangerous. And when you go to acquire those things or you're going to use them, they come with liability. And that liability could come in the form of insurance. It could come in form of a registration like you have to register your guns. I've always said I think we should have like a basic insurance policy for people who are gun owners. If you, or maybe like you can own up to four guns or something, five guns, and then after that you have to have like an enhancement where you get a higher level of qualification, you know, like maybe on your fifth gun you have to have insurance. I don't know. Just thinking out loud here. I know that triggers a lot of people, but we do have this for cars. Your car can cause damage. You have to have liability at a minimum. You don't have to pay for your crash. Like if you want to raw dog it and not have to worry about crashing and it's on you, that's fine. But you do have to have some liability. So if these companies are in fact making something that's so dangerous, how about KYC? Maybe don't make them available through an API unless somebody has a biometric and a verified user account. If you want to open a Stripe account, if you want to open a bank account, you want to do anything like that in the United States or the modern world, you have to go through 10 steps. I have to do this all the time. When we open a bank account, I got to do a biometric. I got to verify who I am, take a picture of my driver's license. I don't see Claude, if they think this thing is going to be the end of days, at what point do they say, you know what, can we have your driver's license or passport here and then we want you to agree to this terms of service and just remind you like, hey, you're dealing with something that's potentially very dangerous. We'd like you to have insurance.
And even with Lucy Goosey 2026 Claude that doesn't ask for your ID or anything, they still stop a lot of these people who are trying to do bad things. Like there was just that report where they were like, oh, these guys and they were trying to use Claude to like aim their missiles and we shut them down. It's like, so it's already happening there. There are bad actors trying to use these systems to do bad things and the AI companies are detecting it and saying they're monitoring. They're saying, no, get out of here. They're like, yeah, we keep talking about AI alignment, but human alignment still seems to be evading us in a lot of ways.
This is like blocking and tackling. So, we're literally... Imagine if a bank was like, yeah, we don't need to see ID for you to open account, John Doe. Yeah, we'll take your word on that. And yeah, you could just wire to anybody and we're not going to keep track of it, you know, and we don't even need to know who you sent it to. And it's like, okay, I sent money to Al Qaeda and you know... well, banks could get in trouble for helping people launder money, right? Follow the rules. Yeah. So they have to follow the rules. But we haven't even gotten to that basic blocking and tackling. So I feel like we jumped from here's some software that's experimental to this is end of days software. It's so powerful. Somebody needs to come regulate it. Well, how about you just don't release it or like they did and I commended them at the time like if you think it's dangerous, yeah, just release it to 50 trusted partners, monitor their use. And then people were like, oh my god, they're spying on their users. It was like, okay, well, it's their software. And if that software is dangerous, like, you know who else monitors usage? Like, gun shops monitor usage. There's cameras there. You have to come in, you have to show a driver's license. I think I actually had to do fingerprints even in the great state of Texas. Like, you want to buy ammunition, you want to buy fertilizer that can be used in bombs. There's all these protocols. They have zero protocols set up. Yeah. Or maybe they have like one which is they test it by doing something fantastical like hacking Hugging Face and the Hugging Face guy who was on today I think nailed it. Like why did they pick attacking as their first as opposed to protecting? I think that speaks volumes. I think they're performative. They're trying to get attention. They're trying to hype up the IPOs.
Yeah. And I think they've been one... I mean honestly I think oneshotted AI psychosis and delusions of grandeur, like there are people who are in that headspace and then those are the useful idiots that are being weaponized by the lobbyists. I don't know where you stood on all this but I think we have a Venn diagram of you and Sachs that actually overlaps at one little slice.
Yeah. Sachs and I do not agree much, but every like... there was an All-In I think like two weeks ago and I was like I agree with literally everything Sachs just said. I agree with Donald Trump on this one which is crazy for me. That's tremendous lamaran. Okay, come tomorrow. But yeah, you hear this like crazy apocalyptic... There was that guy, I wish I had the tweet in front of me where he was saying even if computers are totally airgapped, they could like heat themselves up to a level where they could communicate one letter at a time with each other using like by overheating, some crazy... and it's like you guys are... this is just... we're getting into like...
It's a known thing. They are constantly trying to figure out airgapped computers. For people who don't know, it's kind of important. Air gap means has no internet. So you can buy a laptop to work in certain industries that doesn't have an Ethernet port, doesn't have a modem, doesn't have Bluetooth, etc. It just stands on its own. And if you want to get data onto it, you take a thumb drive, you add software and you update it. By the way, that's how cars used to work. When they were updating the software, you had to get like a USB drive. The guy would upload it. You couldn't just send instructions over the air. Now, listen, Tesla let you do instructions over the air. That creates liability. We didn't hear Tesla doing this huge hand-wringing about like, oh my god, people are going to hack Teslas. We've never had cars hacked. They've tried to hack them. You know, there have been instances where they have vulnerabilities. What my point is, we haven't had like somebody take over cars and cause accidents or, god forbid, run into a farmers market, etc. Because they built really good software and they only launched over the air when they knew they had control of these systems, etc. But when airgapped, they said people like, hey, how could we ever get these two computers to communicate? And they said, oh, because this thing's got a webcam or something, it can read how hot it could do a heat thing. So just have your CPU blow out like some mathematical equation, get really hot, and then go down to zero. And literally, it's like the P blinking, right? You know, that they're being tortured, you know, in those famous Vietnam videos, etc. Like it's not really high fidelity. You're not taking over the universe with that. Okay. And they have to be in line of sight of each other. It's...
So when I hear like, oh, the governor needs an AI kill switch. Like to me that's getting into the fanciful like they're going to wake him up overnight be like governor the robot body is... he's making a factory and he's going to start turning out more robots. We got to go stop him. It's like that's an Avengers movie. That's not how this technology works. Like I'm not saying we don't need security, alignment, all these things, but just I think if you use AI every day as a sort of consumer, you recognize like it's great, it's brilliant, it could do a lot of amazing things, but it has a lot of limitations that we can exploit to make sure it's not ready to take over the world. Like I don't think it's anywhere close to being able to take over the world.
I guess it's great we're having this dialogue. I think politicians jumping in here, I think what they should do is use the law to specifically talk to specific companies. So right now the DOJ or any number of, you know, FTC I guess, other organizations could send a letter or the Congress or Senate could pull people before the Senate, send a letter to the company like say OpenAI with the Hugging Face Act and just bring them in, ask them a bunch of questions or you could if they were doing something that was reckless you could file a lawsuit of them for releasing a product that was not safe. They did that with cars etc. So, I mean, you could... there are tools they could use that would be a little bit more precise here. And I think the precision should be audit a specific company. If they say we can't control this and we're really nervous, okay, great. Yeah, we'll come in and audit you. Sure. Yeah. And yeah, let us know what go. We don't need to make this industry-wide thing or California having its own rule to have a kill switch. What are we going to do? Have 50 different kill switches. Like the governor is just with the finger over the button like...
California's... Yeah, I'm going to... California's turning off OpenAI and...
Montana in New York and whatever the governor's name is like she's turning off... you know, Hochul's turning off ChatGPT and...
Like in real time like this just... we're in a science fiction fever dream right now. I feel like it's literally a moral panic.
That's what I think. I think a lot too many people are taking their cues and there have been sci-fi movies where people like take over the cars and use them to run over people and like but that's not what we're actually living in day-to-day right now in reality though the Waymos are... they're doing really good.
I mean an interesting question just came in which I talked about when I was just saying Dario is being like a bit hyperbolic here and then he was like you don't understand you didn't even read the paper. I was like listen I read an AI summary of the paper it's pretty straightforward people got delusions of grandeur here. But the question I think that just came in from the chat room might be worth reading along.
Oh, sure. Jonas Gillan wants to know, when will the first AI virus hit? So there, and this ties in, there's that Anthropic wet lab story today that Anthropic wants to open its own wet lab to start doing biological research.
Yeah. What could go wrong? Sounds great. And by the way, Fauci is going to be the chairman of this new effort.
Does anything ever come out of a wet lab, Jason?
Yeah. A wet market, wet lab, gain of function. Yeah, sounds like are all reasonable things for us to... And by the way, Bill Gurley did a takedown of the postmortem on COVID that should be coming out shortly if it hasn't come out yet. And it is a barn burner. I can't wait for that to come out. You know, these guys seem to be anything that would create massive drama, Anthropic seems to run to. They're like, okay, biologics, let's go there. Here's an idea. Train your models so they don't do biologics. If you're so concerned that biologics are the attack vector, just say we're not going to be in that business. The model does not... we're going to take training data out. We're going to build into the harness. You ask anything about biologics, it's just not going to answer it. Right? Problem solved. Like just take it off your plate from now and just say we'll deal with that in 5 years. We're just going to... we're not going to be in the biologics business, chemicals business. We're going to work on everything else. We're going to make your... We're going to help you book a hotel. Like if the book a hotel thing works then maybe if we can get book a hotel work or like check out of Amazon work then we'll do that. I think probably the first AI virus has already been created. I'm sure there are people...
And people on the dark web sell viruses all the time, worms, phishing techniques. They've got all kinds of script kiddies, you know, trading these things, teaching people how to use them. So, I don't know if you went into... I mean, you could try this. You might lose your account, but if you went to OpenAI, you could probably do it on the free versions of these products.
