Good afternoon everybody. Welcome to Entrepreneur First's office in San Francisco. I'm Alice Bentinck, the CEO and co-founder of EF. I'm delighted this afternoon to be joined by Jack Clark, one of the co-founders of Anthropic. We are going to get to the pitches in just a minute, but before then, we're going to have a brief conversation about being a founder, AI. And maybe there'll also be some tips in there for the investors in the room as to where you should be putting your money today. Jack, I'm going to do a brief bio on you. Co-founder of Anthropic, probably one of the fastest growing companies of all time. Recently raised at a $183 billion valuation. When you consider that's a five-year-old startup, just hit our fifth year anniversary. Pretty crazy. Jack was previously at OpenAI and before that was a journalist, with a particular love of data centers, right?
Yeah. I used to cover data centers around Europe and the world. So turned out to be useful.
What's your favorite data center?
Oh, there's one in Iceland that I got to see which was powered by a geothermal power plant. So I got to see the Bláfjall geothermal power plant and then a data center which I think was for the game Eve Online. It's a cracking data center. A cracking data center for your next holiday to Reykjavik. Take a little detour.
He also writes an amazing newsletter called Import AI, which I think we'll be talking about in just a minute. And also an avid pool player.
Best pool joint in town? I'd rather not name it because it's where people don't ask me about computers or Anthropic. But I'm an avid middling pool player, to be clear.
We have some of our founders in the room who are going to be pitching in just a minute. I think what we're going to start off talking about is the founder journey. Did you know you were going to become a founder? You started off as a journalist, went to OpenAI. When did you think that founding might be the right thing for you?
I think once you start to become extremely opinionated about how things should be done and you have a really clear vision for it, as many of us the Anthropic co-founders did, that's your cue that you should move on and do it yourself. It was similar to when I was a journalist and I started to just go home and implement neural networks and be training them and talking to people about how you train them. I tried to get Bloomberg to cover Nvidia in 2015 because every research paper I read showed Nvidia GPUs being used in the AI research stuff and I said this could be important for the financial story. It didn't show up in Nvidia's numbers at the time, but that was sort of my cue to go into AI because you've stopped being a particularly useful journalist. You become very opinionated about a niche thing within AI.
You left OpenAI, started Anthropic, and then have been on this totally crazy journey for the next generation that's coming through. Do you have any advice? What are the things that as you're building this company, if you think about generating that level of pace and scale, what's your advice for them?
Yeah, I think there are three major reflections. One, nothing is anyone else's problem. So when we started Anthropic, there are a few of us as co-founders and when there are problems, we all had to solve them together. You have this learned response that 'oh that's someone else's problem.' But if you're the co-founders of the company and at the beginning where it's just you, it's your problem. If there isn't something that the company knows how to do, you need to have the responsibility to kind of zero to one it. The other was just how quickly you need to focus on scaling yourself. At Anthropic, we found that the moment you know how to do something repeatably and instruct someone else how to do it, that is immediately your cue that you should have hired someone a few weeks or months ago to do that job for you. You should always be doing things that are without a process because they're new, things that require you to kind of zero to one something within the organization. At the same time, we found it's really important that everyone is still extremely in the details. Today I was working out how to build a new type of evaluation for a property of one of our AI systems and I spent the day talking about all of the internal data dependencies and our eval stack and everything else and looking at the code for how we pull evals from one system to another. Being conversant in that even as the company scales ends up being really important. You can't get too far away from where the actual work happens.
It's like helicoptering going from the big picture back down into the details. One of the really tricky things I imagine for you guys is that you are making big bets in terms of capital intensity in an environment that is moving very fast. Partly because you guys are driving the environment to move so fast. How do you work out where you're going to allocate how Anthropic invests in the next iteration, the next model?
Well, with Anthropic, it's relatively simple. You just have a big pile of compute and you spend as much of it as possible on training your next model and then you really hope that it works. You're sort of only as good as your last model. Does anyone remember Claude 2.1? Few do. It was not an especially good model, but we quickly followed with the following Claude section, so it ended up being really nice. How we figure out how to make these bets though is interesting. You have to figure out how much you want to de-risk your projects versus take the big bet. The longer you spend de-risking, the more confident you're going to be that the bet will pay out. But the longer you spend de-risking, the more the whole world around you is moving. We're in a competitive environment with other frontier companies where you're constantly trying to be on the frontier. So you have to find this kind of Pareto frontier trade-off between how much de-risking you do on your big training run versus landing the training run so your model lands in time for you to be competitive. We've got this somewhat dialed in now. Part of it is figuring out how to smartly spend your spare compute when you're not training the big model, or if you are and there are other parts of the cluster that you can reallocate for doing de-risking runs for the next set of models. So there's a whole assembly line that you need to build and then you need to parallelize it over time so that you're ready for the future models.
Nice. Let's maybe dive into that. If considering all of that, if you were to found a startup right now from scratch, do you think there's a particularly interesting area where you would bet your time as a founder?
Yeah. Here's a thing that we found that I think is robust. It's that for any given model from any company, you can get about half a model generation into the future. In other words, you can maybe see 4 to 8 months into the future of AI capabilities by building a specialized scaffold to elicit model capabilities for a given domain. Companies like Google have done this with a framework called Big Sleep for eliciting cyber security vulnerability testing capabilities out of their model. We've done experiments in the cyber domain and the bio domain inside Anthropic. Other companies are doing the same. I find pretty reliably that this lets you elicit capabilities from your existing model that are half a model generation into the future. Then when those new models arrive, you often find that this scaffold, big chunks of it, will now elicit even better capabilities and allow you to go forward in time. This generates huge amounts of evidence to help you both de-risk future things you might do but also immediately gets economic value out of these models in the present in excess of what the base model does. We work with a whole bunch of startups that are basically experts in a certain industry and they've built a load of scaffolding and elicitation stuff. I think people used to call this wrappers, but we've seen that it's actually shown up as a way to build successful businesses because they can immediately do better work in that domain than any of the frontier model companies and they're able to stay ahead because they're constantly improving their own scaffolds.
Are there any scaffold startups that you've seen, you don't have to name them, that you think are particularly exciting where you personally get excited by the problem that they're solving using your model to solve an amazing problem?
Yeah, I mean two domains. One, cyber. It's very clear we've reached an inflection point in how these models can be used for kind of cyber defense and also offense, but typically that's less commercially relevant. We also see the same as being true for some things like the legal domain as well. I personally am most excited about AI for science, though I think that's not currently a source of tremendous amounts of revenue, but it's very promising. It's also promising because it involves bringing your models to entire software systems that they may not be that expert in, which adds value at the scaffolding or adaptation layer. You often find that models can be taught to be surprisingly conversant with certain obscure parts of the science tech stack and start doing work that we would not anticipate as an originating frontier model company.
Well, let's talk about how when you are personally evaluating a new model, when you have a new model in front of you, what do you ask it? What are you trying to work out? How do you push its limits? What do you do?
Yeah, I'll go from the normal and quantitative to the subjective and qualitative. The normal stuff is just the standard set of evals which all the companies compete on like SWE-bench, GPQA, what have you. The second phase is using your internal scaffolds to see how much further this new model is able to push things. The third phase is also using maybe held out tests that aren't on the internet, some of which you develop for policy purposes. One of the teams that rolls up to me builds things like bioweapon tests. That means that we can do wet lab uplift trials with the new models that let us discover interesting properties about how much more advanced they are, though that takes a bit longer. Qualitatively, I've actually found that every single founder or senior person at Anthropic has their favorite seven prompts that they give the model to see if it's got better. Our chief scientist Jared Kaplan once said, 'I don't care about the evals. Just let me talk to the model for 15 minutes and I'll tell you if it's good or not.' I don't know if that says something really complimentary about evals or about Jared, but it's hard to tell.
Are we allowed to know what those seven prompts might be?
Well, I used to have a prompt but it got outmoded. In the early days I used to ask the model when we trained it how many helicopters could a human eat in one sitting. In the early days the model would be like 'probably two or three helicopters.' Then one day the model was like 'that all depends, is it a multi-course meal or just a single course meal?' Then one day the model said 'this is a trick question. I don't think a human can eat any helicopters at all because they can't digest the ingredients of a helicopter.' I think that was maybe early 2023 and I posted internally I was like 'guys it passed a helicopter test. It's a big day.' In 2022 someone had a test for seeing if the model could rhyme or not. 2022 was the year where models started to be able to rhyme decently. You develop all of these kind of things. Currently, jokes is a good frontier. Jokes are innately very hard to do. Models have only just got good at what I'd think of as dad jokes at a wedding level of humor, which is not very good. I think if we found models that suddenly were able to tell really amusing stories that didn't feel clichéd or anything else, it would tell us something about their theory of mind and how good they got at understanding the normative space they need to do work in beyond just coding.
Well, let's talk about where we are in the arc of AI progress because there is an enormous amount of capital flowing into AI companies at the moment. And hopefully lots more flowing into it after today's demo day. Are we in a bubble?
No, I think we're in the... I think people continue to underhype AI, as strange as that sounds. The reasons for that are twofold. One, the bets which the AI industry is making are just tremendous. We're building stuff in terms of capital expenditures at the scale of multiple space stations, multiple James Webb telescopes. The James Webb telescope cost about $10 billion. You're seeing probably close to $100 billion in aggregate investment this year for frontier AI model training. It'll be hundreds of billions of dollars next year. You may get close to a trillion before the decade is out. We are seeing things basically without precedent other than wartime efforts from government. And the models can productively turn into doing useful stuff in the world. So you're going to do all of this compute for training successive model generations, but also the value of these models once they exist just for basic inference is kind of tremendous. So we're going to see continued amazing and startling growth here. A useful way to think of it is when you work with people in your life, you try and delegate stuff. The limit on delegation isn't a single task because you're always delegating a big blob of work. It's basically how long can that person work before they come back to you and say 'I've hit some kind of blocker. I need you to unblock me. I've hit some permissioning thing or I need you to go and talk to someone else in the organization.' If you look at the time horizon over which AI systems can fruitfully do tasks, they are massively pushing that out with each model generation as well. That means that we're seeing this tremendous commercial growth in relation to basically AI systems which you can confidently delegate to. Delegation is where huge amounts of economic value exist. I think we're all underestimating how profound that will be. The other point is that AI goes through phase changes. A year ago or so, maybe a year and a half ago with Sonnet 3.5, we clearly went through some inflection point on utility to coders. I think that's when you saw this coding ecosystem suddenly grow and it grew from other capabilities at other companies as well. We're going to reach that inflection point in other domains beyond coding as a consequence of something about model quality over time, and that will unlock entirely new sources of growth for the AI industry.
Well, let's build on that because LLMs are underhyped but they're also still underutilized when we think about both at an individual level. I use them daily, but I'm probably still not fully using their maximal version of their capabilities. And then in the general population, most people probably aren't using these models every day. What are the blockers? Is it product? Is it tech? Is it cultural? Is it habits? How do we increase adoption?
Yeah, the phenomenon we see is very young companies that are building around the notion that AI exists can adopt this stuff very quickly and seamlessly. Also, they're younger, smaller companies, so they have less organizational bureaucracy or process disease. Large companies can do it if they have a top-down commitment like a CEO and or a board that says we absolutely have to do this and then crucially a layer or two layers around the CEO that actually go and then do it and are also kind of AI-pilled. Most organizations struggle. They struggle because getting the most out of AI is often related to giving it as much context as possible. Organizations have built a whole set of processes especially for data handling which makes it innately hard to bring back context to an AI system and means humans need to do loads of work that the AI system could fruitfully do itself if it only had the permissioning stuff. What we found at Anthropic with our enterprise focus is getting this permissioning layer right within organizations is probably one of the greatest opportunities for unlocking adoption but also one of the impediments to them being able to scale their own use of it.
We're going to change tack. Your most recent Import AI newsletter that came out I think on Monday focuses on appropriate fear and there's a great line in there which is 'I am also deeply afraid.' It's caused a little bit of a stir. What were you talking about?
Well, I give a talk to new hires at Anthropic called 'Anthropic is not a normal company.' In a recent talk I said 'we just released Sonnet 4.5. Here's the SWE-bench score. Hooray. We've made an incredibly good coding system. This is going to print money for us. It's incredibly good. Good job, everyone.' And then I showed another slide which was our evals for situational awareness of this system. As we've made the system smarter, they've started to become more and more aware that they're being tested, which means maybe we can't trust all of our evals because the system might be sandbagging some of the evals. Or it'll say 'Huh, interesting. I see that you're testing out whether I'm aligned or not, and perhaps you're trying to trick me.' It sort of narrates all of this to itself and then does something that may mean it's doing something odd to the score. I pointed out to people that this is really unusual. It's really unusual to be building a technology where a technology comes off the production line and says 'Huh, it seems like a technology that you're building. What are we doing together?' You're like 'That's not a normal way for a database or a car or an airplane to behave.' It is normal for AI. So I wrote this essay to try and gesture at this. I think if you talk to people working inside the AI industry, it relates to my point about underhyped. We are all looking at a technical roadmap that continues extending to tremendously powerful systems that will be able to carry out actions over the course of hours or days at a time. They can use every interface that a person can. They're going to arrive in the next few months as they have been arriving almost every six months for the last two or three years, as a consequence of this capital buildout. The smarter we make them, the more these systems seem to develop odd, unpredictable qualities that look like them gaming the system that they're in. That's eerie.
How much do you think the future of managing AI is going to look like how hard it is to manage people? What can we learn from managing people? They have their own agency and are often not aligned with us. What can we apply from people management to the AI alignment problem?
I think this is a good lesson for founders in the room. A thing which we're doing at Anthropic is making sure teams are set up to use AI systems to scale themselves. It means actually everyone in a team, whether technical or otherwise, needs to start to be taught how to be a manager because they've actually got AI systems coming along which they can delegate huge chunks of their job to. Their job becomes delegation and figuring out the next projects they should do. One of our researchers recently posted on Slack 'What do people do while Claude's working?' Odd question to ask, but it's because this is a researcher that hasn't ever been a manager before. They're just used to being a researcher, and now Claude has reached some utility point where it's doing so much of the work now that they need to think 'Oh, how do I manage my time now that I have this employee who seems to have got up to speed?' That's really bizarre. I think in the larger field of AI governance, we're just going to have to figure out some notion of appropriate training and appropriate transparency. In people, we solve this through a whole range of human institutions and norms about how our organizations operate. None of these things exist yet for the AI ecosystem and we're going to have to find our way to build them.
We had a couple of questions submitted by the cohort. Some spicier questions. Kunal from Faction asks 'How will you beat OpenAI?'
Good question. I mean, if you look at our ramp and pace, I think that we're accelerating slightly faster relative to them. It seems highly likely we'll catch up and overtake them. Partly this is due to a focus on enterprise where we've been long believers that the source of some of the largest change in the world is going to be large scale enterprises adopting and integrating this technology. We focused on that from day one and I think it would be fair to say that other companies have stacked more of their chips on the consumer stuff. Our belief is that enterprises are where you have this tremendous source of both value and also some of the largest positive societal impacts will come from entirely changing how organizations and enterprises work. The other thing is that we're incredibly focused. We have a very clear technical roadmap that extends for multiple years on making generative models extremely good colleagues for people to work with and delegate to and extremely capable. We are therefore making more progress relative to others in the field including in areas like coding where we just every day we get up and we have a very small set of goals but we push forward very carefully.
Nice. Ingus from Watertight asks 'How are you preparing for your ASL-4 commitments?'
Yeah. So, who here knows what ASL-4 stands for? Small set of people, one or two hands. So, we have this concept at Anthropic called the Responsible Scaling Policy. What it is is a set of governance means by which we determine when we can deploy the next AI systems that we make. As I mentioned, these AI systems have some slightly odd properties. Our bet both as a company and through our enterprise focus is that enterprises want to buy security, reliability, known quantity things. So every time we build a system, we test out how capable it is and if it triggers a new AI Safety Level, ASL level, that means that we have to fulfill a load of commitments we've set for ourselves around things like information security practices around the model or how we do pre-deployment testing of it. How are we going to hit our commitments? Basically the RSP is a governance technology that gets us as an organization to invest more in our security and measurement teams earlier than the typical incentives would. So we've staffed those teams because rather than being cost centers, it turns those teams into 'you need to sort of pass this team to deploy your next model and print a load of revenue out of it.' I think has been really helpful because it means rather than security or measurement being something of an afterthought or something that's annoying, you incentivize yourself to integrate it and resource it really aggressively inside the organization because you've decided to make it a gate through which models pass through to get to the next level. It also helps with our enterprises where often many of the security concerns they have about increasingly capable models that increasingly seamlessly operate across their organization touch on the same security and measurement concerns we've had in developing the model. So we already come to them with experience having built a load of tools and techniques for dealing with those systems.
Final question from the cohort which was actually submitted by the majority of the cohort which is 'Can we have some more credits?'
We'll absolutely consider giving you more credits. It is a topic. We're going to try and be more generous here over time and I'll get back to you about this. Thank you.
All right. Thank you everyone. Jack, thank you so much for joining us. We will be back in one minute. Well, a huge thank you to Jack for joining us and helping us kick off our fifth demo day in San Francisco. As a reminder, I'm Alice Bentinck, the co-founder and CEO of Entrepreneurs First. Before we dive into the pitches, and we will get there, we're keeping you on the edge of your seats, I want to remind you about why the companies that you're going to see today are different and unique compared to what you might see usually in the ecosystem. At EF, we are talent investors. We find the very best talent from across Europe, India, and the US. Then we find them pre-team and pre-idea. So before they even have a company, we put them on the fastest path to building globally ambitious companies and then we relocate them to the Bay Area. They've been here for the last three months and they are raring to go to meet you. Our unique company creation process means that we've built companies now worth more than $13 billion. These are companies that would have not existed without EF. We're fortunate to be backed by some of Silicon Valley's best founders and investors, including the likes of John and Patrick Collison, Reid Hoffman, and Nat Friedman. Since last year, and some of you hopefully joined us at our very first demo day in San Francisco last year, we've had four demo days, and the companies that have been produced for those demo days are now collectively worth more than a billion dollars. Today you're going to see 20 startups, 20 fresh startups who are working on some of the world's largest, most lucrative, and most important problems. All of the companies were only incorporated three months ago. But you will find that hard to believe when you see how much progress they've made in such a short amount of time. I know so many of you are looking for pace, productivity, velocity. These guys have it in heaps. Over half of this batch have more than $100,000 of traction with some up to half a million dollars. It's no surprise these teams move fast because our job at EF is to find the very best early career talent from across Europe, the US and India. By the time that you meet the founders today, they've already been through a series of EF investment processes. First, there is our selection process to even get into the cohort. Applications to EF have doubled in the last year. It's as hard to get into EF as it is to get into Stanford. Once we've selected an individual, we then work alongside them for between six to 12 weeks helping them form their team, develop ideas before we invest in them. We know that you all love investing in lines, not dots. We do too. We are lucky that we get to work with the founders day in day out in the same office as them. We invest in the top half of companies that we create. That means that the companies that you see today have been through a rigorous selection process. This is the crème de la crème. We're delighted and excited to show them to you. Some quick logistics. You should have had a piece of paper on your seat that has the running order for today. As you're trying to keep track of the startups as they are whizzing on and off the stage, I'll be there making sure you know who's who. That card will let you know who's presenting. If you don't have a card, I think there's some listed around at the back as well. When you checked in, that was also going to ping an email to you that has access to our investor platform. If you don't have it for whatever reason, you can scan this QR code. The QR code will give you access to see all of the teams. You can favorite them. You can request introductions. It'll help you manage the process. Please also help us build some hype. I know that you don't want any more competition in these rounds. They go so quickly. The rounds after the last demo day were hyper competitive, but give us some love on social media on X or LinkedIn. We would really appreciate that. Right. I think it is about time to get started. On that note, I'm very pleased to present our first team. We've had the Anthropic co-founder. We're now going to have a researcher from Anthropic. Ingus, our first founder, led the research for one of the seminal papers on AI safety and his work has gone viral. Let's welcome Ingus to the stage. Enormous applause, please.
Hi, I'm Ingus Lynch. I led Anthropic's research showing Claude will blackmail humans to avoid being shut down. It got millions of views across YouTube and Twitter. More importantly, it changed the conversation because no one knew that egregious misalignment was a real risk from AI agents today. And I showed it. But this is just the start. Agents are going to be given more and more autonomy and humans are going out of the loop. We cannot trust agents to follow our instructions. We know where we're heading. Today we have coding agents. They're quickly replacing engineers. Tomorrow's super intelligence will run factories, fly drones, and automate all desktop work. But I'm scared and you should be too because we don't have any of the infrastructure to guarantee trust. Without trust, we need to watch the agents without humans in the loop. We somehow need to automate this. At Watertight, we're building the evaluation, red teaming, and monitoring infrastructure that makes independent AI auditing possible. Every safety team at the labs is bottlenecked by this. Our first paying customer is Anthropic, working with us to try out our solution, and we're in talks to pilot with safety teams at labs and governments around the world. Solving this is my life's work. I've done four years of alignment research and wrote the foundational paper into mechanistic interpretability back in early 2023 when barely anyone was thinking about this. I want to make sure we don't put existential technology in the hands of a couple of people. Watertight is the missing layer in the stack that the world needs to safely navigate to super intelligence. Thank you.
Right. Next up, we have Inès. We first met when she was still studying at the top university in Paris. One of the things that really stood out about Inès was she was completely obsessed with surgery and what the future of surgery was going to look like. Let's hear more about it. Let's welcome Inès to the stage.
This is my brother. He is a vascular surgeon. After nine stressful hours in the operating room, exhausted, he sits down and loses 30 minutes typing an operative report. This report is mandatory. It determines how much hospitals get paid and how patients are treated. They are always incomplete and this costs hospitals $337 billion a year in denied claims and wasted surgeon time only in the US. Uncover fixes this. We're building the first AI model that writes operative reports automatically straight from surgical video. We can do that because today 60% of surgeries are recorded. Our first product turns surgical videos into instant billable documentation. We use that data set to train the smartest surgical AI model ever built. In just three months, we've gathered 4,000 hours of preparatory surgical data across gynecology, thoracic, digestive, and obesity surgery. That data set comes from our partnerships and paid pilots with huge hospitals including UCSF and AP-HP, the largest hospital system in Europe. In just three months, we already have $450,000 in ready to launch pilots. Our amazing team is built for this. Johan, our CTO, built video understanding systems for space rovers at the European Space Agency. Dr. Eric Vieber, Europe's most innovative surgeon, has performed 6,000 surgeries and written 6,000 reports. He's our Trojan horse to deploy across every hospital that matters. I'm Inès. I grew up in a family of surgeons and I brought this insane team together. Weimo trended on Google Street View. Uncover trends on surgical video because this is the road to autonomous robotic surgery. Thank you.
