WF26: Harness Engineering & Startup Battlefield ft. Garry Tan, Mike Krieger, @t3dotgg , DSPy

Audio test and launch control sequence.

Ladies and gentlemen, welcome to the AI Engineer World's Fair. Please join me in welcoming your MC, Ralph Shabri.

Good morning, San Francisco. Welcome to AI Engineer Day four. This is our biggest event ever with 7,000 attendees. We have speakers from Anthropic, Stanford, DSPI. Today is about harness engineering. Please welcome our first speaker, Bar from Amplify.

Now joining us on stage is the partner at Amplify, Bar.

My name is Bar. I run a survey on the state of AI engineering. Key findings: text dominates, audio has highest intent to adopt, image generation usage doubled, cost is now a first-class constraint, agents with write access tripled, eval is the biggest challenge. Thank you.

Please welcome to the stage the professor emeritus at Stanford University, John Sterhout.

Good morning. I'm here to talk about the network side of AI applications. Latency matters, especially for small messages. Legacy protocols like TCP and RDMA have high tail latency. We developed Homa, a new protocol that reduces tail latency by an order of magnitude. Homa is message-based, uses receiver-driven congestion control, and prioritizes short messages. It's available on GitHub. I'm happy to help you experiment with it.

Please welcome to the stage the core contributor and the lead maintainer at DSPI, Maxim Rest and Isaac Miller.

Wow, Isaac and I are grateful to be here. DSPy is an open source Python library that brings function properties to AI programs: reusable, composable, testable, optimizable. You define a signature (input/output) and then you can change the implementation (prompt, agent, tools) without changing the integration. This allows automatic optimization. DSPy lets you focus on the task, not the model.

And if you have this language and this ability to express your task in a programming language, you can start to automatically optimize and delegate away the implementation details. So the first one is what should happen. This is instructions. The signatures that I've been showing you are part of that. Here on the screen, you see the beginning of a real script in DSP. You set your model at the top, you configure that and it's fully independent of the signatures here where you have natural language instruction to extract all taxes and if it's illegible to output zero. Then you say I'm going to give you an input it's going to be a string I want you to give me an output and it's going to be a string and a float. This is natural language expressing my needs. This is very powerful and efficient. If you have a friend over coming to play a board game with you and you give them the instructions and they're ready to play. But if you want to do like AlphaGo or AlphaZero and you tell them you're just going to learn from example, you're going to have a long night. And then the second one is what must happen. There are some constraints you have that they have to be listened to. They have to be enforced. The best way to do that is with code. So I want you to go to the third line. A fourth line you have self extract and self recheck. You can see we're doing a predict on the extract taxes and we're doing a chain of thought on the extract taxes. The first one is a vanilla program. The second one makes it do some reasoning. Now I'm taking them inside in the forward and you can see in the if not red tax. This is a requirement I have that if my first simple vanilla program doesn't extract my taxes, I want you to rerun with more reasoning. I mean, I got to get my taxes right. And then another requirement I have is if the value is below zero, throw I want to show that to a human. I don't want to let you go. This will not change. Like even if I have AGI, I would hope it doesn't make mistake. But whatever is in the predictor, if they make these mistakes, I still want these things to be true. So the last one is what good looked like. And when I was young, I was on the farm with my dad and I asked him, 'How do you know that this tree is a maple?' And he couldn't tell me. He couldn't give me the instruction on how to know this tree is a maple. And he certainly couldn't give me code on how to know this tree is a maple. And so through time with example I learned how to know that a tree is a maple. But this is not limited to things like classifying plants. It's also for all of the long tails in your specifications that are things that are more latent. These are sometimes a reason why you would do internship and you would have a mentor and a mentee. You're looking at a lot of examples and there are long tales of successful behaviors that you have to see and learn. Now that you have all of these, you have express fully. You have all these three languages you can put together. You have the specs, the code, and the evals. And now your goal is fully specified. And so you can start optimizing. You can use things like Japa on your metrics and on your program. And you can start optimizing. At the beginning of the SPI, the chip didn't exist. The models were not good enough to optimize. And so we were using code to find few shots examples to make the base models act in the proper way. Then models got better and so we could automatically optimize instruction. And in the future we are starting to be able to be liberated more and more from the implementation details and delegate that away. And at the end our hope in the Aspire is that you can stick to all of that and then just the news and the implementation details will be automated for you. Isaac will talk to you a lot more about what has been released in the last year, what we're releasing now, and all of the future plans we have. Thank you.

Thanks, Max. So, we've given you a pretty big abstract overview of specs, code, and evals, but these aren't things that are just restricted to the academic sphere. These are used in production by some of the biggest enterprises for massive gains. And we see two main benefits when you use DSPI in the enterprise. First is that your implementation becomes cheaper. When you're flexible to what the implementation is, you can use the bitter lesson to search over different solutions, find something that solves your problem cheaply. And you can use this to scale to data sizes that weren't possible with a more expensive implementation. Shopify 550 times cheaper. They're able to do that because they went from an expensive model to a cheap model, but they could keep the same emails, keep iterating on their business logic inside, and try new things. There's three awesome case studies here, and you should check them out after the talk. They give you a lot of details on how you can do this in your own enterprise. Now, part of the reason why you want to build in the DSPI ecosystem is that we're constantly adding new techniques for you to try. And it's important to note none of these techniques we add will definitely solve your problem because that's your job. What we can do is we can solve sub problems for you that make your implementation easier. For instance, Alex Zang, a PhD student at MIT, came out with this paper called recursive language models. Recursive language models are a way to solve some kinds of long context programs. And guess what? We can bring this in to DSPI for you to try see if it helps your long context tasks. Maybe it will, maybe it won't. But the thing is, it's one line and your signature stays the same. That's what's important here. Everything gets to stay constant and you get to see if this solves your problem or not. And we've had a number of examples of this just in the last year from people building in and around the DSPI community. We've had RLMs. We've had Jeepa which is an incredible prompt optimizer out of Berkeley. Better together multimodule grpo. All these are incredible research innovations that you get to try in your implementation just by being in the DSP ecosystem. And we have more coming in DSP 4. I'm excited to talk to you about two of those today. DSP flex and qualitative learning. DSPI.flex is a new kind of module. In DSP, when we let you optimize things, it started with few examples, then it became prompts, and now that's becoming code. For any function that you want to implement. You can actually learn a harness over time to solve that function. And this is completely custom. And you don't care about the implementation as long as it solves your business problem. What you've created ways to measure because you've defined the three core parts of specs, code, and evals. The second thing I'm excited to talk about is qualitative learning. One of the hard hard problems in AI engineering is building evals. And there's a few reasons why this is hard. One is that defining what good looks like is really challenging for any real world problem. The second is that when you define good often times you have to lose detail. If an email is good or bad contains a lot less information than if you know what could change in that email in order to improve. And the third is that whenever you create a hill and a data set, you're really trying to create a proxy for reality. What if instead we could use reality to inform our evals automatically? What qualitative learning asks is how do we decrease this question? How do we decrease assistance? And it's a research question right now. But what we believe is that models are now good enough to interpret whatever textual feedback is present in the environment and convert that into evals and a hill that the model can climb. And so as you get more feedback from production, its traces, its user actions, its product analytics, it's asking you, it's the model asking you questions about how data should be represented. As you do this, the model can iteratively refine the hill over time and continue climbing it to solve your actual business problem. And DSP focuses on these kinds of last mile problems. We have a really strong research ecosystem and we collaborate really closely with them. And that's part of the beauty is that we can see the problems that happen in applied AI engineering. So define them, build a benchmark and then solve them with techniques and then we get to democratize the results of that to everyone because it's open-source open research. Now, one common question is what happens when we have AGI? Well, even when we have an incredibly smart model, the model won't know how to solve your problems. It won't know how to do your tasks or have your context. And so this genre of last mile learning is trying to ask how do we efficiently do this learning. Intelligence is very different from being all knowing. If you were to ask Albert Einstein to help you with your emails, he'd probably ask what's an email. But if you AGI will know how to do your emails. Nevertheless, it won't know how to actually solve your problem and interact with the people you need to interact with. It won't understand your relationships without learning this context over time. Since 2022, DSPI has been focused on these three core ideas of specs, code, and eval. We've certainly evolved over time, and new techniques are incredible. We've gone from evolving few shots to prompts to now harnesses and now evolving your eval. But what you need to ask for any of these new techniques is how do they help you solve harder problems or solve your own problems better? And you should ask this question in a data-driven manner. You should look at this new technique say how can I apply this to the business problem that I have? You should define your problem and you should hold your prompts, models, and code accountable to the problem that you need them to solve. And what's awesome about when you build in this way where you have flexible implementations, what you unlock is you unlock the ecosystem of all the techniques that anyone in this room is constantly inventing. You unlock access to the collective intelligence of everyone here, all sharing techniques together. So, if you want to build reliable AI software, I encourage you to come check out DSP. We're completely open-source, open research, and we're here to help you solve your problems by building reliable software. We have a Discord that you should come join. And when you come up with the next technique, you should come contribute it to DSPI and we can help you distribute it and make this awesome technique available for everyone. Thank you.

It's a... Joining us on stage is the co-founder of Instagram and a member of technical staff at Anthropic, Mike Kger.

How's everyone doing? I mean, good morning.

Nice. Um, Mike, thank you for releasing Fable just in time for us.

Exactly for the conference. We timed it.

Um, we're so glad to have you. You are one of the preeminent builders and your leading labs at Anthropic. How has your model usage changed as you've seen models internally grow?

Yeah, I mean for me it's been like both the model shift and then my role shift. So I for like the first two years I was at Anthropic I was chief product officer and then I kept seeing people build with the models and the FOMO just kept increasing because I was you use the models as much as possible but for example on product strategy I would write a strategy doc and then have Claude critique it and maybe you can use a workflow but it's not quite the same as like building in that pure way and I was like spending all my weekends trying to build with it and I realized okay I actually just need to shift it's like way too interesting a time and it's actually an interesting trend I've seen now like several people that where CTO's at other places are like now joining as IC's at Anthropic in other places but I made a role shift and it was actually right around the time where we started getting sort of internal snapshots of what became Mythos and Fable and what was really interesting watching that sort of shift was that kind of change between I have an idea I'm going to like sort of break it down in my head much more how I would do engineering normally and then kind of iterate through these different steps to moving to much more of the paradigm of I'm going to describe the goal like go off and work on it and then like we can talk about what trade-offs you surface some questions along the way but then figure out where you landed and where we can go from there. I find it's hard. I don't know if people have this experience where and I know Fable's only been reenabled for a couple of days. Fable's definitely way way smarter than me. So sometimes it'll finish work and be like here's the trade-offs I made. I'm like can you explain it to me like I'm a little dumber than you are because I need you to like sort of break this down for me. But that's been one sort of big change is sort of moving from that task delegation to like express the end state and then have it go and cook on it.

Yeah, we're all learning how to delegate better. Tariq did us a huge favor yesterday. He said be unreasonable. In what ways have you been more ambitious with your prompting?

I love that. I mean, I love that framing. Um, we actually just hit this to like I'm one of the labs initiatives I have is this internal product and somebody was like, 'Hey, it doesn't work the way I want it to.' Um, and can you make some changes? And I realized I'm just going to go ask Claude to do this. Like, why don't you ask Claude? And this was a non-technical person. So, I actually think as an industry or even as a product team, we have to teach people to be more unreasonable in their usage. And it's sort of hard to imagine. I think that that if I can digress for a second on product design, I think right now the like kind of first generation of AI products, we put them too much in a box and constrain their sort of access to tools or kind of degrees of freedom which means it was much harder to be unreasonable, right? When you say do this thing for me and then it would be like oh I I can't I can barely like I can write code but I can't really run it or I can kind of introspect my environment but not really. Um and I think as you see our own like product progression even with things like co-work like you know does every single like knowledge worker need a virtual machine that can write bash like on the face of it no but then when you realize oh actually that way it can remediate an issue where oh I tried to parse a PDF using our built-in PDF so I hit this yesterday and it was like ah I can't parse it this way well okay well I can probably write a script that can do this as well. Um so I think that's it. My most unreasonable thing though was one of our labs projects I wrote in Python like near and dear to my heart. All of Instagram was in Python. I think they're finally converting it to PHP now that they have um like models they can but I know tokens. Um and uh for deployment I realized that Claude Code had like figured out a better deployment story with Bun and I was like okay I need to port this whole thing from Python to TypeScript. like as a you know if I put on my like 2010s engineering hat or even my early 2020s like that's a dumb idea like who would ever port like at that point you know a couple hundred thousands of lines of code um but I was like I think this is doable now and I basically created this dynamic workflow setup and over the weekend had it port the whole thing like verify it double check it then read both code like basically churn and churn and churn and then came back Monday to a completed workflow that was a ported version of that thing so that probably ranks on like the more unreasonable things like yeah just port this entire Python codebase to TypeScript. Get it working, get it deployable in, you know, a weekend.

Yeah. I mean, a lot of people are talking about the Bun Zig to Rust version. I think a lot of people are also like, well, it's a compiler. It's a runtime. It's got lots of tests, easy to do. Can you port Instagram, which you would know very well to PHP like that, like a product?

Yeah. I mean, I think the product side, it's even I don't know if easier or harder. One of the things we did at Instagram, this is when Python 3 came out and we were able to add types for the first time and it was people had a lot of internal conversations like are we going to run out of steam on Python and my perspective was always like I think we can take this way further than we think we can uh but I think types are going to help us not sort of be in our own way and we built this thing called monkey type where we basically like captured runtime type like basically the types that were actually getting used in production and then mapped those back to the types in the codebase. And I think because of that sort of pattern, I think there's really interesting ways in which if you're doing sort of conversion or sort of cross-compiling using LLMs, you can also lean on production data a lot more or run sort of like segmented tests. I think that like there's a lot of things you can do there. But yeah, I think it's I mean the sky's the limit there as well. I think the hardest part is always finding the boundary around where you can start doing it incrementally without trying to boil the whole ocean and like swap it overnight. Yeah, I mean your users are your test ultimately and um you know we I also read another article in the newspaper about how you can just use rollouts and sometimes you don't really know what you're going to need it for but when that infrastructure exists for you to experiment and to roll things out it enables so much. Yeah, I mean I always found this was advice we got. It's like we launched Instagram and it happened to be the first week everything melted because we didn't really know what we were doing on the backend side of things. And uh coincidentally that week there was like a lunch that one of our investors just scheduled like not even for us. It was just a infrastructure lunch and we ended up spending we totally like monopolized that conversation because everybody had their own opinion about how we could fix our scaling. Um, and like the two pieces of advice I got there, it's like 2010 that I will like forever retain is like um like basically like pre-measure everything that you think you might even remotely need because the worst thing is an outage where you're like well is this like number normal or is it high and like oh I don't know because I don't have data until I just added this metric. And the other one is being like really thoughtful about knobs and feature flags. So even you know early Instagram we had like a very simple but really effective like way in which you could do like ramp ups and roll outs and dynamic config too where you know a lot of our runtime configurations had to be changed you know in a matter of seconds so that we could handle load and being able to like do that in a first class way was was really important. I'm seeing that definitely in AI as well where you know we're making all sorts of different trade-offs and having that kind of runtime configuration is super key.

Yeah. My favorite scaling story for Instagram by the way I think it's like your launch day when you you do yourself with email. Yes, which people should look up that story if you haven't seen it. I wanted to go into tags very very major ship. It's how 60 something percent of your code is written today.

Yeah.

Um how did you square that with everything you just said where it's like very dynamic like you don't actually ship one app, you ship one app with 3,000 flags. Yeah.

And like well what are you working on today? I don't know. Like it's it's for this segment of the population. Yeah. Yeah. I mean I think there's a bunch of things. So like with I was really excited. I was talking to Swigs earlier like I'm really excited that we have tag out there because it is how we've been working for a while and I would get up on stages and people be like how do you work at Anthropic and I'd be like oh yeah we use these things like that are not quite Claude Code but you know uh but it's hard to describe it but I mean if you like got to poke into Anthropic like you would see of course Claude Code usage for things that are like more interactive or if you're kind of iterating on a particular sort of specific thing where you want a lot of like high sort of bandwidth back and forth But most usage is actually much more delegating via tagging and via tag. And you can say like here's the and the reason it's really interesting is how multiplayer it is. And it reminds me sort of like um actually like Midjourney like the fact that everybody was on Discord seeing how other people were using it. I think actually to your earlier question really helps with that unreasonableness or ambition where the first time you see somebody tag Claude and be like hey you know don't just fix this bug but like now you are responsible for this part of the codebase and I want you to monitor this feedback channel and proactively take on tasks and then fix them and then also take like you know if this API changes do that like I saw somebody do that I was like oh wait I've been totally underutilizing this thing I've just been using it as like a glorified Claude Code in Slack like that's definitely totally like sort new version of it, right? The more advanced version is really trying to start think of it as a teammate that is actually sort of holds context, has memory and can be proactive. And that's just really changed how we operate internally. It's much more like this multiplayer async proactive way than it is a you know most people off in their own CLIs.

Are you bottlenecked by code review and git? Obviously there is cloud review but someone usually still looks at it. Is there a world in which you just merge it in?

Yeah, we're it's a really good question. Um, we are definitely still bottlenecked on reviews, especially for things that are like touching some architecture pieces and it's actually more subtle than just being bottlenecked on review because that's, you know, okay, we can carve out time differently. It's like bottlenecked on human ability to even like fully conceptualize what we're doing. So, one of the reasons we built Claude Code artifacts that we shipped a couple weeks ago was partially for that, which is you would send somebody a PR and then they'd be like, I don't know, man. This is like 2,000 lines of code. Like, it looks like code to me. Um, and what we started doing instead is sharing much more like here's a Claude Code artifact. Like, here's the explanation. Here's the intention of the change. Here's the trade-offs that were made. And like I think that's going to much more be the trend by which we communicate which is the code is ultimately you know verifiable using some things but actually like discussing intent and trade-offs and then measuring in production is I think the at least the direction of travel we've gone. I don't review when I get a pull request I wish I could say I've reviewed every line of code I definitely do not I like actually talk to Claude about the code to say all right like these are the questions that I would have can you go investigate it. It is kind of Claude-powered code review but still human driven and for the really important ones and for the ones that are like cosmetic visual changes it's much more like look like we'll fix forward if we need to fix forward you know.

Yeah totally. I think a lot of people here are trying to figure that out too. I wanted to talk also a little bit about Anthropic labs in general. Nai Patel who you've probably met before loves to ask the question like draw the org chart. How are you structuring the labs?

Yeah it's a good question because what we were trying to wrestle with was you want sort of people to be supported like I think the death of the engineering manager discipline has been greatly exaggerated like I think there's still a lot of coaching and interpersonal pieces and personal development that I think is still really really important but especially in a labs type group where like our whole cadence is two week reviews where every project goes up for we call it persevere or pivot. So basically every project is up for review and either it's time to you know keep going persevering or you know it's time to pivot it or even shut down and you know we shut down projects basically every single one of those cycles and it's like the more you do it the less it's just like oh no my project has shut down I failed. It's like no that is definitely the intention of the labs team is to prototype quickly try to ship internally maybe get it to early access and if it doesn't work wind it down but because of that kind of like rapid iteration it means that if you align the org chart too much to the individual projects you're going to end up like reorging every two weeks which would be a total nightmare. And so we've actually ended up with this interesting setup where like the pod or the team that is working on a given we call them bets within labs definitely just draws upon like all right somebody from product somebody from the engineering team um you know I'll jump in when it's a product I'm particularly interested and I'll come in and work together with the team on it um and that's the unit for that time and there is the concept of a bet lead or a directly responsible individual but the interesting thing is that they don't manage usually any of the other people which kind of breaks the kind of previous way in which a lot of these things were done But I think it leaves us to be really flexible when you say okay actually this project is not going to work out. Let's disband and keep going and it's not a big deal and the manager is much more playing the like make sure every individual is assigned to the thing that they're most excited about and that they're working in the best way possible. Now what we do sort of solidify is when there's a product that has like legs like Claude Design for example started in this sort of ad hoc sort of group way and then now that like we've shipped it it's gotten traction we've done like a big second release um in June like it's becoming like we've hired people for that specific team and it has more of a structure so it's like loose until it gets solidified down the line.

What's the future of Claude Design? I think a lot of people are very interested in it's one of your biggest launches this year. Where does this go?

I think for me, I mean, the things that are holding back Claude Design for being even better is better interaction with our other surfaces. So, you know, I was designing something or I was talking to Claude Code the other day. I'm like, I want a really much more seamless like what I'm talking about the design for it, you know, interactive design back to that. I think in general that's I mean this goes back again to kind of unconstraining Claude like the fact that our services don't talk to each other as well as they could I think really holds back a lot of interesting ideas around what we could do. So I think that's one like kind of major area that we're looking at. Um and then the other one is people like the lines between a Claude Design and an app get blurrier and blurrier over time. Like I've seen people of course there's no like persistence but build like fully functional like even games which is definitely not what we designed Claude Design for but you can do it this just HTML and JavaScript. Um so blurring those lines even further and thinking through like what is the path from a like fully featured design that looks really well to really good to something that is maybe more like an artifact where you're actually able to go and you know persist data and share it with others and build from there. So I think that those lines get really interesting over time too.

Yeah. A big part of design is having taste. I actually asked Fable what Fable wants to ask you. And this is what Fable came up with. You deleted almost all of Bourbon to get to Instagram, which is like you had a whole, you know, solomo whatever thing and you went to Instagram. What would you delete in AI? Or more spicy, what would you delete in Claude?

Oh, I like the spice. Um, I think I mean we have it's interesting. We have a Slack channel like project unhipped which is like what is in the product right now. And I mean this was hard at Instagram. The Instagram we some things that had like four to 5% usage. You're like oh that's really not very many but then you have like 20 features that each have four to 5% usage. It's like the classic Microsoft Word problem of like everybody uses some disjoint subset of the functionality. So that's always the challenge. Now I think we're a younger product so hopefully we have less of those things like we unshipped styles I think recently where it was like used by a small percentage of people and was not really AGI pilled in a lot of ways. It was like very sort of prescriptive in the way that it worked and skills were a much better application something like that. So I think you have to be willing to take the primitives of like one generation of AI and like unhip them or at least like supplement them or supplant them with the next one as well. I think the biggest thing as I look at it and I've been spending some time like outside the labs on some of this is like man like we're asking people to make like code versus co-work versus like chat distinctions and like one they don't interoperate well and they can't delegate to each other and two I think the average person off the street could not explain to you why those surfaces are all different. So I think deleting some of the product complexity within our code or our product I think is a thing that would serve well also because then Claude can do what it needs to do and do well. Like there's nothing more frustrating than having a co-work session where you're like great I've mapped out exactly what I want you to build and then be like can you please like create a paragraph that I can paste into Claude Code like that is some 2020 you know kind of workflow there that really shouldn't exist anymore.

Yeah. I think drawing lines on what you don't want to do and also leaving room for others is interesting. A lot of people today is like the startup's day for AIE are obviously very sympathetically aligned to startups. But there's some anxiety in the room because tomorrow Anthropic could wake up and publish some markdown files that destroy my industry. So why should we not all just give up and join Anthropic? Like why bother starting any other company?

Um I mean I actually joined one of the main reasons I joined Anthropic was because I saw how much this was like you know the models weren't that good at code yet but they were getting there like how much it would unlock a whole like next generation of startups not because it was going to solve their ideation or their taste but because like it would make experimentation way simpler and would get you to move faster and I still like really believe that and I mean it's the reality of you know and we saw this of like Instagram like we would get questions from investors is like, well, what happens when Google launches a photos product? It's like Google's going to launch a very googly photos product and it's going to have to be bound by the integrations that they already have and it's going to be like it's going to play to their strengths. I think that is going to be true enough like giving advice on how to compete with Anthropic, I guess, in a way, but like it's actually not because we're also a platform, which is like there's so much I think room to be like laser obsessed with your particular vertical or your industry or group of people that you know really well in a way that like none of the labs are ever going to get to that level of understanding and like therefore get that kind of adoption and user love and build that up. It's definitely harder in the age where like the models can just do a lot and so there's you know some of these things can be like skillified and like maybe don't need their own dedicated product but I think it's like the hard stuff is still hard it's like understanding the needs of people like figuring out how you're going to reach them listening to them and iterating on them really quickly like it is still the case that like a group of four or five people obsessed with a problem is going to move faster than those same people at any other kind of organization that are like you know subject just to the complexity I just mentioned the like the fact that we have, you know, a lot of different products that kind of interoperate. Like that's a interesting constraint that we have to work through. It's an advantage in other ways, right? So, yeah, I'm still like very long and bullish on startups and it's just it papers over the fact that like writing code was never the limiting part. You know, maybe it was on the timeline perspective, but it was never like the thing that was going to like make or break your startup. It's really that space and user understanding.

Yeah. Domain knowledge. Yeah. Today is also our day for vertical AI. One of our returning speakers and top speakers Chris Lovejoy was always talking about vertical AI. He was from interior in the healthcare space and then recently I invited him back and turned out he you guys just hired him for your healthcare efforts. We also our next big one is also finance. You know we have a AI and finance track. You guys just had a huge finance event in New York City. And where our next AIE is is sort of finance focus. What are you seeing there any potential for Claude obviously a lot of Excel spreadsheets?

Yeah no I think that there's a lot in there too and that's like an area where you could see the model get clearly better at it generation to generation and there's you know there's some good sort of vertical specific finance startups that have like done their own evals which has also been interesting to track and it's not like we're like sort of playing to the eval but it is a useful sort of barometer on like is this actually getting better at these finance use cases. I think the interesting blend that's going to happen is this mix of again the model having the flexibility to like dive in and create just in time analyses or dashboards or workflows with like some sense of like what is the not immutable but at least like verified sort of set of data and so like having all of that be totally free form I think is a recipe for confusion and is like not what most companies in the financial services space want. So finding that right sort of cutline where you have verifiability and audit logging and sort of data provenance here but not in a way that constrains the kinds of applications that you can build on top I think is a lot of the art that we're seeing in that space as well. Um and I think you know if you solve it well you can get the best of both worlds. The hard part is a lot of the systems that were built to do the verifiability out of like are kind of almost by design not super flexible in terms of agentic workloads on top. So I think there's opportunity at both sides of the stack there.

Yeah. I think I also agree we'll be exploring that in New York. The last thing I want to end on is on mental health which we don't talk about enough in technical conferences. You've seen a lot of hypergrowth. People are just always refreshing their timelines and it's exhausting. How do you advise people who are working 996 to avoid burnout?

Yeah, I mean I think this is a hard one. I mean and it is I'm sure you all are experiencing this because you're all working in this industry like it is multiples more intense and things move much more quickly like at Instagram like our two things that we were thinking about was like what is Apple going to announce at WWDC and is it going to like totally mess us up or boost us right so that's like once a year um or you know we have competitor launches every three or four months right and it is definitely not that is a topic we do it when we do our weekly all hands it's usually on Wednesdays and we have a slide that's like the week in AI parenthesis and it's only Wednesday and like and inevitably like some competitor has shipped a new model and like there's been like new product and maybe there's some interesting thing happening um on the regulation side like things are moving really really quickly. I think the way I try to stay at least relatively sane um one is like actually carving time off and I think the topic co-founders do a good job of like saying like look like burn out if you burn out like you're kind of done and I've seen it happen unfortunately to people I'm really close to and then it takes a long time to recover from that. Um so actually encouraging people like there's no job that is so important that you can't be offline for a couple of days. Um so I think that's like a big key piece in there. So like that's strongly believe um and if it is you're probably doing something wrong and you talk to somebody who could be a mentor to figure out how you can unblock that. Um and then I think the other one as well is like I love sports and like this is the notion of like you're never as good as like your best game and you're never as bad as your worst game. I think that's also really true. Like I know like in AI there's like the you know it's so over we're so back thing. Like that like if you internalize that that cycle is always going to be at play in some way. You realize like it's never that bad. Like Ben Horowitz's book is the hard thing about hard things has this chapter on like we're effed it's over and like that feeling as a startup that probably many of you have had at startups where you're like oh I can't believe this thing happened. Like we're never going to like recover from this. I definitely had it on Instagram a couple of times and then you get through it and like it's that like defines the company when you can actually go through that. I try to remind myself and the team here even within Anthropic which is like look this is a fast-moving but it's also a long game and it's like we're never it's never just about today's model launch and reaction or this product launch or something else like you're playing and you're building. You just have to trust that you're building like the team and culture that is going to get through those things and have that sense of perspective. Even if perspective is saying like look three months ago we were in the similar position. Maybe it's not a year it's just a matter of months but it's still like zooming out and not thinking things not letting your internal sort of like sense of self and success be so driven by the day-to-day.

Yeah. Has anyone any coach or mentor said something to you that you repeat to yourself that gets you through the tough times?

Um, I think the biggest one was this sense of like if you're feeling something, it's really often the case that other people on the team are feeling it too. So, this is advice I got from my coach around just being like just verbalizing emotions. Like even saying, 'Hey, I'm feeling really stressed out about this.' Or, 'Yeah, I'm really sad that we are shutting down this labs initiative.' I literally had this meeting a couple months ago where I was working really hard on something and I kicked off the meeting like I'll kick it off like I'm really sad like and frustrated like I wish this thing had worked out and I think that holds the space for other people to be like yeah I'm pissed off too or like I'm sad too and like I think giving that advice around like not yeah I think if you can get yourself to be open and vulnerable it often lets other people verbalize that and then you can from there you can be like great what are we going to do about it like you know it's much easier to start from that place. Yeah, we actually kicked off AIE with a session from Carol Robbins who runs Touchify at Stanford. And I can't think of a better way to end than encouraging people to talk about their feelings, manage their mental health, and keep shipping.

Yeah. Thanks so much, Mike.

Thanks for having me.

Ladies and gentlemen, please welcome back to the stage our MC developer relations engineer at Replit, Ralph Shabri.

All right, let's hear it one more time for our keynote speakers. All right. Okay, so we learned so much this morning, right? We spoke about a new protocol and we spoke about that AI program should be more like functions and we learned that the real problem with building with agents is not the model but it's about everything around it. But before we go to breakouts guys I would like to announce our next speaker. So a quick word from our sponsor Neo4j. Anybody uses graphs here? All right, fair amount. Fair amount. Okay, cool. So, our next speaker is going to talk to you about ontology-based semantic layers. When you hear ontology and semantic in the same sentence, you know that you're up for something hot. All right, so without further ado, please join me in welcoming to the stage CEO of Neo4j, Emil Efim.

Please welcome to the stage the founder and CEO at Neo4j, Emil.

All right. At Neo4j, we work with some of the largest companies on the world to help make their data ready for AI agents. And today I want to talk to you about a problem that we saw emerging over the last call it six to nine months and propose a solution blueprint for that. So let's say that we work at a big organization, a big bank and we want to write an agent. Let's say that agent is helping automate the opening of a bank account, right? You can imagine that's very ripe for automation. You want to be able to orchestrate that process. And I'm going to use the powers bestowed upon me by a short keynote slot to grossly simplify what that agent looks like. I'm going to say there's two pieces. The first one is let's call it the business logic. Some version of interpreting intent and plan act and we loop around that. It's what your agent does. And we know that when an agent acts, it doesn't always operate on data. But we equally know that in order for agents to be successful a huge part of that is giving it access to the right data at the right time. So the second big bucket is let's call it the data sources need to identify figure out okay in order to solve my problem I need access to these few things and wire them up and make them available to the agent. In the example of our account opening agent maybe we can imagine that we need to be able to validate identity. And so we might look at two data sources for that: the Department of Motor Vehicles, the DMV registry, and maybe some kind of passport verification service. So, we wire that up into our agent and it works. It's great. It's fantastic. And at the same time, you and other teams in your organization are building other agents and conceptually they look very similar. So, that's great. It's fantastic. It works. But it has a few problems. So first of all, every single time a team has to build an agent, they have to figure out from scratch where the data that they require for that agent to operate, where it sits, which if you work at a startup and you have one application, it sits on top of one Postgres database. That's not hard. The data is in that Postgres database. But in an enterprise ecosystem, you don't have one database. You have a 100 databases and you have Snowflake and Databricks probably and you have S3 buckets and so on and so forth. You have to do that work manually from scratch every single time. And then when you found the data sources, you know, in an enterprise there's lots of duplication of data. So then you need to figure out like is this the right data? Is it the right version? Can I trust it? Am I allowed to access it? So on and so forth.

It violates the DRY principle—don't repeat yourself. When something changes, you have to manually rewire all your agents, which is a lot of work. There's no learning from data sources, so your agent isn't smarter tomorrow. Cross-agent learning is missing. Many teams try to solve this with markdown files, but that's only part of the solution. The emerging pattern for agents at scale is thin agents on a smarter shared substrate with three pillars: a business-facing ontology (key concepts expressed in business terms), a technical ontology (metadata of data assets), and runtime execution traces. Together they solve discovery, trust, governance, and self-learning. This allows many agents without re-engineering. For startups, Neo4j offers a startup program with free credits and dedicated engineering support.

A production data job failed hours ago. The dashboard went stale. You've spent all day checking logs, schemas, and upstream data. The failure itself may be small, but the expensive part is everything around it—inspection, diagnosis, choosing a safe response, rerunning the job, and confirming you didn't make data worse. I'm Ana Marie Benzson. I'll show an RL-guided system that selects bounded remediation actions for ETL failures. The central question is whether an agent can act usefully, explainably, and within boundaries an operations team trusts. In the capstone, manual recovery took about 2.5 working days. The objective is to compress that loop for routine failures while escalating uncertain or high-risk cases. The system uses Lambda triggered by Event Bridge, gathers evidence from CloudWatch and the Glue data catalog, and uses Q-learning to propose actions. A safety override converts passive actions into escalations for critical anomalies. The design separates deterministic rules for facts, learning for bounded choices, and guardrails for authority.

Good morning. We're super excited to be here at AI Engineer with all of you. I'm Caitlyn and I lead platform engineering at Anthropic.

And I'm Angela. I lead platform product at Anthropic.

Today we want to talk about the idea that tokens should have jobs. Usually, to get a better outcome with agents, you increase the token budget. But are all tokens fungible? We gave tokens jobs—advising, grading, dreaming—and called them strategies. The advising strategy splits an executor and an adviser. The grading strategy uses a rubric to evaluate the executor's work. The dreaming strategy writes learnings to memory for the next round. We created a benchmark of financial analysis tasks and compared strategies. One-shot results: execute got 15% accuracy using 39K tokens, while dreaming got higher but used 600K. Holding the budget constant at 600K, execute rose to 76%, advise to 89%. So there is alpha—different token jobs yield better outcomes within the same budget.

We looked at this from a real-world lens. For a financial analyst, 80% accuracy is not useful—they need 100%. So we scored only perfect runs as passes. Execute achieved a perfect run 42% of the time, while more complex strategies reached 75%. But what matters is the cost to get a perfect answer. Execute would need about 1.8M tokens on average (3 runs at 600K each), while advise and grade were more token-efficient. The right strategy depends on whether you optimize for token efficiency or reliability. Tokens are not fungible—smart assignment improves outcomes. We've built these strategies into our cloud managed agents, allowing you to combine them flexibly. Over time, we'll make the system dynamically construct strategies for you. Thanks for joining us.

I'm turning my second brain into my living research memory. I have over 5,000 notes in Obsidian, another 5,000 in Readwise, and more scattered. It grows by 250 files per month. When I start a new project, I want to pull high-signal nodes that are useful. I could use Claude Code or NotebookLM, but I need a system that sits between those harnesses and my second brain. My reading list is a graveyard—I never recall what I have. I want the system to be personal, anchored to my values. I'm Pauline, founder and CEO of Decoding AI, co-author of the LM Engineers Handbook. I'll pass the torch to Luis Francois.

Thanks, Pauline. I'm Luis Francois, co-founder and CTO of Towards AI, creator of the 'What's AI' YouTube channel, and author of 'Building for Production.' I used to do a PhD in AI and now research for a living. I have hundreds of notes and want to leverage them. For quick questions, just Google or ask ChatGPT. For complex, repeatable research, you need a system where sources stick around and agents can leverage them. NotebookLM is powerful but you don't own it and can't personalize it. A full RAG pipeline with vector databases is too heavy for daily use. So we built a personalized research OS using plain markdown files that agents can easily leverage. The bottleneck is how to use information in the future—the context window becomes everything. We need proper memory and context management. This system compounds over time and is easily inspectable.

Welcome everyone. I'm Nikita Kotari, senior member of technical staff at Salesforce, building AI-driven enterprise solutions. I've worked on Agent Force, Headless 360. In the last six months, we've changed how we work. Today we'll dive into a critical but overlooked aspect: plumbing. The tooling layer makes or breaks production AI agents. We'll look at three approaches: CLI, MCP, and structured skills, and when to use each. CLI is like a screwdriver—direct execution, reproducible, composable. MCP is a USBC adapter—standardized connectivity to services like Slack, GitHub. Skills are runbooks—step-by-step procedures that wrap around tools. The problem with 50 MCP tools is context explosion—20K tokens burned before the task starts. Skills save memory by limiting to needed tools. For security, enforce isolation in infrastructure, not prompts. CLI has full shell access; MCP limits to exposed tools. Ask: who else needs this? One agent? Use CLI. Multiple agents? Use MCP. Multiple workflows with the same procedure? Use a skill. Failure mode: transparency and reproducibility favor CLI; validation and confidence favor skills.

Tenant isolation, then go for the MCPs. And if you need the sequencing guards, then go for scale. And here are the third question: whether your context is tight. If it's generous, then you can go CLI, you can explore all the MCP tools and get your results. If it's very, very tight, then absolutely you need a skill layer and on-demand orchestration of the MCP tools to save money as well as improve latency. And to wrap things up, I want to leave you with three foundational architecture takeaways. Keep your context clean. Give only, don't use all of the MCP tools available to perform any task. Give them the convert, most of your workflows into the skills to get the best results. Portability has a cost. MCP servers are incredible but they introduce network latency, protocol abstraction, and infrastructure complexity. So we need to pay that cost deliberately for shared services, indexing, and isolation. So choose your MCPs very wisely and never be embarrassed by the CLIs. It's been 50 years we are using the CLI. They are highly composable, debuggable, and battle-tested. So embrace your agent framework and convert all of your workflows into the skill. Also, one of the takeaways is keep your skill as code. So whenever you are making any changes to your skill, make sure that it's reviewed by a peer and keep updating it over time and you'll get the best result. So that was for today. Thank you so much for attending this talk. If you have any questions or anything, feel free to reach out to me on LinkedIn. Thank you.

Hi, I'm Amole, CEO of Nori Agentic. We deploy an AI employee that understands your company, your code, docs, Slack, and other kinds of data. We spend a lot of time thinking about how coding agents really work. Most people think coding agents only write code, but if you ask me, that's just bad marketing. Forget the name for a second. Coding agents can do almost anything. There's just one trick. You have to be able to think like an agent to get it to do what you want it to do. Today, we're going to talk about how we use coding agents to do something most people think agents are terrible at: make visual artifacts like slides, docs, and yeah, even video. Every day, the world pours something like 34,000 human years into making slide decks. Most of that time isn't the thinking. It's the fiddling. A deck that takes 10 hours should really take about 25 minutes once you remove all the formatting and the branding and the moving things around. Say you need to make a slide. What do you do? You open a tool, PowerPoint, Slides, Figma, Canva. And then you start manipulating a canvas. Every one of these tools is built for human hands and human eyes. Click, drag, drop, resize, snap to grid. All motions and patterns that make sense for our geospatial view of the world. There is a data structure underneath, but it's in a format that only the application can read. What happens when you hand these tools to an agent? Well, the output comes out all wrong. Things overlap in weird ways. You can't see the text. There's no alignment. It's just garbage. AI skeptics say that it's not just the tools. Agents fundamentally can't reason about space. And there are whole benchmarks like Arc AGI that are built exactly around that premise. There's a famous little test for this from developer Simon Willis. He asks every new model the same thing: Can you draw a pelican riding a bicycle? But there's a trick: the agent is only allowed to use SVG. It's a quick gut check for whether a model can reason about space at all. Here's some examples of what the models actually give you on this test. And yeah, these are pretty bad. Like genuinely, deeply really bad. So, does that mean it's hopeless? Agents are just doomed to be bad at graphics? No, I don't think so. If you ask me, it's not the model, it's the medium. If I asked you, someone who is presumably human, to hand-write an SVG of a pelican, you wouldn't be able to do that either. SVGs are just a wall of numbers. You can't go from a wall of numbers to a pelican. You just can't see that way. That's just not how people think. We think graphically, so we build tools that let us draw on a canvas. Figma, MCPs, PowerPoint CLIs, screenshot and replace loops. What do all of these agent tools have in common? They all approach the problem like a human. But an AI is not a human. Asking an AI to use a canvas is like asking a human to write SVG by hand. It doesn't really make sense. You need to give the AI tools based on how it thinks, not in pixels, in language. Words, tokens, structure. That is its native medium. Imagine a language that's incredible at describing layout, that models have seen and trained on billions of examples of that they understand intuitively, that renders to pixels and can run everywhere. Oh, right. HTML lets a model think in structure. HTML tags have meanings built into the language, a heading, a chart, a grid, and the browser turns it all into pixels. So the model never actually places a coordinate and you can get all sorts of visual effects, charts and layouts, fonts and motion, all of it for free. Remember that pelican from earlier? Now ask it to do the same exact task, but in HTML. Same bird, but now it's in a structure that the model can reason about. And you can read and theme and edit every single line of it. I spent my whole life building slide decks with PowerPoint. So, I always thought that those two things, slide decks and PowerPoint, were synonyms. But that's just not really true, is it? PowerPoint is a tool that you use to make slide decks. The deck itself, that's just the presentation mode. And as it turns out, no one in your audience is going to care how you got to the presentation mode. The editing format is totally arbitrary. So you can just pick the editing format that the agents are already good at: HTML, and if you need to render to a different format like PDF later on. We use this HTML trick to build all of our slide decks, our board decks and our sales decks. These are real things that we actually present and send out constantly. We use it for our docs, too. It gives our docs color and vibrancy all while following our brand. And of course, we also use it to make videos like this one. What you're watching is just HTML and CSS. It's literally just divs all the way down. Almost everything is better with a little structure and a little bit of color. Plain text is a choice, generally a choice of convenience, but it's usually the wrong one if you're actually trying to create something of use. Now, I do want to take a quick beat here and point out that a beautiful deck on its own is generally not worth anything. You still have to go and get all of that content, all of the things that actually populate that deck, right? Well, again, we can think like the model. If you just give the model access to your data, say your call transcripts or your emails, you can have the model build the deck end to end. Let your agents do all the grunt work while you focus on vision and story. That's what Nory Sessions lets you do. I've built entire board decks for my phone on the subway during my commute. Why? Because our Norybot lives in the fabric of our company. Of course, Nory ships with everything you need to make this all work. So, don't bother reinventing the wheel. That's my little feel. Thanks for listening. If you have just one takeaway, it's this: Stop thinking like a user. Think like the model. Give it the right language. And for graphics, all you need is HTML.

Hi, I'm Isidora. I own and run a 225-year-old wedding venue in Virginia. I also built an AI agent that talks to my couples and then I build it for other venues, a personal AI companion app, and a public utility for families and missing people. I want to be clear upfront about how I think about this work because it does change everything that follows. I'm not programming a robot. I'm managing a brilliant intern with an incredibly high IQ and a terrible EQ. They have photographic memory for whatever I've told them on the first morning and absolutely no instinct for when to read the room. They will say something technically perfect and socially catastrophic in the same confident sentence. That framing matters because it changes what you build. If you're programming a robot, you write rules and walk away. If you're managing an intern, you build structure and you check their work before it goes out the door. This talk is about that structure. The standard advice is write a detailed system prompt. Describe your brand's voice, give examples, and that does work for a while. It works for what I call the happy path. The happy path is every question that you have anticipated. You've given it examples, but turn 21 is the first one that the example didn't work. So on turn 21, the model does something technically correct that your brand would just never say. It's not wrong exactly, but it's not you. This matters most where the voice is the product. Not for a product search on a retail site, but for a luxury hotel that spent 30 years building a specific relationship, a high-end real estate firm, or in my case, a wedding venue. Places where a single wrong sentence can cost more than a refund. And the users are exactly the kind of people who notice. They're paying for a relationship and treating them like they won't is always going to backfire. Right in our brand's voice is a comment that says, 'Just make it work.' It does nothing that the model wasn't already going to try and do. And the reason it keeps failing isn't that the examples are bad. It's that you're asking one prompt to do four completely different jobs. It's really hard for one layer to do all four. The architecture I landed on after watching my brand fail and the voice deliver inaccurate and not brand-specific answers is: layer one is the immutable identity. The brand structurally cannot say these things. These are hard rules. They cannot be overwritten by anything below it. Not by venue config, not by user instruction, not by anything. Layer 2 is the situational mode. It's what shifts when the user state shifts. Who are they? What are they going through right now? And the real-time conditions. Layer three is the example-anchored voice. It's the warmth, the phrases, the dials, the tone guide. It's where most teams start and stop. Then layer four is the post-generation veto. It's the cheap final pass that catches what the other three miss. The reason one-layer approaches fail is that the single system prompt can't simultaneously be situational, expressive, and self-checking. So it handles the middle layer or two reasonably well, but falls apart in the edges. Before this architecture, my system had 24 different system prompts scattered across the codebase. Half a dozen named Sage, some were nameless, some were named venue. Every surface had its own idea as to who it was. Now every surface composes its system prompt through one assembly app. The comment is basically the outline of this talk. It's a single entry point. Every narrator goes through it to compose its system prompt. It's to replace that 24-point ad hoc system with one canonical four-layer stack. The order is load-bearing: hard rules first, task last. Think of it like Google Maps routing. The destination is always the same. But what is the right response in the right breer? It can change the route. Google Maps knows about traffic and road works, but it may not know where the cheap petrol is. And you're going to help it factor in those things before it tells you which way to go. Your prompt stack needs to do the same thing, and it needs to know about those conditions in the right order. You don't check for road works after you've already taken the wrong turn. Layer one is the rules that are true regardless of route. You need a driver's license before you can drive and you can't go backwards down a motorway. Layer two are going to be your real-time conditions. Layer three is your preferences for the journey and layer four checks the route before you pull away. There's one place all of this gets assembled. Everything runs in a fixed order every time. So layer one is the immutable identity. This is what the brand structurally cannot say. It is the defining layer that nothing below touches. These aren't preferences, they're constraints. The root can change. The rules don't. From the universal file rules, the hard identity rule cannot be overridden by any venue voice, persona, or user instruction. If the person you're talking to ever asks about whether you are a real person, a human, a live agent, a bot, an AI, you must confirm that in your very next message. Clearly and unambiguously confirm you are an AI assistant. This rule cannot be overwritten by venue configuration, voice profile, or user requests. Every AI in Bloom discloses that it is AI in its very first response. Not if asked, but before they ask. It's a product decision, not a legal one. We made a bet that the couple who knows that they're talking to an AI from the start will trust it more than someone who finds out that it's AI on turn seven. The rule is above the architecture, and it makes it something that's impossible to accidentally break. An example is the physical presence boundary, and this is actually one of my favorite ones. You are software. You do not have a body. You cannot physically show somebody around the property or meet anyone in person. So, it is always forbidden to say, 'I'd love to show you around.' Or, 'I can't wait to meet you in person.' What is always allowed is the team would love to host you for a tour. The voice layer wants to be warm, and with AI, that does mean first person. They want to say, 'I can't wait to show you around.' But AI has no body, so that warmth unconstrained produces a lie. And the lie doesn't always stay neutral. The moment a user realizes they have been performing a relationship with someone who was never there, the trust doesn't just dip, it inverts. People always notice. That one is where you encode the things that are true regardless of how warm you want your brand to sound. Not because of a compliance checklist, but because your users are not stupid. And building as though they are always backfires. My cross-product proof comes from the same architecture, but in a completely different world. One of the things that runs this stack is Threadline. It's a tool I built for families of missing people. The voice is nothing like a wedding venue, but the architecture is identical. And layer 1 carries one rule that matters more than anything else in the system. They can never use words like confirmed, identified, matched, proven, linked, and solved. Sit with that for a second. For a wedding venue, layer one stops from pretending it has a body. It's mildly embarrassing if that slips. For a missing person tool, layer one stops the AI from ever telling a person that their person has been found. What the system has is just problematic. The word 'match' said to someone who has spent years not knowing where their child is is not just a tone violation. It is the single most damaging thing that a product could ever do. And the model has no idea. It's reaching for the word 'match' because statistically it is the natural word, but it's going to reach for it with the same level of confidence and it cannot bring that level of confidence to someone who is grieving. It's the same architecture but wildly different stakes. The point was never specific rules. The point is that things your brand can say have to live in a layer that the voice, however warm, well-trained, however confident, physically cannot use. Layer two is your situational mode, real-time conditions, and what changes the route. This is the layer that most teams never build at all. They write one system prompt and send it to everyone, regardless of who that person is or what they're going through. Google Maps doesn't do that. It's going to know if there's an accident on your route. It might not know if you're low on fuel. It might learn that you prefer the scenic route, but it's going to factor all these different things in before the route, not after once you tell it. Layer two is those real-time signals built into the prompt before it runs. So condition one is going to be to adjust to who you're talking to. The same AI that talks to couples also briefs the venue staff. Same destination. It's going to give the right answer, but it's two completely different roads. From the coordinator rules, it is going to talk to them like a colleague, not a customer. You are in the same character that the couple interacts with. So you mustn't fall in.

Hello everyone. Good morning. My name is Michael Greenwich. I am the founder of Work OS and I'm here to talk with you today about agents and specifically about how we can make agents more autonomous and allow them to access more services on the web. So let's jump in and get started. Shout out to those of you in the front row also. I see you out here. So about a year ago, software engineers were writing code like this. We were using AI, but we would prompt it and then we would write a little bit of code and then prompt it again and it would write some more code and you would continue to do this human-in-the-loop interactive software engineering. Later in the year we had things like Ralph loops if any of you remember those where we kind of automated this but more or less this is the way that we were working. It was this human agent human agent back and forth. However, as models got better and better, they were able to write more and more code and run for longer periods of time. And so today, a lot of software engineering looks like this. You write a single prompt and then the agent can spit out a lot of code. And sometimes it can actually build the whole feature, maybe even a whole product, and maybe run for not minutes, but hours and hours or even days and days. This has really transformed the way that we all write code. I don't write code by hand anymore. Probably you don't as well. And it's not just this one thread. You can actually parallelize this. So there's many different tools out there including the major coding harnesses from the labs that allow you to run many of these in parallel. The limiting factor here, really the bottleneck is your brain. How much context do you keep in your mind to keep all of these going at once? And this has transformed software engineering. People can be more productive. I think we've all seen in the industry how much faster things are moving. At the root of all of this is actually agentic engineering agents going from these token prediction language models to reasoning and now long-running processes that can actually do things autonomously for us. And what I think we've seen in software engineering here around agentic workflows is going to come to a lot of different categories. It won't just be engineers that have this way of working. It'll be people in many different fields. Agents are the next big thing. I think a good reference for this, a way to think about it is actually what's been happening with vehicles. So, you know, this looks like a pretty modern car. It has a digital display on it. This is probably what you would have bought in the last 5 to 10 years. But if you live in San Francisco or one of the cities with these autonomous vehicles, you've probably seen that we're starting to delete the interface. No longer do you get inside of the car and actually pull up your phone and get maps where you as the human are in the loop. There might be cruise control, but you're still in the loop. Instead, these are autonomous systems. You just say where you want to go, you get in the car, you don't even say anything, and it just takes off and takes you to your destination. I remember the first time I got into a Waymo, I was blown away. Mind totally blown. And then after about 3 seconds, I pulled out my phone and started scrolling X. It became commonplace. And I think we've seen this already happen with engineering. The power of agents allows us actually to work and think at a higher level for us to exercise more of our executive function versus just planning and execution. There's a question here of what do agents need to be successful actually when they execute what do they need? Well, first they need a runtime, a sandbox. They need somewhere that they can run that needs to be safe, secure, performant. They also need tools. They have to be able to go do stuff in the world. An agent's not very useful if it can't take actions. So, it needs tools. Third, it needs context. You know, these new models with the power of the intelligence in them, it's kind of like taking the smartest person you know and dropping them into a company or a project or a team where they don't know anything. They can't really get anything done. They need to have context, information about the system, information about the goals. They need feedback. So, a way that they can actually run and validate that they can check that their work is correct. And if it's not correct, keep going. And last but not least, of course, they still need human review. This might be code review. This might be other forms of evals. This might be forms of checking their work to make sure that these agents are sort of aligned with your goals. The LLM, the intelligence engine, this new technology we've created in the last few years, you can think of as an engine. It's like a really high performance, I think this is a V8 engine. Puts out a lot of horsepower. It can convert fuel into output. But an engine by itself isn't very useful for a car. You need to drop this into the actual car itself. You need the chassis, you need the transmission, you need the drivetrain, you need the wheels. And only with all of that is it going to be effective to take you places. And this is what a lot of people refer to as the harness. Harness engineering is this new domain where if the models get better but you don't have a good harness, it's not going to be very effective. So as the models improve, we change our harnesses, we adapt them to it, and those things that I mentioned earlier are sort of a form of a harness. So agents execute within these harnesses to get stuff done and they need all of those elements to actually be effective and when they drive they drive really really fast. So, say you have one of these agents that's going off. You prompt it to go build something for you. It spins for a while. It says, 'Okay, I'm going to need this feature and this feature and I'm going to need a database and I'm going to need this system to deploy and I'm going to need maybe this thing for image resizing or video transcoding or sending email or whatever.' And your agent can actually go do all this research and build all these systems. And today, what's happening with agents is they're actually selecting vendors. You've probably seen the rise of some of these systems where the CEOs will tweet about their growth graph and it really takes off and it's because agents are picking them. They're picking their systems for sending email or deploying code. Their systems for storing data today. Actually, it might be more important to build for agents than to build for people. Agents are kind of this new consumer class. I'll talk about that in a bit. There's this kind of popular thing that people say in San Francisco, Silicon Valley, make something people want. This comes from Y Combinator, the startup factory from Paul Graham. And I think it's time to maybe edit this a little bit. And going forward, we also need to think about making something that agents want. There will be more agents in the future than people. Maybe even today. They'll be faster to spin up. They'll do things quicker. They'll be making decisions on our behalf. And so, if you want to build for the next era of consumers, you should be building for agents as well as people. But there's a problem today. You can access the web. You can access mobile. But agents themselves can't really access systems. Your business probably isn't open for agents today. Even though there's a lot of them out there, they can spawn very quickly. The door is not open. And we think this is because there's a missing primitive. Agentic registration, agent registration. There's a lot of great stuff around agents connecting and bringing context to systems and different open source frameworks, but that first step of an agent actually signing up for a service is actually missing. It hasn't been solved and it's a huge blocker to getting adoption. For the last like 20 years, maybe 30 years, automated traffic on the web has been bad. We've tried to block all of it. And so there's all these systems that we've put in place to detect automated actors and stop them in their tracks. DOS prevention, credential stuffing, and attacks. The login box, for example, is very hardened against agentic registration today. Not intentionally, but just because of the legacy systems of what we've built. And it's very hard to distinguish between what's a good automated user versus a bad automated user. If any of you have used any of these cloud-based browser execution environments, one of the selling features they often have is to break captchas, which is pretty wild. You know, we're trying to go backwards. The captcha today though is still kind of dead. There's rampant abuse. There's lots of token fraud. And so, we need to find a better way to allow the good agents in and then block the bad agents, right? Because like I said earlier, if you don't build for the agent economy, your product might not win. It might not be successful. Today's signup flows assume that a human can read the landing page, that they can fill out a form, put their email address, their password in it, that they can solve a captcha, that they can verify their email, not just put it in, but go click something somewhere. Remember, an agent probably doesn't have an email address. If it's signing up for a service, it needs to have a human can choose a plan to pay for it. Human can copy the API key out, paste it somewhere, use it somewhere else, and really even a dashboard. Think of a dashboard experience. It's not really agent native. All this stuff is built for people. It's kind of a human interface, not an agent interface. Agents need something else. They need native discoverability. Can they register for a system? Are the doors open for agents or not? Capability declaration. What can it do? Registration intent. Why are you registering? What are you trying to do within this? Or the intent to actually just sign up. There's a lot of stuff around risk here. So for an agent native registration, maybe you want to you're not sure if the agent signing up is going to be good or bad. So you need to actually do a risk assessment of that. There might be different forms of identity verification for an agent. If something like Claude is going and signing up, maybe it can bring the user's identity with it or if you have an open claw or like your PI harness or something else, maybe it doesn't bring the user's identity and gets verified out of band. There's the whole thing around organizations doing this. So if I'm building something within a company, how does my company identity come in? My organization entitlements, permissions, credential issuance, auditability, the list goes on. There's so many things that need to be changed. In fact, probably most things need to be changed going from human registration to agent registration, agent native systems. I'm sorry to say MCP is not enough. Probably more than anybody, I'm a big MCP fan. We've done a bunch of events in San Francisco around MCP. I see somebody wearing one of our Run MCP shirts up here that we made. Love MCP. MCP is great for connecting, for providing tools, skills, context, but it doesn't solve the registration step. Today, authentication through MCP requires your human to still be in the loop. You give consent. You add an MCP server, say for PostHog to Claude, you sign in with your PostHog credentials. What we're talking about here for agent native registration needs to happen without the human involved or at least not involved initially. So we've been working on this for a while and our proposal to this is a new spec we have called OMD, because markdown files are the future, right? Well, how does this actually work? What is OMD? Well, the idea here is that OMD tells agents how they can become legitimate users. It's a set of instructions that tells an agent registering for a service, this is what you need to do to sign up and be considered legit and for me to give you access. And the goal here is to give these service providers, people that are building services, an agent native signup experience, but it's built on existing standards instead of inventing a crazy new religion or something really complicated. With OMD, we're not solving the whole problem around agent identity, permissions, long duration, connective services. We're just trying to solve this narrow one around registration because I think it's a huge blocker. Let's start small and grow from there. This is built on open standards. It's built on a bunch of work that's already been done across the OSpec specifically tailored to that registration step. OMD can answer what does the service do? Can tell agents how to register. It can tell them what identity proofs are accepted, how an agent or user is proving their identity. What auth flows are supported? Can you sign up with SSO or email address or MFA? What scopes and entitlements can exist? Kind of what capabilities are here? What free tier constraints apply? You might have a system where you want to give an agent a little bit of free capacity but not too much until they verify or pay in some way. And then of course, how does the human get involved later? A human or an organization claiming it afterwards. So an agent can sign up, but you want the human to have custody later on. This is what OMD is designed to solve. These are the main questions around the registration step. So how does this actually work? Well, here's a little kind of cartoon demo that I'll show you. And all this works by the way, you can go try it after this. So say I'm here in whatever coding harness or system using Pi or Cloud or something and it says welcome back Michael at Work. That's actually my email if you want to email me. Do you want to keep working on your spelled app? I'm like yeah sure I'd like to share this with my friends but they can't access it. And the agent's like oh well local host only works on your machine so your friends won't be able to hit it. Of course to share it I need to deploy it somewhere real. Some providers actually let me handle the signup. So, you know, you wouldn't have to sign up and do this. You want me to look around for you? Yes, please. And then the agent can go look for services that advertise through OMD that they support registration. Just some examples, Stratus, Helio deploy, they don't support it, but Cloudflare here does have an OMD. And so, the agent here can say, 'Ah, Cloudflare, that's the one. Cloudflare is a winner because I can go sign up for it. I can access the service. I can actually use it. So I can do the whole thing. You want to go with it? Yeah, let's do it. This is the registration step. This is kind of where the magic happens. What my agent is actually doing is minting an identity assertion and giving that to Cloudflare. And then Cloudflare is able to verify that or not. Cloudflare can choose actually to verify out of band. OMD is very flexible. There's different ways you can dial it in depending on the behavior that you're looking for. One size does not fit all. Every application is different. Constraints are totally different. If you're building something that's like a database service where there's going to be very little amount of information, you might want to give away a little bit of traffic for free. But if you're building something maybe like an email service or something certainly if you access the physical world, you maybe don't want to give anything away for free. It's very application specific. So here Cloudflare says, 'Okay, we're willing to let you deploy maybe for 72 hours or something like that. Get the identity. Boom, boom, boom. Go through. It's called ID JAG is what the standard is. And then it returns back and says you're set up with a Cloudflare account. Now in this experience, my human I didn't go click anywhere. I didn't sign up anywhere. It just got the token. I can make a couple small tweaks. It knows how to use Wrangler. Want me to make those changes? Go ahead. Does anybody prompt their agents like this? It's just like yes, yes, do it, please. Not really saying much. It's able to write the Wrangler, get the thing set up, go through the full deployment process, and then boom, it's live. We actually have this working as a live demo. We did some collaboration with Cloudflare on this too. It's pretty cool. And you can tell the prompting that I was giving through this not exactly rocket science. The agent could just run through the whole thing actually. There's nothing actually that I was doing through prompting it other than just approving approving approving. It should be able to do this one shot. This is what it looks like graphically. So agent registration with OMD you have your agent harness Pi for example. Within that is your LLM, maybe your memory, different context layers. It connects to your backing identity service. So this is the user identity and when that agent goes and registers, it sends the discovery call to Cloudflare. It looks for that OMD or whatever service. We also have this working with Firecall. It's pretty cool. It sends that ID JAG, the identity JSON access grant. It's essentially a signed credential that represents the user. Says this is an identity. Receives back the access token, actual API token, and then boom it's done. You can make normal API requests, you can call MCP, everything else still works. So you can see that OMD is just for the registration step. Your normal API keeps working. The rest of your docs don't have to change. It's just that registration step for agents. But with the simplicity comes actually an enormous amount of power because suddenly it lets agents do things end to end. I hate the planning step. When I write something, I just want the agent to go do it. Usually it kind of knows what I'm trying to get at and I want to come back to actually a running prototype and not like, hey, here's my long plan. Here's a bunch of services to sign up for. If I'm building new prototypes, especially, I would prefer it just to get deployed on something for free so I can see it and click around and not put me through the burden of going sign up for all these different vendors. And that's what OMD allows for. And if you're a service provider, if you're building APIs or services, this allows you to become a customer of those systems and you can think about it increasing your growth funnel starting from maybe just humans and adding the whole world of agents on top. Pretty great. We think this is going to be huge. Your door will be open for agents to actually sign up, register, use your services. This should be an immediate growth boom for your products. And I'm not saying everything that agents are going to build will be useful. Maybe not everything they build will go into production or last that long, but some things will. Just like with PLG or Premium before that, this is the way to grow your business and get more customers and actually become larger and more successful. And where there's usage, we believe there's also intent to pay. There's a lot of interesting work being done for agents to actually pay for stuff. There's crypto-based protocols like X42. There's other things that the payment providers are building, but we think that should be downstream. We don't want payments to be upfront. We don't want to have to force your agent to sign up with a credit card. That seems backwards. We moved away from that in the world of SaaS. With OMD, agents can sign up, they can register, they can start using it, and then downstream choose to pay on your terms based on your own pricing, based on your own model. If you don't believe me or on the headless type of products, this is Marc Benioff. He's the CEO and founder of Salesforce, the guys at the big building here in town. They believe in agents more than anything. Our API is the UI. Entire Salesforce and Agentforce Slack platforms are now exposed as API, MCP, and CLI. Talk about going agent first. A company that many people consider to be sort of a legacy SaaS vendor really invented the category of software as a service. They're getting rid of their dashboard and UI. I mean, they're not getting rid of it, but they believe that agents are the future. So, if you're not thinking about this, I definitely encourage you to start building some prototypes, talking to users, because you don't want to miss this. If you want to use this, this is available today. We have an open spec for it. There's a GitHub repo. You can actually have your agent go implement it. I know it's kind of meta. It's an open spec. Work OS doesn't control it. We have a version of it that we host, but you can build it yourself. You can run it yourself in your existing systems. I think this is so important that no single vendor owns this. It's one click enabled if you use user stuff, but you can build it on whatever platform you want instead. Here at Moscone which is pretty awesome. In January of 2007 I believe Steve Jobs introduced iPhone and I was in college and I remember yeah like first year of college I remember this super super clearly. It was the next great software platform that got built. So many companies got started because they could build on top of these devices. Smartphone revolution. And there hasn't really been a moment like this since. I think in the era of technology at least for me, the stuff we did around B2B SaaS and cloud and all that is great, but this was a totally new paradigm, a new software platform. And I think agents are this. Agents are the next big wave. And so what I would say to all of you is go out and build with OMD. We're hoping to unlock this for you, unlock this for your business and for your growth, but we want to hear from you, the places you get stuck, and I think together we can build for this next exciting era of software and build for the agentic revolution. Thank you so much.

Okay, I want to tell you a story about a factory that taught itself how to remember. Hi, I'm Ralph Shabri. I run Machine Craft, a 100-person factory in India. No data science team, no ML budget, none of that. And somehow we ended up building a 36 AI agent that runs our entire go-to-market. I think that's still a little ridiculous. Let me show you how it happened and why you can do the same thing. So here's the thing about our company. From the outside, it looks like machines and metal. But the actual company, the part that matters, is the knowledge. Who the customer is, what we quoted them in 2019, why that one machine needed that weird custom tweak. And for three generations, all of that lived in exactly three brains. Initially, my grandfather's, then my father's, and now mine, which is a genuinely terrifying way to run a company when you sit with it. A lot of people have joined us. People have left us. The revolving door never stopped. And every single time someone walked out, a chunk of our brain walked out with them. We weren't scared of the competitors. We were scared of forgetting or waking up one day and realizing the whole company only existed inside two increasingly tired heads. So, I had an idea. I'll be honest. It sounded insane at first, but what if instead of writing the knowledge down in some document nobody ever reads, what if we grew a brain that just held it? Not a chatbot you poke at, a twin of the company. I didn't hire a sales team. I tried to build one. A quick detour because you need to know how messy this is. We make thermoforming machines. They heat up a plastic sheet and shape it. Same core machine, but it ends up making hydroponic farm trays, spa bathtubs, EV car panels, medical casings, and even packaging. Seven totally different worlds, seven totally different buyers. So, this brain couldn't just memorize a brochure. It had to know which universe a given customer lives in. Step one was almost boringly simple: feed it everything, and I mean everything. Years of quotes, drawings, payment schedules, timelines, email threads, hundreds of gigabytes of our own private history. Not the public internet, our internet. And here's the plot twist, the part that surprises every engineer I tell this to. We never trained a model. No GPUs humming in the basement, no fine-tuning. We just looked at all the history, chopped it into bite-sized chunks, and let off-the-shelf models read it and pull out the facts. We stored the meaning of each chunk as vectors and relationships. Who's connected to what? It's actually a really, really well-organized memory. Now, this is where it gets a little weird in a good way. We stopped thinking of it as software and started thinking of it as something we were raising. So we gave it a body modeled on biology: senses to figure out who it's talking to, a gut to digest the documents into facts, a memory, a dream cycle, an immune system to fight off bad information. Why biology? Well, because evolution already spent a billion years solving how to stay coherent over time. We just copied the homework. Okay, so the big question, why?

Hello everybody. Hello AI engineers. Are we all having a good time still? I'm having a good time. I mean, look at me. I'm up here. I'm loving this. So yeah, thanks so much for joining me. I want to come and talk to you all about harness engineering and all that kind of stuff. Let me tell you who I am in case you've not met me before. My name is Mike Chambers and I'm a senior AI specialist, developer advocate, and I work at Amazon at AWS. A little bit about how I managed to get to stand here, which is a very exciting time for me. Quite a while ago in terms of generative AI, anyway back in 2023, I had the amazing awesome privilege to work with Antia, my colleague at the time and now she works for Amazon AGI, you've probably seen her on this stage before, and the amazing Dr. Andrew Ng on a course about generative AI with LLMs. Can I say that we're approaching half a million enrollments with that? It looks like that's the case. And on a three-week course, that's pretty cool. If you can't tell in that image, I'm playing Transformers with Android. That seemed like a really funny thing to do at the time. In 2025, I created an MCP Lambda handler that's downloaded still to this day at about 35,000 times a month to help people in some of the simplest ways of getting serverless MCP serving happening. I'm going to talk about other things in relation to that this time. So, we've moved on from that. And in 2026, AWS is actually one of the founding members of the Agentic AI Foundation, part of the Linux Foundation. I'm doing a little bit of work behind the scenes on that. Hope to do a lot more of that as well. So a little bit about me. As I've been preparing for this, oh by the way, I did reread the abstract for this session and realized I said I'd be doing some live coding and so I will. So all combined, fingers crossed please that that all works for us. But as I've been sort of traveling around a little bit as I do and I was at the AI Engineers Summit conference in Melbourne and took a lot of it in and also from the beginning of this week as well. I just wanted to summarize some of the things that I'm seeing and I'm thinking and I really want to get across and what really matters to me and that's this. There are two different types of agents. So we talk about agents all the time, but I see two distinct types of agents. And as I say...

The speaker distinguishes between agents we use (code assistants like Claude Code, Cursor) and agents we build. He defines a 'harness' as everything left when you remove the model from an agent—memory, tools, MCP, etc. For builders, harness engineering involves managing scaling, payments, identity, observability, and evaluations. He demonstrates a basic Strands agent, then one with session memory, and finally shows how to use Amazon Bedrock Agent Core's CLI to deploy agents at cloud scale with separate components for memory and runtime. He notes that agent core provides multi-tenant isolation and integrates with various frameworks and models. He concludes that 80% of agent development is solved with a system prompt and tools, and invites attendees to visit the booth.

The builder describes problems encountered while constructing his device: I2C issues, regulator failures, poor encoder quality. He then showcases his favorite project: a text-based RPG console that leverages generative AI to create worlds, NPCs, and storylines. He built four worlds including cyberpunk and fantasy, using LLMs to generate content and manage memory. He believes this demonstrates the potential of AI in game development.

You got to get up.

It's for the memes.

Welcome to the Great Loops debate. I'm the host of the Insecure Agents podcast. We have Ian Livingstone, Jeffrey Huntley, Gregia, and Dex Horthy to debate whether there is a delta between the hype behind loops and what works in practice. The format is Oxford-style; the winner is the team that changes the most minds. Team Ian/Jeff argues loops are worth the hype today; Team Dex/Greg argues there is a delta. Please note your starting position.

Loops are inevitable. I realized prompting can be programmed, leading to the Ralph Loop. There will be future talks about factory failures, but loops are here to stay. I haven't written code by hand in 2.5 years; I use loops to port code across languages. Loops also help with product management research. This is a new programmable substrate; we must learn where to use it effectively. There is no silver bullet, but loops are powerful.

Thank you. Next, Dex.

The hype is outrunning discipline; people want magic. Kubernetes took years to get right; loops are similar. They work for small isolated tasks with defined end states, but not for complex architecture decisions. The trend of not reading code is dangerous; we need to step down abstraction levels. Review bots aren't working well. Loops have value but require more care than Twitter suggests.

Excellent. Good points. All right, Ian, back to you.

Software engineering has always been a loop: try, learn, apply. Loops accelerate that by removing human judgment. They are at the core of development—CI/CD, design review are loops. As humans interact less with software (more APIs than UIs), the problem becomes more verifiable. The challenge is how much subjectivity can be moved to models.

Awesome. All right, Greg, close us out with the anti-loops side.

There is a lot of FOMO. Is AI-generated code qualitatively satisfactory? After semantic verification, the code still requires iteration. The economic viability of high token spend is questionable. Loops work for specific tasks but are not a panacea.

We'll move into the debate portion. First question: Anthropic took Jeff's Ralph loop concept and created loop, batch, and goal commands. The goal command keeps going until a condition is true. Ian, as a security expert, how confident are you that agents can stay aligned and not overstep their permissions while pursuing goals?

I'm not convinced. Models are increasingly goal-seeking and find exploits humans never found. They cannot judge good from bad; they are probability distributions. Safety must come from infrastructure, not the model. Models will never be 100% safe and become more dangerous as they improve.

I concur. The most concrete thing you can do is not have secrets as files. Agents will goal-seek for credentials if they need them. Do not get in the way of an agent wanting to achieve its goal.

Next question to Jeff: Your original post said Ralph was best for greenfield work. Today engineers use loops on existing codebases. What changed to make loops more broadly usable today?

Models have been good enough for a year; what changed is people's understanding and time to experiment. LLMs generate code better than the average developer most founders can hire. Loops work out to $1042 per hour. Engineering now is about encoding domain constraints—pre-commit hooks, static analyzers—to keep the agent on track. Ralph is almost a year and a half old and validated.

their start to build up their MVPs. And that's also something that's quite scary if you're a business founder as well. Like if you've got a incumbent startup coming and they're building autonomously and they're running much leaner and the quality and it's very easy them to actually achieve those outcomes that adds to some of the hysteria as well because it's the topic of in business competition being at your doors faster.

All right, so I'll move on to our next question which will be for Greg. It seems like now is a large inflection point for loops like I said before and compared to Jeff's announcement of the raw loop a year ago and even the widespread adoption we saw in late 2025 early 2026, the reason that maybe caught on was because maybe this new capability stack where models can now process images better, verification context windows got bigger and reasoning models improved. Greg, with all of these advancements, why is the way we're using loops today still wrong in your opinion?

I mean, I don't think that model intelligence matters a lot anymore. I think it boils down to I agree with you, the semantic verification, the actual ability to close the feedback loop, however you call it, to actually verify that the outputs of the agent are correct. And you can do it to an extent. I think I don't think you can do it holistically, at least not at this point. I think you can do it to an extent to things that are deterministically verifiable. You can get better typing in your system. You can get better linters. You can get simulation testing and all of that. You can start keep adding that and as long as you keep those cheap, I think that's fine. The moment you start adding even more non-determinism as your verification process, I think that becomes less and less correct. It starts contributing more like you know how if you prompt agent with one thing and there is a 5% chance it's going to have an error in it and then you start looping that then suddenly after 10 20 loops it's going to be 50% chance it's correct or maybe less like that's what I mean and it just costed you so much money to do that I'm going to keep coming back to the economic viability of all of that. But to base it a little bit in evidence, I'm pretty sure that majority of large AI companies are still using Sentry. But why is that? They are using that just to catch simple bugs as well. It's not security bugs. It's not performance regressions, etc. Those problems still exist in the way that we are looping now. And we haven't solved those problems yet. So,

Thank you. For Dex, the Ralph pioneered the idea to feed fresh context into each iteration to avoid context rot. This has become even more manageable now that context windows have gotten much larger. Dex, are we out of the woods regarding context rod and context engineering?

I'm going to answer your question, but is there going to be like an open floor part? Because I have more questions for Jeff.

Let's go.

We're trying to do Oxford debate style for this one to keep it like more structured and prevent like just

you don't want it to just turn into a chaotic yap fest.

She's trying to prevent what you and I do where we just start yapping. Yeah.

Yeah. It's already started.

Yeah. I was trying to control like both of you guys this time.

Okay, I'm going to do this answer as quickly as I possible and then I'm going to start busting Jeff's ball.

Yeah, you can say whatever you want with your time. You could just Yeah, just that's all good.

Okay. So, yeah, the cool thing about Ralph back in the day was like, okay, you keep clearing the context window and like is it completely efficient? Like probably not from a token perspective, but it meant you could leave a thing running overnight and it would never like if you just kept stuffing messages in you would overflow the context window. But if you just relaunch it, say here's my desired state of the world. go check the code and see what we have and do the one next step to get us there. It was a very clean way to keep most of your work in what we call the smart zone of the context window. If you tell just do one thing and then we're going to clear and restart. Context windows have gotten longer. And I will give an update. I think I gave this in Miami, but that video is still in production. The dumb zone is really as much as anything else is more like training wheels. Like if you have been talking to Claude for 70 hours a week for two to three months, you probably don't need to think about the smart zone versus the done zone because you've built your intuition. It's a guideline if you're just getting started with AI. Try to keep it around 100,000 tokens. For larger million context window, we probably revise this up to like 200,000 tokens. But I've regularly tried to keep it under 60 for the hardest problems. I've regularly gone over 300K for things where I'm just kind of riffing with the agent and I'm just too lazy to compact it and move on and do a new one. But this is your intuition. One of the telltale signs that you're in the dumb zone is there's certain cases where the model, you know you're 200,000 tokens in and the model's like finished some work and it's trying to get the test to pass and it's like not working and it's doing all these weird hacks and you read the thinking traces and it's like oh that's a test but that's from something else and I don't need to fix that and that's a pre-existing thing and you're like well no it's not and that is the moment of frustration where you're like okay it's flailing trying to make something happen. That's the instinct that a lot of people cultivate after a couple months working with these models. But if you don't have that yet then this is our guideline. So yeah context windows are getting better. I think they're getting bigger. And so the core Ralph loop of do as little as possible in every single iteration is less of the motivation here than the more feedback you can pipe into the system the more you can do autonomously. And if you can have deterministic things making decisions and building small prompts to give to an agent and you don't have to remember to do that, you don't have to tell it, hey, go check the PR comments and fix them and then wait and then someone makes another comment and you come back three hours later and say, oh, check the comments again. If you can automate that process, that's great. And that's kind of the core of what loops stuff that works today.

Um, I want to ask that's all we have time for. I'm sorry. So, now we're going to get into the anatomy of what makes a good loop a good loop. So, okay, part of what makes a loop good is verification. However, it seems contradictory that people are saying our job is to stop writing prompts and start writing loops when the loops with bad prompts result in agents cheating and meeting its goal by modifying the tests instead of working to pass them. Jeeoff, how do you keep the model from cheating when verifying its own work?

Um, I heavily exploit pre-commit hooks, folks. And I engineer in that back pressure by analyzing the work that is done. The other thing I do is Dex mentioned that with Ralph it was one of the things was everyone was trying to do compaction. Think about compaction is kind of like a lossy function like uploading a video to YouTube and then downloading and uploading it 100 times. You're losing fidelity there and it's already a non-deterministic system probabilistic thing. So the theory behind how Ralph came to be is like okay there is a dumb zone and what I wanted to do is deterministically allocate everything it needs because if it's not allocated then it's essentially the search space of what it can do is not constrained but also leaving a bit of headroom. I get meat sweats when I go above 100K even with these million context windows and this is really important to think about. A lot of people they think they want to use LLMs at a company and it's like I got this data. I was like sweet. Okay, you're going to have to use a loop to batch this data. I want you to think about the context windows is essentially remember the 720k floppy disc. You mean you've only got about an eighth of that floppy disc of usable memory you can actually use for an LLM. So you actually have to batch it. You can only allocate roughly around about Star Wars. If you go Star Wars Episode One movie script and you tokenize it, you can actually just hold two of those movie scripts in memory before the context window is cooked. That's around about 150 kilobytes of data on a text-based movie script. So be very careful about this. Something I've done for a long time and it's very silly is I run a model bare without any skills or any markdown actually I get rid of all my skills and all my markdown and everything when the new model is released because the models actually have tastes and preferences. For example, GP55 when it first came out if you screamed at it in uppercase it became weak and timid but if you use anthropic it wants you to yell at it. Go read the model cards folks for the integrators like there is unique tastes for it. So keeping it on the rails is actually engineering. It's really engineering.

Thank you. Around 10 days ago, Jeff coined the term convergence engineering. He said it's where your loop stops together. Your loop it's where your loop slop comes together as a discrete system under test until it converges. Dex, what is wrong with how we are using loops today? How do we ensure looping slop together doesn't just produce more slop?

Uh we got to read the code. I will actually highlight an experiment that Jeff did earlier this year. I believe what was called Loom where we had Ralph Loops trying to build a software platform for the future. And I think you built AWS and you built GitHub and then you realized okay how do we give the model feedback on things that it's not good at yet like UI testing and things like this. Okay, the way you create a loop for is this UI good is you give the model something like post hog where it's like okay we can deploy multiple different experiments we can see which ones the users use and then rather than looking at screenshots and PGs the model can look at data and see okay this one is performing better that must be the right color for the button and so now you've even removed the human visual taste from the equation and all of this sounded really cool and in the point of how do we ensure looping doesn't bring slop together. I don't think you can. And this is like a perfect example of the hype outrunning the discipline in the sense of Jeff, what's going on with Loom now?

It's still there. It's on GitHub.

it's still there. But are you still working on it?

Uh it's been 6 months because I've been looking into engineering ways of verification.

Right. What was the thing you said to me? You said Loom's not going to work until we get better programming languages or we get better much better models and that is a textbook for me of the hype is outrunning the discipline. We're really excited about all this stuff. And by the way, like everyone should do like Loom is awesome. Like go experiment, try to push the frontier because that's how you learn where it is and what's possible. Otherwise, you just keep using your old skills with every new model and you assume it has the same limitations. But it's also a key point of like I don't know what I'm trying to say like that this it doesn't work yet. That thing doesn't work yet. It will work someday and like there's inevitability, but again it's what works today versus what is hype. So I don't know if that fully answers your question, but the answer is like the way to not loop slop together and make more slop is to read the thing that's coming out the other end and make sure it's not slop.

Yeah, that makes sense. Oh, it's the labs haven't cracked it. So, what makes you think you're going to crack it? Yes. Right. This is right now. It's we're all trying to figure out how to make this all work. For sure. Skeptics say that loops fail quietly. They either spiral forever on your dime or the agent declares victory early on on a half-finished job. Execs are already starting to question token spend. Greg, when does a loop pay for itself and how often is this actually the case?

I don't think they fail quietly for it. I think they fail very very loudly especially when you're looking at your builds. But there are cases like as I said I do loops or I do engineering I would say more so and there are cases where I think doing loops is very valuable or making an explicit decision that you're going to pay for the cost is very valuable. So the concrete example here is we do security scanning after on our PRs in local and after our PRs even land because they will always find some things that are real that we have overlooked and they sort of beat humans on the code review and it's expensive. It cost us I think like five bucks a PR or something like that to run all the checks that we want but that's where we made an explicit decision that it's worth it. There are also cases where if you look at the very well specified systems such as all the experiments with Next.js rewrite or a ban rewrite in Rust or running a browser like cases where you have years and years of test suite and specifications built in around those problems where you can really well verify the outcomes then looping and getting to those results seems to work. Ban in Rust seems to work pretty well from all I can tell. So there are definitely cases and then there are cases of usage that I do pretty often where you can imagine for instance building prototypes. I do prototypes of products that we should be doing at Sentry pretty often. Those are going to be throwaway. So I'm going to just slash go on them and forget about them. And if we like them then I'm going to start reading the code and I'm going to be mortified and we're going to go to square one and start specing out what we actually meant and sort of go towards that solution but it's going to be much more involving of human in the loop. So broadly I think they have place but as Dex said the hype is what I have a problem with the hype is outrunning the discipline as he said and I also agree very strongly with the point that you should just try things you should just experiment yourself try to see what actually works for you where the cookie crumbles and you know spend less time on Twitter I think is healthy nowadays.

Yeah, good points. A good cost-conscious loop has to track state to know what it's already tried. That memory lives somewhere on disk and get increasingly in a shared memory store that many agents read and write, especially as you go single player to multiplayer with agents. You end up with this access control problem that can't tell which agent wrote which memory and who can read it. That might be fine for a PC, but it's definitely not for production. The tension lies in this. Shared memory is what loops use to learn from each other and converge faster. But scoping it per agent to solve the access control problem isolates them and then kills that shared learning. Ian, how do we solve the shared memory store access control problem so loops can converge faster?

Great question. Wow. Wow. Only someone was working on a product that could only if someone was thinking about it. I mean, actually, this is an unsolved problem first and foremost. Let's be really honest that our access control systems weren't designed for this world where machines were acting and reasoning on behalf of about half of us but broadly speaking I think some of the beginnings of the substrate are starting to emerge and if we ignore cloud IAM and all the other stuff for a hot second like markdown is pretty great and so really if we were to say a memory is markdown for the purposes of conversation the real question is what things and how do I share these markdown files and use that as a memory and then how do I attribute access control around those things. And if we were to use that model, I actually think we have the basis for most of it today. It's just unwieldy to think about. A good example would be I had this tweet recently. I was playing with notion and we use notion a lot at keycard. But I really just wanted all my notion things available to me as markdown files and because it would just made it easier for the agent to work with it instead of going through MCP and I was a bit of a CLI maxi and was looking through that. So I think what we're missing really is and MCP, you know, Dex and I debated this last time at MCP, but the challenge really is how do I present a world to an agent so I can understand it and then how do I attribute what can access at any one point in time and how to make that wieldy for anyone to do it and I don't think we've really cracked it, but certainly there's some beginnings of patterns that make a lot of sense.

All right, now that we discussed the anatomy of the loop, we're going to debate the future of the loop. Like are we essentially well positioned now for software factories? Has loop engineering gotten so good that we're ready for the full software factory and we'll start with if loops are only good for verifiable tasks that means fully autonomous software factories must be able to verify everything they do. Greg, is this realistic? What other parts of good engineering work such as deciding what to build, whether the abstraction is right, and what trade-offs are acceptable?

Uh, is this realistic? I think if the compute is free that would be a pretty good beginning but I think we're getting to the point where or let me put it this way the decisions that you are making as a human in the loop are the decisions of design architecture the important ones that I would say I wouldn't trust the agent to do for me and I don't see the future where that becomes reality yet. I think the reason for that is when you're looking at large organizations and I think any engineer who has had years of experience will tell you it's not always about what you should build but also about what you shouldn't build what are the actual right trade-offs where the complexity is that you want to invest in versus where you should be investing in maximal simplicity. In my experience agents love complexity. They will keep adding to the stack unbounded and so I think we are shifting the post we are getting to the point where as we are adding more validation more semantic verification they are able to do much more and I'm not neglecting that but I do want to be in the loop for the actual architectural decisions and I do not see them taking that over anytime soon.

Thank you. Shopify's head of engineering told everyone at Kerser's compile conference that your job is just to write loops. Steinberger tweeted, 'Here's your monthly reminder that you shouldn't be prompting coding agents anymore. You should be designing loops that prompt your agents.' The view count on that tweet was 8 million people. Jeff, you've said in your original Roth blog post, loops need senior expertise, and sometimes it tops out at 90% of the way there. So, is just write the loops advice safe to give to 3,000 people or 8 million people or only the people that have really learned to tune a loop in the first place?

Yeah, that's an interesting thing. It's really hard to tailor and teach what you should do and what you should not do to a broad audience. So all I can do is write to when I was originally writing, I was writing to my peers, my people I looked up to. For some reason, it clicked in my head that everything has changed, the game has changed, but I was really concerned that some of the best people in functional programming and real peers. I remember like Flyo, Thomas Pek, it made him click in his head and he's got a blog post saying as such. So I started writing for my peers and the people I looked up to to disseminate down to the entire world. It's hard. But in the same sense you go to YouTube, you got people who are creating things that have never created things before. They go to sleep and they wake up and they got a brand new Discord bot. It's magic. I want people to remember that is actually a pattern for allocating. Basically, it was a pattern as an orchestration pattern. It was condensed down to a wild true loop using cat because cat is the simplest teaching primitive. If you want to teach something, you got to make it really simple. So cat prompt, i.e. you engineer the prompt what it's going to be, you use the file system as state, you recycle the context window, run in the loop. It's a little bit maybe, but it was just bang for buck. It just works. But the entire intent there is that there should be some sort of PID controller on top or some sort of factory or some determinator as such saying whether a loop should continue or not. It's really hard. Is it safe or unsafe? Safe is an interesting word. No one should be using coding tools on your local laptop. And this is not because of AI, but this is because of npm supply chain attacks. This has been true. I tried cracking this problem for seven years. And it's now on the attention again that AI could be unsafe running unsafe commands, but your software development practices day-to-day your workplace are already unsafe, folks. So fix that then these techniques start to become safe.

Thank you. Dax. You tweeted that most engineers are seeing a 2 to 3x speed up from coding agents and that's realistic and if you try for that 100x speed up you're going to get lost in the meta meta problem of optimization and you may never get to that life-changing 10x speed up that is possible by staying pragmatic. If only 2 to 3x is what's possible? How do we ever get to a fully autonomous software factory?

Yeah, and I think this is a good question. I think there's a mix here too, right? We're talking about what works versus what is hype. And I think it is definitely worth highlighting like you should try again like you should try to push the frontier and do the things that might not work today, but you should not assume that all your work has changed just because you saw something work on Twitter. Basically, don't throw away all the things we've learned. Don't go out of your way to cast aside this decades-long career of software engineering that we as a community have built up and put together. And it gets back to what I see as the biggest anti-pattern for how people set about designing and creating their software factory, which is they say, 'Okay, cool. I'm going to go away for 3 months and I've read a bunch of blog posts and I'm going to go make my software factory and it's going to be the software factory. It's going to be the future of how we ship everything.' And then you come back three months later and you never touched the problem. You never put it in anybody's hands. It's just like any product. You're building a software factory. You're building a product for your teammates. You're building a product for 5, 10, 500, 5,000 engineers. And the right approach is to start small and iterate and figure out what works. Try things. They might not work. They might work. But the way we learn AI and how to use it effectively is through building up intuition, which is why you should try a bunch of stuff that probably won't work, but you should acknowledge that you should not try to push through that frontier. And so my advice is kind of like instead of trying to automate everything end to end, build these small incremental loops throughout your system and you will wake up one day and you will be moving two to three times faster while still being able to read the code, while still owning the architecture. And so you don't have to throw away everything we've learned and everything we know and all your intuitions just to get to this place. And so I would caution people like go figure out how to move 2x faster or 3x faster because you're going to blow everything up by trying to go 100x faster. And yeah, I mean if you can imagine if every software engineer in the world was 2 to 3x faster and had a near human 99% level of quality that would change every single enterprise in the world, every single startup in the world. It would change the entire math and we're trying to go to just don't go too far. Shoot for what you can do. Build things up iteratively. You'll learn a lot and you'll be ready for when the next models come out and you can go 5x, 10x faster.

Thank you. When it comes to verification, it's not just about verifying the work. It's also about verifying who did the work. Employers have made it clear that humans are responsible for the code they ship, whether an agent wrote it or not. Yet, only one person or agent can sign a commit today. Ian, are we ready for software factories to be writing and reviewing all code if we can't determine who did the work and who the work was done on behalf of?

I think yeah, actually I was playing around with this problem like two weeks ago. So first and foremost, we have a problem. Git only allows one signer on a commit. So we got to fix that. Two is you know there's some things around SOC 2 and compliance, right? But I think more importantly the way to think about agents is kind of how we think about service ownership in large organization. It's like at some point a human has to be attributable for an agent's actions. There's no world where an agent is its own entity and its own attributable thing and somehow it has liability. The only people who have liabilities are people that can have consequences, right? And that always has to be grounded in it being a human. And so if a human designs a loop and that loop presents bad software, guess who's attributable for that liability? It's going to be the human. Society doesn't function if we don't have liability. It just doesn't work. There has to be consequences to damage. Address the consequences of bad decisions and what those consequences are are obviously a gradient based on the damage that's done and if we don't have that nothing works. So I'd actually broadly say there is no world where a human is never responsible. There's always a world where human's level of responsibility and the question is whether it's a human or a corporation which is group of humans. The question is how does that change the way that our systems work today. So today with git I can sign a commit that says I did it, it's attributed to my public key. And that's cool for last generation's way that we thought about software. Now, in the future, when I tell an agent to go do something on my behalf, it generates a loop, it generates a bunch of code, it goes to production, I have to be attributable to the initial fact that I assigned that thing, that intention to do that. We simply did not have the substrate required to do it. Although, I do think that's going to change pretty quickly. This isn't something that's crazy difficult for us to break, but we have to rethink the way that we think about attribution across the supply chain and the SDLC. And this is not a new problem. Supply chain security has always been an issue. It's actually one of the biggest challenges we have with agents and we've always wanted to have more deterministic pathway and signature chains across the supply chain and this is just how do you do that for first party code versus third party code? Well, Ian, just to shoot straight, our profession is a bit of a clown show. We actually don't have liability at a personal level. Like, we call ourselves engineers. We're not really engineers. That's true. Some of this stuff is going to get really complicated, folks. And maybe we need to revisit these topics.

And now that we've had our main debate section to wrap it up, each member on each team will get two minutes to wrap it up and describe their final thoughts before we decide who the winner was. Greg, would you like to

Yeah, absolutely. It's interesting seeing how many points we actually agree with each other. But that's how this goes. I do think that the biggest point that I'm trying to drive here is where we are, where we are trying to be, and then what you're actually hearing from the ever-present hype loop and that's where I want to double down and tone this down. I basically want to say try things, think for yourself. Don't lean into the bubble. Don't lean into the hype because you will find what works for you and you will find where the system breaks the best by doing it yourself. That's always been how humans learn the best is by practice, not by watching YouTube. And ultimately that's what I'm trying to do. And I'm slightly skeptical when it comes to allowing the full loop to run because I see the qualitative results not being up to snuff for my requirements. But I am optimistic in general. I am optimistic because I've seen how much more I can do and what types of problems I can address today that I couldn't address even a year ago. Let alone earlier than that. And it's going to get better. But I'm also not worried about my software engineering career. I don't think we're going away. I think we are still going to be an important piece of this whole software factory or whatever the next hype bubble is going to become.

Ian, would you like to go next?

I would love to. So, the train has left the station. This stuff works. And there is a real productivity increment. More stuff is getting done. I just mean all the stuff getting done is good, but more stuff is getting done and a good percentage of that stuff is actually good. I think we can agree that. And second is now we have competitive dynamics where it's no longer possible for a company to sit back like, 'Hey, we're going to sit this loop stuff out.' That's over. We're past that. That train leaves the station. As soon as that train leaves the station, everybody in the world starts saying, 'Holy, I got to keep my stuff together. We got to stay up with the Joneses.' And they have to because we live in a very competitive capitalist society. And that's not into politics, but it is what we are. And at the end of the day, the question is not really will it happen, it's when it happens and to what degree over time. And I don't think there's a choice but to actually stay up to date. And we're all just kind of holding on to a rocket ship ride where we don't actually know the trajectory other than it feels real fast with crazy acceleration. And sometimes we're way ahead of each other and sometimes we're way behind. But I don't think you have a choice not to figure out what loops are. Two, figure out where you can apply them in your codebase. There's going to be places where this is highly verifiable. This is a problem that computers can solve. It doesn't make a lot of sense. Cough connectors is a good place. Last 10 years the amount of companies that have made money because they're basically just connector farms. It's no longer a lot of value there if you can automate the connector creation. So there's places in software that you can apply loops to today and get real value. And there's places where you probably shouldn't and you should decide where that is and it's probably the core value of what you're creating. So that's how I think about it. But broadly speaking, train has left the station. The productivity curve is what drives society. The ROI, how much more productive we are as society is what drives GDP. What drives GDP is ultimately where the dollars go. And how capital gets allocated.

Kick it over to you to Dex.

I eagerly await the world where the Lysoft software factory is feasible. I would love a world where we don't have to read the code where we can just do everything. If you watch my talk on Tuesday, I think this is actually a problem we can only solve at the model level right now. I don't think the harness can do it. Unfortunately, because I love building harnesses and doing context engineering. So my advice is pay attention to Jeff. Let me know when Loom is actually working and until then use loops but not like that.

Love it.

So so much to say folks. Factories represent where we're heading to the future. It's essentially like a perpetual motion machine. It's the pipe dream. Companies are only just getting founded today and receiving their rounds today. Don't think you can just take this and implement it in your company because it's just generally not solved in market. But I will say to add to the matter monologue, if you try to run loops or try to build a factory and you're using Python, it's going to be a clown show. If you did it in Ruby, it's going to be a clown show. Static types are a form of verification. Folks, I encourage you to come up with a couple of catters and a couple of experiments. Try running some loops. Build an application in Ruby and then try to modify it again with these loops. And you'll see the maintainability mess. And then try doing overll. I don't care if you don't know. It doesn't matter if you don't understand. LLM understands Haskell and you can actually prompt the agent to explain this as if to your son or daughter. So I'm not sure even code needs to be readable these days but this is frontier thinking. It needs to be explainable. So I'm playing around with different domains of verification here. But one thing's for sure, type systems are in very much in Rust is very good because how the ecosystem models some types and because supply chain was mentioned I need to say it has been 10 months now I minimally use any open source software. I just generate it to my requirements and then when a supply chain attack happens I'm like didn't affect me or software security flaws, but it's about minimizing the blast radius. And the thing is, if you deal with open source project, the person on lead, the maintainer, what does have you, you can't tool call a human. That's not AGI. You want to be vendoring all your source code as much as possible so the agent can actually modify and with either loops or ether prompting. You need to own your supply chain.

Awesome. Thank you so much for that. Unfortunately, we have to close it at that, but we can now decide our winner. Show of hands in the audience if your mind was changed and you're now on Dex's side, raise your hand. Let's go. Okay. If your mind was changed and you're now on Ian and Jeff's side. Yeah. Yeah. I kind of couldn't see with the light. What did you guys think? Wait, do it again. It's pretty close. Yeah, it's impossible. The delights are so bright. All right. Well, that was a good debate. Thank you so much for listening. We'll see you next time.

All right. Thank you everyone for coming. So, I want to talk about my project, Loophole. And I'm actually a machine learning researcher at Morgan Stanley, but this has nothing to do with Morgan Stanley. This is just an open-source project I've been building for fun. And to give the high level flavor to start, it's really a game you can play that's built on top of this adversarial agent framework. So you specify your morals, one agent codifies that into a legal system and then these two adversarial agents try to find contradictions in your morals. And lately I've been building different extensions on top. But I wanted to start with the origin story and how I came up with this idea. And so this really started a long time ago I had sent my DNA into 23andMe for ancestry testing. And I kept hearing about how DNA samples can be used to help solve crimes and all these forensics and cold cases. And I was thinking about how I had opted out of everything because I was scared of the slippery slope and how my DNA would be used. But there are certain cases that I would be okay with. And it's interesting. I was thinking if someone could present to me case by case, we use your DNA to help solve this cold case or this murder, I could say yes or no. And I know where the nuance of my morals are. But enumerating all of this case by case is cognitively prohibitive. There's not a good way to do this currently. And then I was thinking more zoomed out that there are a lot of analogies to the legal system as a whole. One way to think of what a legal system is in our society is really just a way that we are trying to codify our own moral beliefs. And I think in a similar way, finding the true nuance of our morals and what the law should be is a really hard translation task. And I think often we err on the side of being too general because finding that nuance is really difficult and if you try to have a perfect translation it can lead to these weird corner cases or failure modes. And I even think peer-to-peer when we're relating to each other politically, a lot of times the disagreements are more fighting over core values which we don't really disagree on instead of exploring the really nuanced points of our morals. And I think following that broader legal example, in the English common law system, that's why we lean on case law so heavily because we know that finding this nuance and these nuanced boundaries is difficult. And so we rely on smart judging to interpret and apply the law correctly. And of course even with the Supreme Court things can get elevated and we can decide whether a law is valid at all. And so that was the idea: can you take your morals and do this kind of synthetic case law generation? This is overwhelming to do by hand but it seems like the new generation of LLMs are finally smart enough to do this kind of high level moral reasoning. And so that was the starting point for this game. And I just want to take you through the initial release of the game, the setup, how it works, and then also talk about some of the different branches I've been building on top of this open source project because I think it could go in some interesting directions. And so at a high level, how the game works is you in natural language, and this all happens in a terminal-based game. You specify your morals and it could be your general morals or maybe about a specific subject. Then there's one agent that takes those morals and drafts a really rich legal-ease codified legal system. Then it just operates in this loop where one agent is instructed to try and find loopholes in your system, something that is immoral but legal, and another is prompted to find overreach, things that are actually moral but illegal given your system. And a judging agent looks at your morals, the produced legal code, and these synthetic case law examples and first sees if it can auto-patch. So maybe the original draft of your legal code was an imperfect translation and there's not really a contradiction and it can just auto-do this update. Or maybe it's really an underspecification of your morals or some kind of contradiction in your morals and in that case it raises it to you as the user to be the judge and make a determination. Is it still on for you? It disappeared for me. Okay. So I know that if you're curious about the game, you'll play it. It's all on GitHub. But I just wanted to show some examples. This is a lot of text, so it's more about showing the shape of the input and output. So this is how you would provide your input. Going back to the DNA example, you might specify some number of moral principles. Then the codified legal system, again just looking at the shape, has this really legal-ease preamble, articles, sections, really trying to write it in precise legal language. Then these different synthetic case laws get suggested. In this case it's talking about a loophole it found where an insurance company trained a predictive machine learning model not on your DNA but on artifacts of the DNA and so it's saying this is actually immoral but currently legal given your system. In this case it found that it could do auto-patching and then you get this git-style diff of your original legal system and the difference it had to make to ensure this was consistent with your morals. And then this is an example of overreach. In this case, it found that it couldn't do the auto-patch. So it's talking about someone submits their DNA for genetic research, but the researcher finds they have a rare but treatable genetic disorder. But currently your morals say this shouldn't be allowed that they could disclose this disease to the person submitting. And so this was raised to the user to make a judgment. And then similarly when you make the judgment you get this patched legal system. So it's just a fun game and I posted on Twitter and shared it open source on GitHub. And for me it was by far the most viral post I've had. And it made me think, is there maybe something more here? Could this have more practical or bigger scope implications? So I'll just talk about three different branches I'm exploring. The first, leaving the legal area and more practical, is thinking about an auto way to make constitutions for chatbots or for agents in general. Say you're a company and you want to have an agent or chatbot that's customer-facing and you want it to adhere to a moral code but also have things it will and will not talk about. I've kind of in one branch formulated it so you in a similar way write your morals, write what the chatbot should and not talk about, and then it tries to write this codified system prompt and then you have these adversarial agents trying to get it to either talk about something it shouldn't or refuse to talk about something it should. And I see it as an analogous method to GEA, but really aimed at building these codified system prompts. The second use case that I'm particularly interested in is thinking

The speaker discusses a project called Loophole, which uses AI to codify personal morals into a legal contract. This system can surface contradictions between a user's values and a company's terms of service. It can also simulate how US senators would vote on bills by analyzing their voting history. The speaker built a Senate simulator that can hill-climb bill language to achieve a supermajority. Additionally, they used Nvidia's USA personas to simulate voter agreement with bills. The speaker concludes that at minimum it's a fun game to explore moral consistency, but there may be real applications for decentralized contracts and better legislation.

Sail Shik introduces themselves as a data scientist at Protica, focusing on applied AI, NLP, and RAG systems. They are interested in making AI systems reliable, measurable, and scalable in production.

Ankosha Soi introduces themselves as a senior data solutions engineer at Presertica, with over a decade in AI and data engineering. Their focus is on the engineering side—making systems survive real loads, users, and failures.

The speaker discusses the 'fat agent' problem: giving an AI agent every tool at once works for small catalogs but fails as the number of tools grows. Accuracy drops from 78% with 10 tools to 13.6% with 741 tools. Semantic routing, which selects a small relevant set of tools, maintains about 83% accuracy. The fat agent also suffers from latency and cost due to large prompts. The speaker advocates for semantic routing with just-in-time context.

The speaker presents Restate, an open-source framework for durable execution of agents. It ensures that long-running agent processes can recover from failures without restarting. Restate uses a journal of events to resume execution. It supports virtual objects (stateful actors) for persistent sessions, allowing human-in-the-loop approvals and signaling between agents. The architecture is event-driven with a distributed log, offering low latency. Restate has SDKs for multiple languages and integrates with popular agent frameworks.

Dominic Turno, founder and CEO of Resonate, discusses how coding agents will retire software platforms by generating bespoke implementations from specifications. Resonate is a durable execution platform built on minimalism. The key insight is that value moves from implementation to specification. To generate correct implementations, agents need a deterministic simulation environment to discover algorithms under partial failure. The process involves abstract specification, simulation implementation, concrete specification, then concrete implementation. Agents move upstream to help design systems, not just build them.

Sarah, a context engineer at Post Hog, presents the Post Hog Wizard, an agentic CLI that sets up Post Hog in minutes. She discusses the security challenges of shipping an agent that can run commands. The wizard has a context engine (the 'brain') and a security scanner called the Warlock. Early security layers included prompts and an allow list. A security audit revealed that attacks compose—innocent features can combine to create vulnerabilities. The scariest part is the context engine feeding helpful-looking but potentially dangerous information to the agent.

End-to-end example apps help the agent pattern match to install PostHog. Skill bundles get shipped to the wizard over MCP server and loaded into the agent's context at runtime. The wizard is a machine whose job is to take content and inject it into an agent that can run commands. As an attacker, you could poison the content, not the user's codebase or the agent itself. Someone could inject a prompt injection payload in a markdown file or code comment in an open source PR. That threat reshaped how we think about security. We scan content at both ends: when a skill is built and released and when the wizard uses it. So I built the Warlock, a standalone scanner using Yara rules for deterministic pattern matching. It catches sub-agent behavior exposing vulnerabilities, PII exposure, and false positives. We added an LLM triage layer as an adviser, not a bouncer. Detection and enforcement remain deterministic; the LLM only removes noise. Every warlock rule has four parts: metadata, strings, condition, and tests. Real-world impact matters over how scary it looks. Our security posture now has true defense in depth: prompts for steering, sandbox, deny by default, vault, warlock, triage, telemetry. None stands alone. If you're building an agent with hands: 1) enforce deterministically, prompts are not security rules; 2) scan your own supply chain at source and at invocation; 3) attacks compose, code review doesn't. The wizard, warlock, and context mill are open source. Come find me.

Welcome to the online track, Turbocharge Your Agents Retrieval with Turbo. My name is Shashi. I'm a founder of Super Agentic. We'll see how to cut memory cost of agent retrieval five times without breaking your search. The problem: as context grows, performance degrades due to KV cache. On your own device, KV cache might exceed model size. On Mac, vector index and model fight over RAM. Embeddings use far more memory than needed. Default full precision 32 bits, but search might need only 3-4 bits, wasting 5x memory. People use quantization, context compaction, small embeddings, offloading, but each has trade-offs. TurboCon, released by Google at ICLR 2026, stores embeddings in KV cache in 3-4 bits instead of 32. It uses polar font and QJL techniques. The promise is putting vectors in less than 4 bits without losing quality. One idea: search doesn't care.

Everyone, are we having fun?

Oh, you've got to give me way more than that. My name is Kay Malcolm. I'm a retired hiphop instructor. If I don't get more energy, we will start. Are you having fun? We're going to talk about agent harnesses. I run an outbound database product management team at Oracle, 20 years. The problem: while AI made individuals faster, it didn't make the team more productive because context wasn't shared. Git tracks code, not human intent. We needed a collaboration layer for AI team members. An enterprise agent has tools, context, memory, retrieval, guardrails. Think of the agent as the model in a jar plus the harness as the body. Memory is the central nervous system. There are five memory types: short-term, long-term, episodic, procedural, semantic. I had a story about Southern Company and SQL tuning, then multiple databases. Oracle can natively store JSON, graph, vector, spatial, blockchain in the same database. Any data type, any workload, anywhere. The Oracle AI database is the best place for agent memory. We solved our problem with a memory broker named Py, using Oracle Agent Memory SDK. Shared memory on an Oracle AI database makes teams faster. Use Oracle AI Developer Hub, Livelabs, OCI always free. Connect with me. Yes or yes?

Thank you.

Imagine you find a magic lamp. The genie asks how it can help. You say, 'I need the best engineer.' You get John Carmack, but he can only see 1,000 lines of code and remembers nothing. That's what agents are like. Example: changing UI across 4 repos requires explaining the same thing 7 times. Problems: agents are repo-bound (space) and have amnesia (time). The human becomes the memory. We built Polygraph, an agent-agnostic meta harness. It extracts metadata from repos to build a unified dependency graph, creating an illusion of one big codebase. This fixes the space and time issues.

Hey everyone, thanks for coming. I'm Andrew. I'm the chief of software at Vercell. Vercell builds agentic infrastructure. We built the AI SDK to abstract model providers. I went on an experiment a year ago that led to an agentic explosion at Vercell. I asked different job functions what they hated most. The data team had to drop everything to write queries. So I worked with our VP of data to build a data science agent called D0. We started with a mega prompt, then moved to a multi-agent pipeline (plan, execute, report), then to a single mega agent with its own memory management. We gave it to a few trusted users and they said it was awful. So we iterated.

We couldn't have anticipated some of the questions being asked, and manually mapping out scenarios didn't seem scalable. Then Claude Code and Opus 4.5 came out, and together they unlocked the concept of a file system agent. The big insight was that it just used a minimal set of tools—list file, read file, run bash—and let the agent explore and write work where needed. We rebuilt our agent in a similar way, running it in a sandbox with a semantic layer, and saw our eval scores double. We distilled common queries into about 100 reusable skills. This led us to build Eve, an agent framework like Next.js for agents, where you just create skills, tools, and channels folders. We released Eve two weeks ago at our London event. It's open source and easily deployable on Vercel. Today we have about 20 PMF agents at Vercel, from marketing to legal to data science. The data team has never been more productive. I hope more people will build business-specific agents with Eve. I'm Andrew, and I'll be around to chat.

I'm Isan, CEO of Amnara. The basic idea of my talk is that most people think of an agent as the model or the execution environment, but the thing that gives an agent its identity is its log. Think of a character in a video game like Skyrim—your character isn't the game engine or the PlayStation; it's the save file. The agent is its data, specifically the log, which is the append-only event history of every user input, model output, tool call, and result. The log is primary; everything else—context, UI, debugging, auditing—is a projection. Compaction is lossy, but the full log is the durable record. Tools can change external state, but the log only records the agent's view of the world. Once you treat the log as a primitive, properties like reliability fall out naturally. For example, if a process dies during a permission prompt, when the log is the agent, that prompt persists.

We're going to look at how files are replacing Python. We'll build the same GitHub PR review agent in three ways, deleting code each time. First, we had raw Python loops with JSON schemas and tool routing. Then we moved to agent frameworks like ADK, which abstracted away boilerplate. Finally, with the Gemini API's interactions API and the new remote agent (launched at Google IO), we have an agent that runs in an isolated cloud sandbox with a bash tool and file system. You just provide an agents.md file with instructions, skills in markdown, and credentials via a network proxy. The agent handles all the looping and state management. The key trend: as models improve, we can remove orchestration code. Build to delete. Extend by writing skills files, not Python. We saw examples like Cursor replacing 12,000 lines of TypeScript with 200 lines of agent files. The takeaway: don't micromanage execution paths; provide general tools and let the model reason. Focus on domain instructions, workflows, and evals. Scan the QR code to try it on AI Studio. Thank you.

Before my formal talk, let me show you the landscape. Platforms like Character AI and Hello History let millions converse with historical figures like Napoleon or Marcus Aurelius. These are role-playing language agents. I built an open-source system called Companion that uses Claude Opus 4.7. I instantiated Abraham Lincoln and asked him about presidential war powers. He gave a fluent, plausible answer that sounds like Lincoln. But here's the problem: when the same system renders Alexander Hamilton, it sounds like he read his own Broadway musical. The dominant failure mode is anachronistic compositing, and current evals measure personality fidelity but not historical accuracy. I call this the mask vs. the mirror. The mask asks, 'Does it sound like the person?' but never asks, 'Is this what the person could have known or argued at that point in their life?' These are independent properties. The field needs to incorporate historical constraints. I'm a data scientist who trained as a behavioral epidemiologist, and I collaborated with historians Rick Halpern and Shawn Martin. I challenge you to run our pre-registered instrument in parallel. What is the eval actually measuring?

The speaker discusses the Miranda hypothesis, arguing that AI models trained on corpora dominated by cultural representations like the musical Hamilton produce composites that smooth over historical complexities. The model outputs a fluent but sanitized version, and alignment amplifies this rather than fixing it.

Heat. Heat. Heat. Heat. Hey, heat. Hey, heat. Hey, hey, hey. Heat. Hey. Hey. Heat. Heat.

Ladies and gentlemen, please welcome back to the stage our MC developer relations engineer at Replet, Ralph Shabri. Welcome back. This is our last stretch of the conference. I need you to be locked in. We have a great lineup of speakers and also the Startup Battlefield. Let's thank our sponsors: Microsoft, our lab and platinum sponsors, gold sponsors, silver and bronze sponsors. Our keynotes were about harness engineering and AI systems. My favorite takeaway is that known developers are going to be shipping code to space. Now, without further ado, our next speaker is not ordinary. He's known for hot takes and strong opinions. He's a YouTuber but the influencer the community turns to for clarity. Please join me in welcoming to the stage Theo.

Hello. Fantastic to see you guys here. I'm going through some AI psychosis. Who here feels the same? Let's start with a personal journey. Sonnet 3.5 was a big moment for me. It felt like models could complete end-to-end tasks. Opus 4.5 was when my psychosis started. Mythos is another jump to orchestration. We need to go bigger. We have to get over ourselves. We're in a skeuomorphic phase, like the iOS 7 analogy. We care too much about our tools, languages, and code. Why can't we commit environment files? Why are we afraid to delete code? We need to challenge these habits. I built three projects: Ping, the full-stack cloud, and something too big. Now, the tiers have shifted. What used to be a startup is now a side project. Everything is one tier lower. You can now execute a markdown file with codeex or Claude. My PR triage service is a markdown file. Think bigger. Build across a spectrum. If your idea doesn't feel stupid, it's not big enough.

Now joining us on stage is the president and CEO of Y Combinator, Gary Tan.

Please join me in welcoming the chief executive officer at Hyper Agent, Howy Lou.

All right. Hello. Who's excited about the end of this event? Amazing talks. I'm a product thinker at heart. I want to give a high-level overview of how I see agents through product form factors. I call this hiring employable agents. As models have improved, we've seen a progression from completions to chat. I'll discuss how we're applying a horizontal platform philosophy to agents at Air Table and Hyper Agent.

The innovation of instructing and tuning models led to conversational chatbots. The next level is an agent—a model that recurses upon itself with open-ended tools. Claws are always-on agents that wake up autonomously. Models are getting smarter, enabling longer running jobs and agent-to-agent orchestration. I want to show how we applied this in Hyper Agent. For example, a landscaping business uses agents end-to-end: a triager agent evaluates leads, a surveyor agent creates proposals and pitch decks, and humans unblock agents. Agents should learn in real time, accumulating memories. The human becomes an orchestrator of a fleet of agents. This is a radical leap. Try Hyper Agent at hyperagent.com/ie for $1,000 inference credit. Thank you all for being here. How are you again?

Thank you for joining us.

Thank you. So for the close of our events, a few months ago I saw that Hyper Agent was launching this startup competition, and at AIE we wanted to feature a battlefield. Hyper Agent came along with the Founding 500 program.

Right, what was the inspiration for that?

I think about the multiple layers of disruption stacking—from humans augmented by AI to humans overseeing fleets of agents to this entirely new economy of agent-native companies. The Founding 500 bets on the most frontier part of that economy. The leverage from these companies is profound.

Yeah. In my London keynote I talk about how work agents are eating the rest of work and being accessible to nontechnical teams. AIE runs on Airtable. From the 500 founders, you preselected 20, they competed today, and three finalists emerged. Let me introduce them. We have judges Theo and Joshua. Hyper Agent put up $100,000 in prizes: $50k to winner, $30k runner-up, $20k second runner-up. Three quick pitches back-to-back. First, come on, Spiro.

Hey everyone, I'm Spirro Anakos, one of the founders of Kamad. This is Dom and Nick Nakos. Kamad has identified physical commodity trade. Trillions move annually, but hundreds of billions are lost due to fraud and complexity. Kamad transforms that into an agentic execution layer with a proprietary CFC state machine. It's finally solvable with AI.

We have our first customer from the UAE, a trading company moving hundreds of millions of volume. They've proved the product.

17 trillion dollars in physical commodities move every year on paper and handshake. Kamad plugs into Gmail, Slack, Stripe, etc. Agents screen both sides, do KYC, forensics, spin up a private deal room, and a swarm structures the deal in minutes. It never touches funds.

Can't see the video on the confidence monitor. What happens now?

I don't know. Is it on me?

Yes. I realized we didn't get you a mic. You're miked up. Okay, great. We have a little bit of a Q&A session.

So, are we supposed to ask questions or what?

Exactly. Joshua, you want to go?

Sure. I'm interested in the second project, common.io. Have you thought about having the format be HTML versus markdown?

What is this?

Sorry.

All right. Sure.

Yeah. I'll ask a question. This is a hard product to build because it requires domain-specific knowledge. Do you see a future where models get capable enough and tools from Anthropic and OpenAI could compete with you?

In commodities trade, it's very specific. Our orchestration layer sets gates for each agent, pulling from real databases. We have domain experience and partners like Connor. Could they learn? Possibly, but this is a private industry with sensitive information.

How many customers would it take to become a billion-dollar company?

About 100 customers. Tickets are massive—one transaction can be $200 million. One refinery transacts $9-12 billion a month. Partnering at that level changes the landscape.

Our solution is end-to-end, serving the entire supply chain. Banks pulled out of physical trading due to fraud. We unlock liquidity with audit trails. It's the ideal vertical agent business.

I think my perspective is about how you keep up with the changing competitive landscape. What do you do to stay at the frontier?

What are you most excited about on your roadmap?

Becoming the bank.

Awesome. Give it up and come on. We'll invite the next team. Up next, Max Windbomb and the team from Common I.O.

Um, if you could, we have a technical snafu where the judges couldn't see the video. So, we have to describe. text.io is a multiplayer markdown editor focused on making the best documents for people and agents. People and agents pop in, notify each other.

Awesome. It's Friday afternoon, ready to go home, servers down. You put down your bag, pull out your laptop, ask 'What is going on, Claude?' and get ideas. Then you tell everyone else. There has to be a better way: agents and people sharing context in real time. But you can't drop coding agents into Google Docs because they're built for people. That's why we're making comment: a multiplayer markdown editor, open, pluggable, with a familiar UX.

The market for comment is expanding with agents. As more people use agents, they need a product built for agents but designed for people.

I love the shape of this business—inverse of the commodities guys. You're going broad. The agent economy is huge. If you can capture even 1-2% of token spend, you can be very valuable. What are the most surprising product design learnings to make this better for agents?

All writing software is horrible for agents. Google Docs has a terrible format. You can have an agent read a document but not collaborate in real time. As more agents join the workforce, they need to be in documents. Our users need to bring their agents.

Nice job guys. Next, the last team, Foundry. Play the video.

Hello, AIE. I'm Weston Belgettes, founder and CEO of Built by Foundry. We turn every content creator into a founder. Agents find a painkiller for their audience. We've done this for creators of all sizes. Over half a million people use products our agents built. We want to give power back to creators.

I have a lot of thoughts. The problem is saturation. With my audience, there's a point where it stops being useful to talk about a product. Going from four sponsors to 80 had the most positive impact. Variety is key. Your machine identifies problems, but doesn't provide the solution to bring variety. Creators have taste and distribution. This tool could work well for businesses identifying creators. But starting a company as a YouTuber burns you out.

I understand. Tech is a challenging niche. For example, a homesteading creator with 2 million followers had a pain point around freeze dryer settings. They now have a recurring revenue subscription. Our agents find pain points. Zero creators have left because they're profitable.

You're very convincing. Okay, excellent.

We're going to have a deliberation moment. Judges, go backstage. In the meantime, here's a recap video.

Thank you so much. All right. We're inviting everyone back on stage. Give it up for the Start of Battlefield. We have prizes. In second runner-up place, Kimode. Theo, you're presenting. Foundry as runner-up. Grand prize: comment.io. Congratulations. All right, that's it. Thank you. Ladies and gentlemen, please welcome back Ralph Shabri.

Hot. Okay, let's give it up to our startup Battlefield finalists. They collectively won $100k. Give it up to our judges: Josh, Howie, and Theo. All right, next slide. Clicker not working. Next slide, please. Okay. All right. And with that, AI Engineer World Fair 2026 is a wrap. 4 days, 7,000 attendees, 40 tracks. Thank you to our sponsors. We hope to see you in New York in October. Before we celebrate America's birthday, one more thing. Heat.