We're thrilled to kick off our afternoon session here at Convex Abstract. Welcome back, Jamie Cawling, who will be moderating a session today on whether agents are the future of software engineering or an extremely expensive way to create bugs. So, here's Jamie and our panel.
Hello everyone. Okay. Everyone in the back, please come on in. I'm personally very happy to be on this panel with these three people, whom I genuinely respect for many reasons. I respect them because each of them believes in doing the work. I'll introduce them all to you shortly.
Actually, before saying a lot about them, let me just introduce them now. Last but not least is Theo. If you're active on YouTube, Twitter, or any other platform, you must know Theo Brown. He is the creator of T3 Code, T3 Chat, and T3 Stack.
Heard. What's that again? Heard. Heard.
He is a big supporter of Convex and has contributed a lot to our early success and recognition. Next to Theo is Jean-Denis Greze, a true Renaissance person and an extraordinary engineering leader. I've known Jean-Denis for a long time, and we're good friends. He was the CTO of Plaid, and now he's the CEO and founder of Town, which is an agent for your personal and professional life.
Next to Jean-Denis is David Cramer, founder of Sentry, who was the CEO, CTO, CPO — held every role — and has maintained his firm opinion at every step of the company.
I respect these three because they're all builders. They actually do the work, and each of them has a specific opinion. I'm very happy to be on this panel with them because their opinions are based on real experience. Right? It's very easy to have an opinion about the world. But having your own perspective with real values is very hard, and I see that in these three people.
Today's discussion is about agentic development. I'll see where the conversation goes, but right from the start I want to know: so far, how much business value has agentic development brought to your company?
I'm sure we'll get different answers from the panel members, especially regarding usage, because agentic engineering has had a big impact on how our platform is used. But specifically talking about development, how much technical or intellectual value have you gotten from using agentic development?
As the non-AI company on the panel, I honestly don't know. It's kind of funny because we don't have those fun growth bubbles that some companies enjoy. But we're always looking for ways to use it, and I think we do find value in it. I always look at software. It's a productivity tool. That's all it is. That's the only reason software exists. AI is another step in increasing its effectiveness.
Getting value from it is quite complex. So I'd say, so far, most of our value has come internally. We're spending a fair amount of money on frontier models for coding and other tasks. And it definitely helps. Getting information is a big deal, and it helps people be more self-sufficient. But as a product, I'd say we're still not sure about it.
Everyone knows what Sentry does, but we're not seeing any big change in the overall production space. A lot of people are writing code now, but that code doesn't really have users, so it's not like there are more bugs being created on Earth than before. The code isn't getting much better or worse than before. It's not like all the bugs from existing software have gone away, because we're just writing more. And I don't think the overall way production works has changed. Honestly, it probably won't change much at all.
So we're working on two phases: R&D, where we're trying to do things we couldn't do before, and finding new markets. Personally, I find the latter more interesting, which is basically expanding the market — meaning more people who can use our software, even though there's nothing new in the software itself. We're just continuously improving it.
Jean-Marie, I think your answer will be different when it comes to productivity. Your team is small and the product is new.
Yeah, it's hard to say that this improvement curve ends here, right? Because we built the company from day one in this new world. So we're post-GPT, that's probably what they call it. Okay. I don't know. I think for the new generation, everything changed last November, right? And we focused on our current direction right away.
The thing is, I have a mental model of how many people are needed to do a job, and secondly, a lot of old work feels much easier now.
For example, Town is like an assistant that can connect to many platforms. I previously built Plaid. At Plaid, our engineers would build integrations or connections with banks, right? It was complex and required extremely skilled engineering. And at Town, we've basically standardized the process of building integrations, and it's now done entirely in an agentic way because the contract is easy to define.
So, building integrations, which used to be hard, expensive, and a company's core strength, is no longer that important for this business. So I think, for some businesses, where value is created and how it's captured has changed. I find thinking about it this way more interesting than overall efficiency.
And the second thing I want to say is, there's a kind of engineer. We're an applied AI company, right? So we don't have to do much systems work, right? We use Convex or Sentry for systems work, right? We don't rebuild those from scratch.
But in applied work or product engineering, I think people who can work at the intersection of product engineering and design and don't need to communicate with anyone else can work much faster. Because previously, communication constraints between engineers, designers, or PMs were a big obstacle. It took a lot of time. They had to agree on everything.
But now if one person has all three skills, they can work very quickly on a feature. So I think, for any new feature, while the speed of finding product-market fit increases, it can be a bit fragile or less sustainable, right? I think once you've found product-market fit, you have to come back and focus on proper engineering.
And for proper engineering, agents can't help much because they repeatedly destroy abstractions or the core structure. They seem to really like spaghetti code. You'll tell them, 'No, don't go that way, I've already created the abstraction. It's right there. Please don't destroy it.' But they'll do exactly that.
It adds a new library. It's a new layer. Congratulations. I gave you a new test framework. Exactly, yeah. I like that you can't do TDD because it'll just change the tests. Uh-oh, yeah. You should only keep two repos. Just don't give it the keys.
So, Theo, you're someone who's very close to your product's developer community, right? How do you see it — if you were writing this code yourself, what percentage of the impact of development comes through agents?
It wouldn't exist. I mean, it's clearly much better and unbelievably good. And how much better the models get will show how far they can go — everyone has their own criteria for that. I see some people make fools of themselves and keep changing their criteria as the models improve.
That's why I try to ask questions like: at what point will you actually stop reading code? At what point will you actually click the merge button when the agent tells you the code is good enough? Because if you don't set a criterion now, you'll keep moving it. So pick what seems reasonable to you, because we'll get there. For a lot of things, we've already gotten there.
And honestly, maintaining a huge open source project like T3 Code is only possible through agents, because now our users can contribute for fixes or features they need. They can fork and work on it themselves if they want, but a team of two like ours couldn't review thousands of PRs if agents didn't test them and flag the important ones for us to see. Meaning, without AI, our entire process would make no sense.
Yeah. I mean, we noticed one thing: people get a bit annoyed when reviewing code, right? It's not a pleasant experience anymore. It was never fun, but now the situation is a bit bad. I'm asking — Theo already answered — do you review code at your company? Do you read code at your company? Do you think it's important, and is that changing now?
Yeah, for us the question is whether it's safe? There are parts of the codebase related to the API layer, authentication, how data is stored, how it's encrypted, and the keys. So, I think when we touch these things, there will be some friction or delay on GitHub.
But there's a lot of work where, especially when you're in the product layer and not dealing with how data is stored or managed or ackles, there's often not much need to test it. And the funny thing is, we've been using Devin more recently. One great thing about it is it shows you small videos of the feature you're building, right? Meaning, from a product perspective, you can just watch the video to test whether it works as you need, without having to click around repeatedly.
And it saves a lot of time. You're still testing, right? And if your CI/CD is very good, your automated tests will catch a lot of problems, right? Or the small issues that LLMs commonly have. And I think this brings a huge change in mindset.
Yeah, I agree. I think, where's the accountability? Like, you wouldn't make a big change to a database without reading it. That's just not sensible. Why would you take that risk? Do you want to delete all your data? Because you know the agent won't work the way you want, because you can't fully explain what's in your head. So you have to calculate: if the agent makes a mistake, will it be a big problem? How important is this? That's where you make the decision.
I often regret it when I don't read code, because the agent will always do something annoying. But it depends entirely on what the task is and how easy or hard it is to fix if something goes wrong.
And code review is quite annoying, it's the most frustrating part. And when you didn't write the code yourself or don't know the context behind it, it becomes even more annoying. And it's especially hard because even though agents are good at completing tasks, they're not very good at following ambiguous instructions, one reason being the memory concept that could solve this isn't there yet, and you'll never be able to describe things well enough to them.
And even if you could, what's the gain? You're just wasting time, spending more than the time you were supposed to save. So I don't know. I don't think this will change without some big change.
At Sentry, code review is mandatory for all production system code. Yeah. And I want to make it a bit flexible in some cases. But it's hard, because when can you skip code review?
Yeah, actually I was never a fan of code review because I think at the end of the day you have to take responsibility for your own work. At Convex, we do review all code. We have a big responsibility to our users, but I think ultimately code correctness depends on the developer.
But code review does serve some important purposes, right? It's a means of education. Thinking of code review as a way to find bugs isn't really a good idea, right? But code review is a good way to evaluate someone's architecture or verify whether they built the right thing. And especially code review as a means of developing talent. Meaning, as a way to make your engineers more skilled.
So, have you thought about how you'll build these skills in the next world without code review?
Have the agent explain things to you. I get annoyed when I hear people say they can't understand how the codebase works because it's getting too big. Because there's a genius on your computer who can tell you exactly how it works. So you can keep asking questions without feeling bad about annoying the team's senior engineer. Getting an idea of how everything works is much easier now. Sometimes it can be wrong, but any engineer at a company can be wrong too. You just have to keep working with it, take a second opinion, ask questions, and understand through inquiry how everything works.
Going back to the earlier point, I agree that code review was never a very good way to catch bugs. I can say, there have been many code reviews I did where I thought everything was fine but bugs were caught later, and that happened more than catching bugs by looking at the code.
The hardest bugs will be solved, or they won't ship, if they're easily caught while writing or reviewing. That part remains the same as before.
What I'm seeing here, and what's quite interesting and a unique engineering opportunity, is building in a way that helps your agents deliver work more effectively and reduces the risk of potential failure. Think of it this way: if your team size grows 10x, how much more code will be created? The solution of increasing the number of engineers 10x can never be reviewed 10x more time.
Instead, you should build your system so that failures are less costly. So that if something goes wrong, you can catch it quickly, fix it quickly, and ensure that the impact of mistakes is as low as possible given the amount of software released.
And now the two of us have a team running a codebase where thousands of PRs come in every two weeks, which we review and release. We merged over a hundred PRs in the last 24 hours and had no big problems because we gradually built the necessary systems: to catch problems before release; to identify them during the PR; to test and review the work we create with agents; and to verify the changed systems.
And most importantly, to structure the system from the start so that everything has a clear place, so agents don't make irrational mistakes and don't try to fill gaps in strange ways.
Yeah, you said something very important, and that's architecting the system correctly from the start. And most of what you described is abstraction, right? How you'll build the building blocks to represent your system. How do you think about building architectural sensibility?
Make mistakes many times, then you'll gradually understand the right thing.
Do you think gaining architectural skill is becoming easier or harder than before?
Good, I was waiting for this discussion.
Even now.
Architecting a system is hard now, because the agent can never get it right. The fundamental problem with LLMs is that everyone has attention deficit or ADD. You're thinking, 'Oh, I want to do a lot more work.' And then it seems, 'How can I take more shortcuts?' Which was always true, frankly. And LLMs are the worst version of that tendency.
You're thinking, 'Oh, I'm quite excited about this change. I want to finish it quickly.' And you're thinking, 'How do I finish this quickly? How do I quickly gain confidence?' And then it turns things into a mess. The worst part is, some problems are very hard to find, and you won't know until they hit you.
I think the fundamental problem is, they're very good at designing in isolation. And usually you don't need that kind of intention there, because with architecture or systems, the problem is it's usually dependent on subjective or personal opinions. It's built on various constraints that the agent definitely doesn't know because you haven't told it. But it won't know that by default. It just can't.
So it might ask, 'How big is your system's scope? How many users do you have? Are they all there now?' All these small things. And I think isolated problems are usually correct. But when you integrate everything, basically everything goes down the wrong path.
And in my opinion, personally for many people, software design comes through repetition. It comes through the process of building something, building it again from scratch, and repeatedly building it, and there's a lot of subjective work in that you have to keep in your head. And agents can't do that very well.
So even though you can theoretically build things quickly, you're just adding patchwork work and it's very hard to clean up. I think you might not fully understand how the system works, because at some point there are many moving parts. Because you're taking on work of a much larger scope. And I think this actually makes things quite hard.
If you give an agent to an old system and all the engineers working on it, I can understand that a bit, but...
I'm also talking about new systems.
Building systems to test new things and verify their effectiveness is much easier now. For example, I'm currently working part-time on my own cloud, and I spent the last two days on a new model. Thanks to Enpropic for Fable 5.1, it's quite fun. I did a lot of load testing to verify the performance of different architectures, which was in my head and was also suggested by Fable 5.1.
Testing these things and seeing how these theories work in a real system is quite fun. My ability to verify these things, work with different solutions, and try something else if the first three ideas don't work is much greater now. This kind of learning would have taken me a few months before, and I might not have done it at all because writing 5,000 lines of code, checking performance, and not seeing it as fast as hoped...
That would have been very frustrating. This is very liberating and enjoyable, and I'm learning a lot more because the cost of making mistakes is now nearly zero, since you can rewrite anything from scratch if you want.
Yes, theory and practice — because practice... I don't know about you all, but there's a saying: you learn by seeing, you learn by hearing, and you learn by doing.
Who here likes to learn by doing? Yes. So why are we watching code and listening to talks when we could be learning by building something real?
>> Then stay here for the rest of the day. >> Sorry. >> That's for tomorrow. Think about tomorrow.
Um, here we've discussed two things. I think, I think what Kramer said here — psychological anchoring and talent development — are two extremely compelling topics in the industry right now.
We're noticing one thing: OpenAI and Anthropic are quite openly saying that they're now only hiring experienced engineers. Part of the reason is they can, because they have the money, right?
Um, but they're also aware that we don't need junior engineers at our companies, because we have agents and so on and so forth.
But that means they're buying already-formed engineers. You know, they may not be creating talent. Um, so they're hiring.
It's a lot like the early days of COVID when we all started working from home, when everyone's productivity jumped a lot because we already had plans and suddenly everyone started executing those plans.
But when it came time to plan anew, that became quite hard, right? Um.
Are we taking advantage of architectural wisdom? Like Theo, you have good taste and you built all of this. Are we taking advantage of these existing competencies?
And is this — maybe this is a question for Jean-Denis. Is it becoming harder to build a team? Is it becoming harder now to instill a sense of systems aesthetics in a new graduate?
>> Yes, sorry. Um, I'll try to answer that, but honestly this was a great discussion, although it felt unsatisfying to me.
>> Oh, okay. >> Actually, it's very... yes, I'm just trying to figure out what's underneath our discussion here.
I think it's a lot like here — there are elders among us who share my view, and we... David too. And James is reaching that stage as well.
But we have a notion of what good and bad architecture is, which was formed at another time. Back then, if you joined a company as a junior engineer and started building software,
and you were given a very complex project — like someone said, 'Build this thing, you have two months. We'll talk in two months.' The junior engineer would usually come back with garbage or something unusable, and it wouldn't scale, right?
And fixing those mistakes was very expensive, so you'd have your junior engineer work with a senior engineer who would help them with the design.
I'd say we had a mental model around people's capabilities and the amount of complexity they could take on.
And I think what's happening today is that a junior engineer can create much more technical complexity than before and solve more complex problems,
but compared to the old world, the scope of architectural drift — of what good architecture is — is much broader.
Now, the point you're making is that it doesn't matter, because you can quickly re-engineer it, refactor it, and I completely agree with you.
And I think when I think about what's good and bad architecture, I look at it from the perspective of what was good or bad 10 years ago.
And I don't think that's the right way to judge good and bad architecture now, because now correcting a certain amount of drift or deviation is very easy.
But I still think there comes a point with AI where the deviation becomes too much, the complexity grows a lot, and then the cost of correcting that deviation becomes enormous; even the LLM struggles to fix it because there's a mess all around.
>> So is the fear here that junior engineers using these tools will create even more spaghetti code or a mess and ship it?
