The world that the AI lives in is all digital, and there's the actual world, and like those things are not necessarily the same.
When we started doing AI for coding, it was autocomplete. Nobody really knew to what extent you can actually build complex software using AI, and then it basically broke through the expectations because we were able to model more and more complexity in the systems.
I just don't know if operations is constrained in the same way. Where's even the data to learn how to like build a system that understands like all of the operations?
My assumption is that we have all the data. The question, I think, is maybe about complexity and like to what extent can you take the things that you've described that are operational and convert them into something that an AI can reason about. The dirty secret for the longest time of running like a distributed cluster with just human beings managing it.
Do you think we're at a spot where you could like use AI to run one of these services?
I think that there's enough to be able to...
All right, Jad. Thanks for joining me on the podcast. For those of you who don't know, Jad is the CEO of Greer, but also we went to grad school in the same group. And when I first met you, we had a new student. I walked into the office and there was a motorcycle helmet and a pack of Marlboros. I'm like, 'This guy must be unbelievably cool.' So my question is, what happened to all your coolness? Question number one.
Maybe just for context, maybe give a quick overview of your background. So just for those that are listening, you know, Jad is one of the best systems thinkers I know. I love talking to him. We're going to go kind of a deep dive of like engineering from a systems perspective and how AI is impacting us, especially when it comes to things like operations, but just to ground it maybe a bit about your background.
Sure. So, you met me at one of the peaks in my coolness.
Yeah, you're very cool. Motorcycle and Marlboro cigarettes. I remember thinking, I'm like, 'Wow, are cigarettes even legal?' It was like so discordant.
I remember Nick seeing me one day smoking outside of Gates and looking at me and going like, 'Nick Bun is our adviser.' Gates was the building.
Yeah, actually one of my favorite things... Do you remember Balaji Prabhakar, the professor of course? My favorite thing about him is that he smoked. I was like, the only professor... I just thought it was like great.
Just to be clear, I don't smoke anymore.
No, too old, too middle-aged for that now. But let's see. So you know, I went to Stanford with you. You were always the guy I had to look up to and be like, hey, this is the guy I want to be when I grow up.
I think it was... I started four years before you did. So I was ahead.
And so you were four years ahead of me. You graduated a while ahead of me. You went and started Nicira. I stayed and stuck around, worked more on OpenFlow. Helped build a bunch of stuff on the hardware side from a security perspective. Actually, one of the most fun projects that I worked on there was I built the first set of SHA-3 implementations that people were considering using, and I built them in hardware to kind of show whether they work or not, for one of my projects for my thesis. That was fun.
Yeah. Yeah. Cool. And then I decided after a while that I actually don't like networking because it's just about moving bits from one place and putting them in another place.
How hard can that be? I think you would have to either be philosophically aligned that it's deep or not. Most people assume it's not. I think it's very deep.
I mean, there's something interesting about it, but ultimately to me that's like, it's not...
Yeah. It doesn't do it for you.
Yeah. And then I actually went and did a postdoc at MIT with Nickolai Zeldovich, who's also amazingly deep and super smart.
I did some application security work at the time. Started a company based off some of the research I had worked on with Romesh Chandra over there, who was a PhD student at the time, and ultimately the technology was too early.
So what we did was, well, you know, ultimately a lot of the work was actually really Romesh's, and his PhD was on, well, hey, if you get attacked and you get exploited, can you recover from that by rolling back the state of the whole world that's related to that application and then roll it over on top of a patched version of the application to remove the exploit. And so we started doing that in the real world and we're like, ah, that sounds really hard. It's actually kind of relevant to what we're talking about, which is like to what extent can you actually bound the state for like operational systems, which I would argue is very hard, but yeah, we almost got it working.
Almost. I feel like for all long-tail disciplines, this is why people get in like the fly trap for smart people is like you see a bit of progress but you're toiling. But anyways, I digress. Let's keep going. Let's keep going with your...
So then I joined AppDynamics. I was there for about three and a half years. They went from about 200 people to 2,000 people. Got acquired by Cisco. I was a Cisco employee for one day. And I said...
And then you're like, 'Jad, I know, I love you. Come work with me. Come do this VC thing with me.'
I'm like, what the... What is VC? What? Like I'm an architect. I do like engineering stuff. What do you... He's like, 'Great. The less you know, the better.'
Yeah. Literally what you said. It's true.
And then the first day I started here, you said, 'Slow down.'
It's actually people that move from an operating world to VC, it requires a lot of reworking the brain.
Yeah. Just like going from a PhD to operating, by the way. Like you have to like undo the brain damage.
This is my experience. This is what the real world is like.
I know. And then I stayed here for a couple of years. Did a bunch of work with you. It was a really fun time. Worked with a lot of great people like Jennifer, Mark, and Ben.
Satish. Matt was here when I first... almost towards the end when I was leaving. And then joined Apply as their chief product officer. Helped them build their engineering product teams. Kind of scaled from about 15 people to over 90 across product, engineering, infosec, and design. And after about three years, I kind of had a head for each one of these pieces that I owned, and I kind of was sitting around just coordinating and doing meetings all day and wanted to go do something else. So left and then started talking to a lot of engineering leaders about the problems that they're facing and what they're trying to do. Everybody at the time, this was in '23, everybody was saying, 'Hey, we're trying to reduce spend.' And I asked them, 'What are you spending money on?' Everybody would say observability. And you know, well, first actually it was infrastructure and then observability. And so I started looking at that problem trying to understand like why do people think it's too expensive?
Turns out there's a few things. First, people were spending somewhere between 5 and 20% of what they spend on infrastructure on observability. But ultimately what they were really trying to spend it on was reliability. They wanted to buy reliability and...
Reliability of their infrastructure and applications.
Right. And so they were putting money into observability as a proxy for reliability. And for the longest time, the more money you spent, the more reliability you got out. But as architectures changed and turned into microservices that are much smaller, that are much more vertically integrated, the volume of data that's being collected at every layer expanded. There was much deeper market penetration for observability, and the volume of data that you're collecting and the cost of observability was actually starting to grow faster than the units of function that you're outputting. And the marginal increase in reliability for every additional dollar you were spending on observability was just flattening out.
And so that's why people thought it was expensive, but people were still willing to pay 5 to 20% on reliability. And so the question was like, can we actually solve that problem and bring back that marginal increase in reliability, not necessarily by spending more money on observability, but some other way. And so the thinking was like, okay, great. If we can figure out what is the signal, what is the noise, can we actually turn the observability data that you're collecting, that is mostly fluff and mostly insurance at this point, and do that kind of extraction of information upstream, do whatever you need upstream, and be able to ultimately massively reduce the cost of observability downstream. And if you can extract that signal in real time upstream, then you can one, reduce the cost, but two, you can now feed it into other systems and use it for driving reliability operations afterwards.
And at the time there was something that was called AI coming into the picture, and we're like, well, great, if we are in the data stream and we're extracting the signal, then we are in a strategic position to feed that to whatever is going to come in the future because that's going to be necessary.
And so that's what we built. And so Greer helps companies, you know, we started with helping them reduce their spend on telemetry massively. And so you put us in the data stream, plug and play, pass the data through, reduce your cost by 90%. And now we're building out that second layer where we're feeding those signals into the AI layer that's now going and troubleshooting and figuring out when things are going wrong and fixing them.
I'd love to just talk through kind of your experience on how AI is changing dev generally, which is like a lot of people get asked this question, a lot of people answer, but I just feel like the engineer's engineer, like you've been an architect, you've run product, you run engineering teams. Like, actually, how big is your company now? How many people?
Yeah, so you know, and you've got a... I mean, you're building a lot of software with a relatively efficient team. So maybe just a high-level sketch on like, well, if you're a senior engineer like you, you know, that has architectural experience, like what sort of gains do you think people are seeing? Maybe talk through like how you use AI, what gains you're actually seeing, what is the stack, what is your process, how you think about it.
So maybe I can give you like a historical view of like what's been happening in terms of like running a team and so on. And so when we first started, AI was still kind of autocomplete, right? I remember like at the time we were using Copilot and IntelliCode, and we're like, oh my god, look, it autocompleted these tests. We're like, yeah, but it's just autocomplete. And very rapidly things started evolving. And what I've noticed is that really good engineers are able to somehow put it into their workflows in a way where the AI starts training them.
In terms of like how to talk to it, how to get the best out of it.
And what I do, as an aside, do you think that's model specific? Like is it, do you learn how to use AI, you know, writ large, or do you get efficient at a given model?
I think there's like some nuances that different models bring. I'm sure looking at your tweets you've... There's like nuances, but there is a way to work with AI. And I think that there's also a way to work with different harnesses. And I think like, you know, we tried for example using OpenCode or like different harnesses, or like using Codex versus Claude Code, like different ones have different ways of managing context or thinking through problems or when to spin up the right sub-agents and what to tell it, like the kind of prompts. Like for example, with Claude Code, I find that if I don't tell it don't spin up, you know, 20 fable agents to do basic research, it'll go and spin up 20 fable agents to do research and burn all your tokens. And so like a lot of these things, you start learning about the harness and the models and like how they interact with each other and then how they can interact with the rest of the world. But I digress, like I'm just going in my mind. Like this is kind of what I started noticing is that like some people were adapting and learning and kind of doing it that way, and others weren't as adaptive.
And then as we started hiring more people into the team, that disparity between, you know, okay engineers and really good engineers just multiplied 10x.
Was it... can you predict who would be good? Was it like good engineers got better and bad engineers didn't? Or was it really... Okay. Yeah. So like if you're a good engineer, you're just a better good engineer. It's not like there's some like bad engineers all of a sudden became good engineers.
No. No. It always ends up being like you have to really understand ultimately the setup because the more complexity that your brain can handle, the more you can break down the tasks and kind of assign them. I think this might actually be like a non-consensus opinion that like, I mean, I do hear it sometimes, but the idea that like the better engineers are much better and those that are less...
And you found this to be pretty universally true?
I mean, look, my team is still small. We haven't done it across 100 people.
In my experience, for the things that we do, that is the case because we build distributed systems and distributed algorithms, and like your brain has to handle the complexity because like...
Yeah. Yeah. Yeah. And so for the problems that we solve, none of the models can actually think through the things that we do. Like there's a lot of, for example, we have a query translation layer. We were just talking about query optimization.
And so you have to really understand the semantics of the query and how to get to the semantics of the query and then explain it to the model and then have the model go and build the translation layer, let's say.
Yeah. So for the people that are listening that may not be so deep in computer science, what do you mean by semantics?
So semantics are really the meanings of things. Like the semantics of the word dog is the animal that actually walks on four legs and barks.
And the syntax is the fact that it's just a three-letter word.
And it has three different letters. The shape of the word is different, and that shape can be different between different words. But different words can actually also mean the same thing. And so they might have the same semantics, but they might have a different syntax. So a dog and a puppy might be slightly different semantics. They refer to the same, but they're still called a dog. So the way that... and just for computer science, the way that I describe it very simply is syntax, you know, is like the characters you type, and semantics is what the machine actually does, right? And so it's like the actual runtime properties of the system.
So you did computer science, I did electrical engineering, you became a computer scientist and I became a VC. So there we go.
That's the problem. I see. Yeah.
Anyways, I just wanted for people that are listening, I just want to understand like there's the actual syntax which is writing the code, you know. And then when you're talking about syntax, you're just talking about the code itself. Then when you talk about semantics, you're literally talking about you have a machine that interprets that code whether it's compiled or not. And that machine does something. The semantics is what the machine actually does. And these things are, you know, clearly very different, right? Like so for example, some semantics you can't even derive by looking at the syntax, right? You don't know what the machine is going to do because whatever, you kind of have to test it, you have to test it, you know, like and it can go all sorts of different ways when you test it. And like a very classic version of this is SQL, for example. Like you can write, you know, some SQL query, but like how it actually performs, how long it performs, depends entirely on like say what the schema looks like or what the data looks like in the database, etc. But that's... I would not call that semantics though. I would call that performance characteristics.
Oh, sure. Yeah. I think that would be like... I would say like the semantics part of that would be like depending on like what the shape of data is or like what the configuration, you might end up with different semantics because like you said, you want null to be treated as false. And you won't know that unless like you also know the configuration of the system and so on.
Okay. Okay. Okay. So for the purpose of this conversation, I think we should just say that there's like the actual code you write and there's how the system behaves. And we'll call the second one semantics just so we can have a conversation.
There's a... Well, what do you want to call them? We need to dive into the semantics of this and make sure they're clear.
That's right. Because there's a lot of... I know. Okay. Okay. We'll keep it high level. Okay. So, sorry. So let's go back. So, you know, actually before we started the podcast, you and I were just shooting the shit and you were talking about like how you can do projects now that take a week where in the past would take a team and months. Can you maybe walk us through a little bit about like how you personally kind of do projects and then kind of the gains that you see? So like a little bit about like your stack and your process.
So I use Claude Code. I'm going to say something that's actually pretty funny, but I use Windows.
What was that created by? This company Microsoft? And I use WSL?
So I use Linux for everything like development and so on. There's no like X or UI for it. I just use the command prompt.
Because I just never really wanted to learn the keyboard shortcuts.
Are you a gamer? It's like you tab over to Counter-Strike or whatever. Like why else? I'm not hating on Windows. I mean, you know, like I used it in the '90s.
That's really like the only reason is I never wanted to learn the shortcuts. I hate the one-button mouse. I hate the one-button stuff. I hate the number of modifiers you have to type for shortcuts and stuff. Like I just... I'm like, why, why, why, why waste time.
Okay. So like yeah, use Windows. Yeah.
And so WSL, I use Claude Code.
That's Windows Subsystem for Linux. So they actually run the Linux kernel within Windows in a hypervisor and provide you access to it. It's great. They've done a great job.
Great. Windows, WSL, and Claude Code.
And like the latest thing that I do is I use Opus 5.5.
Which is a really good model.
And I've shipped in the last two weeks these massive 20 PR things that I just dump on my team to go and review and they're like... I'm like, don't worry about it.
Horrible. It's like the new reporting structure, like the people below review the PRs for the CEO. Is that where we become like...
No, everybody's like writing the same PRs too.
Yeah. I review my people's PRs and like everybody's like just PRs flying around in PR.
Yes. And but like the thing that has changed a lot in the review cycle is that like we started much more focused on the tests and like doing things like golden expectations and like are they shifting in different ways and so on. But Opus 5, and like the trick that I've recently learned about is that instead of using sub-agents, you basically tell Claude to spin up command-line sub-agents because those maintain their persistent session over time. And so you can have it spin up these agents separately. You can actually run them from other sessions and so on. And that allows you to be much more efficient from a context perspective.
How many agents do you run? How many... how many what? Agents?
Agents. Typically I work on like three or four session windows at the same time because I found that more than that I just start losing context between the different things because like they start finishing faster and then they wait.
And do they interact via the PR? Is that how it works? The sub-agents like together?
No, they interact directly through the command line.
Yeah. So like I ask the main thing that does the implementation to have a tech lead reviewer and like they basically ask it questions. Does this make sense? Does this not make sense? Review my PR, etc. And so they iterate on these PRs together. And then they have a PR doc that basically they work on with a design doc and everything in it. Like the first thing that I do is I ask that agent to do it. Ask the tech leads to review the PR doc. I go through the PRD. I go through the design doc. Make sure it makes sense. Go through each piece in like detail. Make sure that all the algorithms work and like all the pieces work, the division of labor works, the division between PRs and their relationships work. And then I tell it just go start doing it. Figure out if there's like parallelization that you can do and then spin up other agents to actually go do them. But ultimately you have to kind of put it all back in one stack and be able to go through it as a human one by one because otherwise it just gets messy and bushy and you can't really tackle it, you know?
So how much time... so can you give a sense of like a project you do now versus I don't know, six years ago, like time differences? So the latest project that I shipped was something... by the way, this is like, to be clear, this is production code. This is not demo mockups, whatever. So like the statement you're about to make is the gains that you get for like shipping production.
Yeah. So I'll give you the example of the last PR that I worked on, or like the last set of PRs, was what we call auto-promotion in a data lake. So in a data lake you store all the data and typically the data is column-oriented parquet files and Iceberg and so on. And when you get data from logs, they're semi-structured, they're arbitrary columns and so on. In order for it to have the best layout of the data and to be able to query it efficiently and fast, the ideal situation is that every field in the log event becomes a column that you can filter on independently of the rest of the rows. And so we call that auto-promotion or shredding. And doing something like that is incredibly complicated, right? Like you have to go and figure out how does the metadata system work, how do the commits of the metadata system work, how do you coordinate across different writers and so on. And that took maybe about 10 days. In previous days that would have taken maybe about six months.
Wow. And was this just you or...
Insane. Yeah. Sort of like factor of 10, factor of 20 improvement.
For production code, for new products.
You know, one question I always have is like I can very much understand, you know, these models are trained on all the code ever written, let's call it, right? So I can understand why they're able to generate code, which is the syntax piece. But then like for, you know, you're doing core infrastructure, let's say you're doing distributed algorithms, like the properties, like how it handles itself when it's running is very, very different. Like often you just can't even look at the syntax to know what's going to happen, right? Like let's say you're writing some distributed system and then like the way that it will perform, the availability, the correctness even is going to be dependent on what the workload looks like, you know, like what are the access patterns, what is the traffic...
But are you talking about like the performance aspects of it though or are you talking about like the...
Well, I'm thinking like availability, like you may have to lock things so it's not available for a period of time. I'm talking about correctness, like maybe you've decided to do eventual consistency and so like if you do a read it's not going to be consistent because there isn't a lock, right? And then performance, like you know maybe you cache, maybe you don't. These are all kind of trade-offs in the design space, right? And like unless you know like the access patterns, you won't know how it's going to perform in these things, right? And so I just think to myself, okay, I've got an AI, I can understand how it will write code for a generic distributed data store, for example. What I don't understand is how it will know to make the trade-offs needed because it doesn't even know what the access patterns are. And it's not like these things can... like where's even the data to learn how to build a system that understands like all of the operations, right? Like today we rely on human beings to manage clusters and to optimize databases, etc. And so can you maybe talk about how you think of like, I get the writing the code, but like you have these deep operational systems, like how do you apply AI to that problem and like have we made any progress at all or is this still an outstanding problem?
The question I think is maybe about complexity and like to what extent can you take the things that you've described that are operational and like that happen in production and convert them into something that an AI can reason about.
And exactly, like how can an AI reason about this system? Well, first off, like to what extent do you see that it, writing the code, that it can actually predict what's going to happen? Like how good of a job does it do at that? And the second one is like, okay, now how do we make it reason about that? So let's ask the first one, like if I use your method, the Jad method of writing code and I built some big distributed system, like how often does it actually get it from a performance, correctness... how often is it like actually operationally like pretty sensible? And by the way, I work on this distributed Gaussian splat renderer and I would say it's horrible. Like because like you know you've got millions of Gaussian splats that you've got to page in and out and stream and like it just totally depends on like the running system and so like none of the models do a good job.
So in our world what I found is that we've defined basically really well what performance benchmarks look like and how the system should run and what the constraints of actually building within like our stream processor look like to get the right performance. But every once in a while it'll do like... my latest project has been all about query optimization and query performance. And so I was like, hey...
The classic example of something that's...
I was like, we are basically generating code that corresponds to SQL and we want to run it in a pipeline.
And we're like, okay, well you're generating code, fix it.
And it started basically going through and what it didn't realize was that it wasn't actually doing initially any optimization on the code. It's like, oh, it's fine. It looks like, what's the problem? And I said, well look, there's these conjuncts over here you're doing and like this particular term is repeated and it should have been taken away. And then it's like, ah, okay, let me figure this out and so on. And then like it basically went on a thread to try to figure out what are all... it built a bunch of hypotheses around like what are all the things that it can go and optimize to make this better.
But ultimately it was able, because it was able to ground that in performance benchmarks, and I was like, look, here's a data set of the queries and like what the users are asking and so on. Go work on it. And so it can basically do a relatively good job that way. But when it comes to saying something around like...
How much memory should I give this system?
It has no idea, and that has to come from actual experiments that run in production.
And so what I think actually these systems seem to be doing well at this point in terms of operational stuff is that it knows that, hey, I probably won't be able to answer this question. Let me run an experiment somehow and come up with that information. But of course, when it comes to operations, like production systems don't behave the way that you expect, and ultimately that's why incidents happen.
So have you found or are you doing any of like the AIOps stuff where you actually have a running system that has logs and you feed it to an AI and that kind of helps you like tweak it? Yes. And how well does that work?
Because anecdotally, my experience with other people doing this has not been that great. Like the reports I've been getting anecdotally have not been great.
So, a couple of things. The first one is we're doing that for both operational pieces and for security, but the way that we're tackling it is not the way that other people are doing it from an AIOps perspective. A lot of those systems basically start from a lagging indicator like, hey, this metric spiked or like this alert happened. Go back through a very complex system and try to determine why that happened.
