Great. Hi everybody. My name is Tomasz Tunguz. I'm a general partner and founder at Theory Ventures. We're an early-stage venture capital firm. Half of us are AI engineers. We build and invest in lots of technical products. And before this, I was a product manager at Google building large-scale machine learning models for ad targeting, and then worked at a couple of startups. And today I'd like to share with you my journey with local AI. So I love local AI. I think it's a very important part of the future when it comes to inference, and over the last six months I've been developing a system that I will walk you through. So the goal was to build a personal productivity system powered by AI agents. Obviously during those six months the capabilities of local models have exploded and they're basically now three months behind state-of-the-art, even the ones that you can run on a laptop, and things that I do with it: investment research, deal pipeline, communications, mail reading, all kinds of stuff. I started in late 2024 and so I learned Ruby on a typewriter that was tied to a fax machine and then eventually started to learn how to use it really effectively and developed an initial system that was built entirely in Ruby, 37,000 lines of code in December. And then I'll walk you through why I migrated to Rust and then the impact. And then we'll talk about a new technique that we've been using internally called skill distillation, which allows you to take a lot of the intelligence of a large model and actually condense it into something that a small model can use, and what the overall impact is. So started with the initial system, about 103,000 lines of code across 213 files, and we were using techniques like DSPy and GEPA. So DSPy is a way of breaking down a query and parameterizing it. So taking a component of the query that goes to a large language model and making it a variable, and GEPA is a real optimization system for getting a prompt to perform. And what we noticed is it actually added a tremendous amount of complexity to the overall system and so we abandoned it. But this is the way that it kind of works, right? You have a user manage all tasks in a sauna. A large language model will pick up that task on schedule, say every 30 seconds. At the time I was going between Claude and Gemini. It was going to an execution file called execute.rb, hitting a bunch of APIs and then generating a result. And it worked really well. Like you could do email on it, you could research, it would validate what was going on. And then I read about Rust and I don't know, and I still don't know today how to program a line of Rust, but I will tell you it is an amazing language. And so if you don't know it, you should not know it the way that I don't know it, because it's really good. And so I started migrating with Codex over to Rust and we were able to reduce the total number of files in half and the total number of lines of code in half, from about 103 to 46,000, so about 55,000 fewer lines of code, obviously a lot easier to maintain. Again, we were able to double the total number of APIs or data sources or things that we could ultimately do with Rust.
And one of the reasons, there's this great blog post that Cloudflare published called Code Mode. And the core insight behind Code Mode is LLMs are really great at writing code. So whenever you want to do something with an LLM, have it write code. And to do this, like across those 50 APIs, what we do is we say, well, if a query comes in, hit a discovery API. So instead of feeding the context window full of every single capability that a system can do, just tell it it has one capability, which is it can search for the stuff that it can do. That's called a discovery API. And so there's two different things that you can do with that discovery API, or just one, which is really just a search. Tell me what I can do. How do I hit email? How do I hit Notion? How do I hit the CRM? And we got a 72% reduction in token count just as a result of this because we're not stuffing the context window with every query, and a 99% routing rate of success. And we couple this to QMD. So QMD is an open-source library, it was started by Toby at Shopify. It's a markdown repository and it contains lots of different information that might be useful for an agent. So when I tell an agent I want to do some really complex task, it hits the discovery API and then it calls QMD. That new task comes in, large language model extracts the intent. What is the purpose of that task? It searches for any relevant information which includes skills. We'll talk about that in a second. Maps those skills to the APIs using the discovery API. There's a pre-flight gate which is, will this work on a local model or not? And then if it does, it hits a local Qwen 3.635B, 635B, and if it doesn't, it escalates through a cloud model. And that's the architecture of the system as a result of migrating to Rust. In this system, this is on a logarithmic scale, so in December I generated 51,000 errors. It's a lot of errors for a single person. And then by March I was able to reduce it by 99.3%, effectively asymptote to zero. So the system works very well.
And then a key part of this system, in addition to managing tasks, is email. So every single one of my emails actually goes through the system. It hits, the system hits Gmail. There's a whole bunch of Rust code that's associated with it. And then it's really important to, when you read email, read it in a secure environment because you can imagine the prompt injection attacks can be significant. I'm not sure if you saw the, there was an Israeli entrepreneur who had a video where he sent himself a Google Calendar invite and the second he opened the Google Calendar invite, all of his garage doors opened. So that's a very, you know, quirky way of demonstrating the kind of prompt injections that you can deliver in an unknowing way. And so the way that the email is triggered is there's another daemon that's running every five minutes. It reads Gmail on a local machine. It spins up an Apple secure container which is a relatively new technology. And inside that Gmail, inside that inbox, it converts the email into JSON, structures it, processes it again with local Qwen, validates that the JSON is correct. That container has literally no permissions to do anything but this. It cannot talk to the network, cannot read the file system, cannot call anything else. And so as a result it's just as close to a secure enclave as we can get. And then once that JSON comes out, then the large language model goes through the routine that I just walked you through: QMD, pre-flight, hitting the discovery API, and ultimately generating the tasks. In addition, secrets management is really important. Everything's on the macOS keychain. We talked about the local-first email architecture. So this is the way that it works. An email comes in. There's a secure email processor. The email is analyzed. It's actually categorized and then the relevant skills or workflow is associated with that email. Is this a scheduling email? Is this a referral? Is this, you know, a pitch deck or whatever it is. And it's super secure. And so, okay, so what are the results of all this? So 92% of the decisions taken by the LLM are local when it decides what to do. Across all model calls of everything I do on my computer, 58.2% actually run on the machine. This approximates Stanford research that was published last week that shows that for most white-collar workers, even for many basic coding tasks, about 70% of all your work can actually be executed locally. And in terms of all task completions, 99% of them are completed locally. So I live in this system and just to give you a sense of like, you know, what does this look like day-to-day, there's actually an agent that goes and captures what percentage of tasks actually complete end-to-end locally running Qwen. In addition, as a result of running local Qwen, I can do lots of parallelization. So what I found with Claude, particularly in Q1, was I was often rate-limited, but on a local model I can have that Qwen answer three, four, five, or six or seven queries in parallel. And what we do is we take, we take a task and we immediately classify it as a simple task or a complex task. If it's a simple task, it automatically routes to the local model, fastest possible execution, typically less than 50, 15 seconds to execute that task. If it's a little more complicated, it fills a local queue. And if it's really complicated, it fills a cloud queue. And so, as a result of that, across 23,000 completions, you know, we're able to reduce the queue length from 8.6 to 3.5 tasks being managed in any individual batch. And we get the QH to drop from 73 seconds to about 27 seconds. So overall, able to reduce the overall task duration for completion about 82%.
The last technique I'll walk you through is something that we call skill distillation. So many of you might be familiar with model distillation, where a very smart model is used to teach a very small model to be smart. You can also do this with skills. And so how do we do it? Well, you take all the trajectories coming off your system, the work logs, and you pass it to a frontier model, and you tell it, I want to find and cluster all these different logs across different kinds of tasks. And once you have the top 10 clusters, you have it, you say, create a skill and create for me 15 different, 15 to 20 eval categories and make sure the skill can pass on the cloud model 100% of the time. And then once you've done that, you tell the cloud model, spin up a Pi agent or an OpenCode agent or a master agent, as I learned recently, and iterate on the skill until the local model achieves 100%. And then when you have a task for the local model that looks like that, the local model just executes the skill with better than 99% accuracy. And that's called skill distillation. And it works incredibly well. So yeah, anyway, that's a local system that we've built. It's hugely, I mean, I think it's incredibly performant and I think at, you know, at some point in the future here, 50 to 70% of most people will be, most of their tasks will actually be executed on the MacBook that's on your knees. So I'll open it to any questions about the system. Yes.
