Hi, I'm Andy and I'm in San Jose at the Compound AI systems conference. We rented an abandoned steakhouse filled with the ghosts of Morton and other dead steakhouse founders. That's not who you hear in the background, though. Those are my friends who are the leaders in AI, and we're going to put microphones in front of them and ask them questions about power dynamics of open versus closed, where the frontier in AI is headed. And let's go ahead and get our first researcher and put a microphone in their hands.
Thanks, Matei Zaharia and Heather Miller for joining me here at LDC. Matei, you're a co-creator of Databricks and you're a professor at Berkeley, previously Stanford and MIT. You created Apache Spark as a PhD student. Heather, you're a professor at Carnegie Mellon right now. You were advised by Martin Odersky and were a key contributor to the Scala programming language. You're also doing a startup in addition to your current professor job, and that's in the space of compound AI systems. Is that right?
Cool. And you are both co-chairs of this conference, right?
Both of you are big champions and cornerstone stewards of the compound AI systems community, which we've convened today here at an abandoned steakhouse haunted by the soul of Morton himself, who, for those who don't know, his soul can still haunt things even if he's alive. I'm pretty sure my soul is haunting three steakhouses right now while interviewing you.
Wow. Okay. Let's talk briefly about compound AI systems. How do you pronounce it, first of all?
Oh, the name of the conference.
Yeah. Well, after I've been saying CAIS, but everybody says different things. I want to say CAIS, but my brain keeps saying case. Okay. So, what makes this conference different? This is the first run of it.
Yeah. The point was to bring the, you know, researchers together for this new field. And I think what's really cool with this conference that like Heather and the PC did is they actually got great reviewers from many fields.
Yeah. So the composition of the selection committee was intersectional across the various subcomponents in the compound systems ecosystem.
Yeah. Exactly. A thing I'm spending a lot of time thinking about is the community as a whole that champions and works on open research in the context of AI has a new set of ecosystem and economic dynamics that it exists within that I think are new compared to previous technology paradigms like the one we built Databricks around, like big data, cloud computing. And one of the big differences in AI is the amount of resources required and the resources available in industry versus the scarcity of resources in academia, and not just academia but anybody trying to do open research, since so much industry research has become closed research. Is there something like a gestalt property or some sum is greater than the parts that the open ecosystem and researchers working on different parts of the compound system, as we interpreted at this conference, can lean into more to keep open at fighting weight with the closed labs where they're so much better resourced right now? Is that a part of the theme that you think about organizing this conference?
I mean, I was looking to you for guidance on how to... I'm looking to you. So we have an infinite loop now. Yeah. No, no, Matei. Break it. Break it. I think we have ideas on both sides. Actually, the whole Berkeley SkyLab, I mean, like, you know, the Patterson Lab style of having a lot of people work together on kind of larger, more impactful projects together. I mean, that's kind of a step in the right direction, honestly, right? I think that we should be doing more of that.
Yeah. Yeah. I think it would be cool to get some cross-institution efforts like that going. There aren't too many in computer science, but it's starting to happen. I think so, for example, like OpenThoughts, this dataset of like thinking traces that you can use to train models, ended up with a lot of authors on it from different places, and I can imagine more things like that. There's another example of a pretty academic field where this happens, where experiments are expensive, which is like physics with the physics experiments, right? You build like the Large Hadron Collider, it's a huge project, then you do these experiments and you end up with a paper with like a thousand people on it or something.
Yeah, I think an observation about the Hadron Collider is it's infrastructure that serves as a platform for researchers who have independent research hypotheses to go explore, test, and have access to the shared infrastructure, a common infrastructure.
Heather, you have tried some creative forms of fundraising that I was involved in. Describe in brief how you've approached fundraising at the intersection of startup land and academia. At the moment, there's a lot of interest and many things happening and taking off, and there's no shortage of interest in academia. In fact, you know, everyone is panicking because the NSF has basically been kneecapped. And so let's just match up the two things, which are the desire for, you know, research transferring quickly to industry to impact product, right? So I mean, VCs and early-stage companies typically are trying to pull something out of academia and then build a product on it, right? That is a strong desire. And then there's a strong desire to do that kind of work. I mean, academics want to be relevant, right? And so it's just like bring these two things together, and you have done that. You've brought them together.
You know, we have this model where hopefully we can use investment for a startup to fund research that's relevant to a startup, right? It's trying to bring them closer together and put people in the same room.
So when you fundraised for your startup, you said we want to earmark a certain amount of the seed or pre-seed round that'll just go straight to fund PhD students that are working, that who downstream of their research to startup will productize it.
Yes. Correct. And kind of like align all the incentives at the intersection of open research and the tech transfer to product. Go to the researchers that, you know, build things, build things in the open and want to see these things take off. And, you know, the point is it's a cross-university thing. Hey, maybe, you know, finally we get some of these cross-university efforts off the ground.
Cool. Oh no, it's Andy Pablo. People banging on the window. Hi. Cool. Okay, last thing. If there's one outcome that you would like to see happen from this conference that you're co-chairing and organized, what would it be?
I think it's people in different fields talking. I'm already seeing it, but yeah, I'm excited to catch up with everyone and see the next, you know, wave of ideas that come out of this.
I want to see people, you know, submitting next year's papers together.
I mean, it's not just like collaborations for hippie-dippy feelings, but it's like again, you know, you can only carry the weight so far by yourself.
Yeah. We need the sum is greater than the parts.
Great. Okay. Well, thank you for chatting. Yeah. And thanks for organizing the conference.
What are we going to talk about?
Whatever you want. Talk about research directions, optimism, pessimism for the field. Would be fun.
Okay. Joey Gonzalez, thank you for chatting with me at CAIS. You're a professor at Berkeley. You did a postdoc at Berkeley, same time that I was finishing my PhD there. You've done some startup work and multiple startups, GraphLab, and more recently Letta.
Run LLM. Yep. You're intersectional. You work, you're passionate about technology research and transfer of that into startups.
Yeah, I work on the kind of intersection of AI systems. Actually this conference is kind of a neat place bringing agentic systems and AI and the ACM and the systems community together. And it's kind of a neat story how systems and AI, separate fields as they came together, this idea of scaling became a central theme of that mission. My thesis work and Matei and others, you know, in that time period were how do we take machine learning and not just make the models better but bigger? How do we access more data and would scale unlock new capabilities? And it turns out scale did play a pretty big role in unlocking new capabilities. And so that kind of fusion of fields, of different minds coming together, systems people, engineering mindsets, data mindsets, and statistical modeling mindsets, that made AI systems and really made the modern era of AI.
How do you see what's playing out now in the kind of landscape from open research and closed research going? And are you worried or do you think everything's copacetic?
This is a hot topic, a concerning topic for me. Where is open research going? Where is closed research going? It's crazy that, you know, five, six years ago, OpenAI was an open thing and like open meant something. We're in a really scary and exciting time. So research is going much faster. A single grad student can do way more. And so there's a lot of opportunity for us to innovate, to not just, you know, climb the hills that we have, but to invent new hills to climb, which is awesome. But at the same time, the great students doing really cool stuff are getting plucked out of the PhD programs and pulled into these labs, offered comical amounts of money that I can't compete with as a professor. And I think there's a real risk that...
We're talking like multiple million dollars per year in...
Millions of dollars per year. Yeah. And I think there's a real risk that the next generation of innovators, of thinkers, won't exist because either they got pulled out of grad school or they won't even go into CS or into AI research. And I think there's so much opportunity for graduate students, for computer scientists to shape what AI is for all of us. And it would be a shame if that was just, you know, pulled into a small set of organizations.
Yeah, I've often thought about the risk in terms of the PhDs who would have likely stayed in academia and created the next generation of faculty that would mentor the, you know, subsequent generation. There was this recursive dynamic in academia because the core program is adviser-advisee. So you have a close partnership and the gist or the main reason to do it is to train up the next generation. And it turns out that the best way to train them to be as luminary as the current generation is to work on research. Like you're actually learning a vocation, which is to be one of the world's best researchers. And the best way to do it is to spend five years glued at the hip to one of the world's best researchers. And the big labs don't have that mechanism. So I've thought of that as the main risk because they break this recursive loop that we've had for millennia. But you're also pointing out another risk factor, which is the discouragement of the generation one below that coming into undergrad and halfway through the undergrad dropping out of CS and not even getting to the PhD program or even through computer science as an...
I want to emphasize this is a very important point. We got to where we are today not just because of great academic research but because of that collaboration between industry, between academia, OpenAI being open, Google Research being open. It inspired the next generation of graduate students, maybe the next generation of undergrads to go into this field. That excitement, that energy is critical to the future of our field. We can't just hand that off to AI. And I think I'm very worried that we're quickly moving to a situation where that excitement is lost.
So what are you doing about it?
Yeah. Yeah. So, what am I doing about it? So, I'm still in academia. I haven't left my job. I'm trying to... Yeah, that's one thing. I'm trying to encourage graduate students to take that risk, to bet on themselves, to be able to be the pioneers that will take us where we need to go next. Part of that is picking research agendas that aren't just climbing the hill that the billions of dollars are racing to the top of, but finding new hills, anticipating where the field's going. I think for my career, if I look back at each point in my career, I basically assumed I would lose at the thing everyone was doing. So when I started my PhD, everyone's building models. I go, you know what, I can't build models as well as them, but maybe I can scale them. So once they have good models, I'll scale them. As I started as faculty, I was like, you know what, I can't scale models as well as Google, as Facebook, Meta at that point, but maybe I can serve them because once they build really big models, the next problem will be how do we use those models? And so I went to inference. And now I'm... what comes next? I think maybe what comes next is the challenges that AI will create for the world around it. So one of those challenges is the systems we build need to be able to handle this new workforce of very capable agents that can do far more, far faster than a human. And so I think there's a real opportunity to rethink the software systems we built that were designed to serve humans to now serve AI more efficiently, more effectively.
What's a specific example?
So a great example, take a database. We're at an ACM conference. A database today is designed to take queries from a human or maybe from a very simple dashboard. But you could ask an agent, hey, solve my business problem, and it can ask thousands of queries per agent. And they have thousands of agents working against that database. They'll ask harder queries, more complex queries. But the database doesn't have to answer those queries. It can say no. And this is a new era for how we build systems, that the system doesn't have to be as forgiving, as friendly as it would be to a human. It can say, here are a thousand other queries that I can process more efficiently. Can you rank them and choose your favorite? And I think that'll be critical to making our world more effective and, you know, more adapted to what AI can do and what it can offer.
And your sense is that the big closed AI labs aren't focused on this right now. This is sort of your call for where the puck's going.
The big labs should be and need to be focused on really advancing what their AIs can do. And in many ways that's adapting to the world as it is today. But I think we as researchers have an opportunity to think of what the world could be tomorrow to make that more effective. And I think maybe the more important part of that is to figure out our place in that world. And so as we scale systems, as we scale AI, the remaining thing that doesn't scale is us. And I think our ability to communicate, to command these new forces is something we can be studying today in academia.
I'm curious. You said that you're advocating to your students about the advantages of staying, and I'm curious how well is that resonating? Are there other edges or advantages that resonate with a young generation of PhDs?
I think when I look at the next generation of grad students, one thing I see more excitement around is being involved in policy and shaping what that world looks like that they will inherit and how they will adapt the technology and the ideas around it to influence that world.
Yeah. Great. Cool. Okay, we'll call it there. Thanks so much.
Have you done a podcast before?
Yeah, once for a Greek newspaper. I did a podcast.
Okay. On a scale of 1 to 10, how would you rate that podcast interviewer?
No, I think it went really well. I don't know. Eight.
Really well. Eight. Okay. That's the score to beat.
Good. Good.
Giannis, thank you for joining me to chat about ACM, about this conference, about your background and the current state we find ourselves in in data and AI. You're a professor at University of Athens and you're the president of ACM right now. So, the Association for Computing Machinery, that's what ACM stands for. As your sweatshirt says, I talk the talk, I wear the wear. So, oh yeah. No, I need to get one of those. It also says open by design. Tell me more about open by design.
As of January 1st, 2026, the ACM Digital Library is fully openly available to everyone from anywhere without having to pay. And that was something that we started a path in in 2020 and through changing our business model around it because open, yes, it means free at the point of use, but still there's a cost in publishing. So we changed the business model. We have more than 3,000 institutions around the world now that they subscribe not for reading but for publishing. And so now everyone, whether they are rich or poor, in Antarctica or in Mars or in Stanford, they can read all the papers. It's a democratic way for publicly funded research to be openly available to society and of course all the scientists. So that's what we're after.
Yeah. ACM is one of the most foundational institutions in computing and it's got impact and influence across all of kind of the main fields of computing. And it sounds like that was a move that updated and kept ACM at the bleeding edge on a key pillar, openness. Do you envision other foundational shifts in the way ACM organizes as a reaction to how AI is changing the world around us?
Well, in every group, in every pillar within ACM, the discussion is how AI is influencing this. How is AI influencing education and how are we going to teach computer science or how AI is influencing education in general? And how is industry changing with the use of AI? Is AI influencing all the issues of diversity, the haves and the have-nots? So issues like inequalities and fairness are at the middle of what we're looking for. And open as a pillar, democratizing access to the knowledge.
Openness is one big piece of this puzzle. Do you think that the amount of funding that researchers need to get, the amount of pay being so much smaller in academia and open research, whether it's at a lab or not, is a new type of threat and a worrisome threat? Do you have thoughts on that?
Openness, I don't think it brings any threats. Other aspects are more threatening and more worrisome. For example, the reduction of funding that goes into universities and labs. The funding that is on chips or on AI and so on, I think remains high. And I'm not talking about just the US, also I see it elsewhere as well. But for basic research, curiosity-based research, funding seems to be being reduced and that is worrisome because, I mean, Dave Patterson was here, his research, you know, had an idea, it was driven by something, but it was not a particular need for a particular application. I mean, the US academic system was built by some pioneer in thinking that we should invest in basic research and many of the great things, Google for example, came out of basic research and so on and funding by NSF. So it is crucial that this funding is not reduced.
Do you have a call to action for people at the conference or people watching this, what we can do?
Talk to your congressman, talk to everyone. Support NSF and its efforts and not just NSF, NIH, all the funding agents in the US.
I think a call to action that I've heard that echoes yours is that computer scientists now are more powerful than they've ever been. We are building technology that's changing society at rates and levels that are unprecedented. And still our tendency is to speak to our people who know our language, our communities and our fellow computer scientists when the world is starving for people like experts and people with power to speak truth about how the technology works and why the decisions are being made. And all they can do is tune in to Sam Altman and Greg Brockman and Elon Musk entering an Oakland courthouse or something like that. I like you more and more. And so how I think the call to action that I keep hearing and that I hear in your words is for computer scientists to go beyond just discovering and building and start leading and inspiring and speaking beyond the bounds of computer science to the world. And I think kind of like actually activating and actualizing the power they already have and maybe aren't feeling the weight on their shoulders as much as we need to be.
I agree with you completely. We have to step up, get out of our sandbox and our comfort zone. The impact that we have is tremendous. We are responsible to do that.
Great. Okay. Well, thanks for taking the time to chat with us.
It was a great pleasure. Great pleasure.
That's it. Good. Thank you.
What should we talk about?
Well, at NeurIPS, we talk a lot about open versus closed. So, we could do something different this time. You can talk about Arena.
Okay, good. So, Ion Stoica, thank you for joining me.
Thanks for having me, Andy.
Professor at Berkeley, co-founder of Databricks and Anyscale and Alamarina most recently and other things and one of the most prolific startup founders in top research right now. We go back to co-founding Databricks together and you were a co-advisor of mine during PhD work and you have been very active in your most recent company, Arena. Tell me about Arena. What does Arena do and why are you excited about it?
So Arena, first of all, it comes from another Berkeley project, Chatbot Arena, which initially started as having evaluating models for chat applications in the wild. You go there, you ask a question, you get answers from two random, randomized and anonymized large language models and then you can vote which of the answers was better, where both are good or both are bad. And then you take this data, this voting, user voting data and you compute a leaderboard very similar like you compute leaderboard in chess using Elo rating. So that's kind of has been very successful and all that was as a Berkeley project. That was a Berkeley project and then model providers wanted more and more to use this data to only not only to evaluate their models but to improve their models. Right, so that's kind of what has been and this is actually right now the main business model of Arena and help model providers to improve their models. And, you know, it has been a wild ride. You know, I think that one of the favorite stories is about nano banana, how the name came to be. And I don't know this story. Google's vision model, right? So basically Google evaluated and tested their models on Arena and when you test the model which is unreleased you give a code name, right, to be anonymous. And the code name for that model was nano banana. And now the users, you know, kind of heard that, you know, there would be this kind of great model is going to come and they came to Arena and it was a huge jump in the traffic, huge and everyone was talking about nano banana, you know, how great it is and things like that. Now Google owned the list, they came up with a name which is, you know, Gemini kind of name, themed, yeah, marketing people. But then people when they released that, no one knew about what it is, they knew about nano banana. And in the end Google, they decided to use nano banana as a name of the product as well.
That's a good story. It hadn't occurred to me that there's a silver lining to the multi-sided network that Arena uses, which is it's like a structured way for leaks to build up marketing momentum and anticipation in the market for a new capability because the public can see that a capability is coming and make guesses about which of the frontier labs is previewing it.
Actually, it's very funny. Yeah. Because if you think about what value we provide to our users is basically, you know, there are one, you come there and you use real task, you have access to the most powerful models and you compare them and see which models works best for your task. So like you're bringing your real problems as a user. You're using this thing as an alternative even to paying your $20 a month to whatever vendor. You can see, okay, which model is performing best and so forth by comparing them and things like that. And indeed this kind of hits the value and we see that actually a lot of users are coming with their, you know, real problems, right? You know, like close to, you know, 15, 20% math problems and their science, 6% is healthcare and life sciences problems. Wow. And I have a problem in the world, I'm like at a company, I want to figure out which model to use, I can go look at, I can go run it Arena and compare it to a bunch and see which. But And it turns out that one of the main things that attracts users is exactly like you said, you know, it's about trying to get their experience with a model which has not, have not been released yet, right? So that's kind of became one of the unique, a unique value proposition because where, where else you can go and try to play and test with, you know, with the models you never, you know, they are not, they haven't really released yet. Nice. That's great. I had, I hadn't thought of it. It's again, it's like improving the model in terms of quality, reliability. And I think that the natural progression, it's, it's really what excites me is to help, you know, developers and organization to build reliable AI applications, right? There will be agentic application and so forth and then reduce cost, right? These are the two things many people want, almost everyone wants.
Nice. Well, congratulations on being at the forefront of pioneering this model. So, I hope others do it in much larger numbers the same way you have been.
Thank you. Thanks for having me again.
Of course. Okay. Okay. And we have editing control, so no, it's like all free easy conversation. Okay. Hey Graham Neubig, thank you for joining me to chat here at CAIS Conference. Thanks for coming to it. And you are a professor at Carnegie Mellon and also the chief scientist at OpenHands, which is a startup that you're a co-founder of. And you're focused in both your research and your startup work on agentic applications of AI. What is OpenHands and how's the company different than the open research agenda?
So yeah, OpenHands is an open-source toolkit for agentic software development. We are one of the ones that's been around for longer since I guess March 2024. Some of the characteristics of it are that it's kind of language model agnostic, highly extensible. We have an SDK that's easy to use and plug into things. So a lot of people use it for both research because it allows you to expand it and for like incorporating it into products because you can like rip parts out, swap them in, use whatever model you want, that sort of stuff. And then in terms of kind of my longer term research agenda, I'm working on coding agents or agents for software development now because I view software as a way to achieve a lot of goals. So it's not just, you know, writing software for software's sake, but it's writing software to solve problems. And, you know, some of the problems I'm interested in are how to accelerate research, how to do automated science, how to help scientists with the tasks that they're doing in the hard sciences like computational biology or physics or astronomy. Yeah, I think having it be completely programmable would open up a lot of possibilities and kind of like for software engineering, if we can automate away the boring parts, we can let people do the more creative parts. I think it will open up a lot of creativity in the, in whatever space we apply it to as well.
Cool. And how do you feel about the state of open models, open weight models? Do we need better open models? Is it fine if we're all just kind of building a layer of advantage on top of proprietary kind of intelligence on top?
I believe strongly that we need good open models. We need something that's competitive. I think right now we're not, we're not there yet. We have something called the OpenHands Index where we evaluate models on like five different benchmarks and if you look at the diverse evaluations you can see the closed models are kind of still, I wouldn't say head and shoulders above but maybe head above, you know, a lot of the open models. But I think, you know, also just having good general purpose open models is always very welcome. So anybody who is listening who wants... Yeah. And a big thing that we're focusing on with this conference and with Open Frontier is getting the various players in the evals plus data generation and recipes and pre-training and post-training and context optimization agentic frameworks to talk to each other a bunch more and share the way you have with Marin the road maps of the projects. If you're consumers, many people, you are a consumer of almost everything in the else in the stack, including data for, you know, model fine-tuning and context optimization. It feels like we, you know, there's this sense that a Qwen or a Kimi or some of these large extremely highly invested in open model, open weight models are hard to dethrone. But if you pick a narrower lane and you know the seven researchers that are going to want to have their per for their use cases for their research agenda state-of-the-art performance, we could probably get some models to beat any of the frontier even proprietary. If we just kind of stack the deck for our specific use cases first with much smaller weight models and some of these other techniques that and then maybe expand one bowling pin at a time after that. Feels like the way to cheat a little instead of needing to build a model that just benchmaxes and kind of wins at the broad suite of applications is just go straight to a scientist or three scientists that you want to use yours instead of Qwen. And if that's your objective, you start with three, then we can know how to turn three to six and six to 15. So, does that resonate?
Yeah, definitely. Okay, cool. I think you're already doing it.
That's good. Well, great. I think that's it. So, thanks for taking the time to chat.
Okay, Don, thank you for joining me. You're a professor at UC Berkeley, but you've also co-founded companies. You organized a summit that we're going to talk about. You wear many hats. What else did I miss?
And also I'm a co-director of the campuswide center of Berkeley Center for Responsible Decentralized Intelligence as well.
And you are spinning companies out of that. So tell me about that center briefly.
Oh okay. Yes. So this is the Berkeley Center for Responsible Decentralized Intelligence that works at the intersection of responsible innovation, decentralization, open source and intelligence AI. Really we want to push forwards the frontier actually on responsible agentic AI and decentralized. Yes, ultimately I think we are going to have a lot of decentralized agents. I'm going to have my agents, you're going to have your agents and...
That's a grand vision. The trends are not necessarily lining up for that right now though.
I know, I know, I know. So I think this is an important question to figure out what the future is going to be like, who is going to actually control such powerful technologies and how much of individuals gets to control their rights, their agents. I actually talked about this even like I think as early as like more than even like about 10 years ago even with the last wave of deep learning. I mean back then I was already thinking about personal assistance, how we actually can build this personal assistance. I call it almost like a fiduciary is that you want to have personal assistant that's really working on the best interest of individuals instead of trying to maximize revenue of some, you know, platform. And I view that it's particularly even more important as AI capabilities continue to increase and everything we do is going to be intermediated by AI, by agents and so on. So we need some ways to actually really have AI to be on the side of the user.
The willingness of many people to give away privacy rights in order to get the advantages that come with an ad-driven model or just in general like a walled garden approach to the internet has made me a little more cynical. What are the fundamental knobs that we turn or levers that we use to actually make it play out so that agents have a fiduciary responsibility? And by fiduciary I don't mean just financially, right? You also mean just like a moral obligation to defend and do right by person. Exactly, exactly. Do you believe in your heart that there is just the right incentives that people will get it and they will choose an agent that has that versus an agent that seems that has mixed incentives obviously because of like corporate overlords deciding that it needs to make them money?
Yes, that's a very good question. I do think yes, this is actually a critical moment to actually think about these questions. You know, which road, which path we are the world as a society we are going to take. It's a different thing for like, you know, what website you get to view and what message you get to view versus actually, you know, what advice you get and managing your finances actually at some point literally and also your relationship like being a proxy in certain types of relationships how you manage your time. I would say right the stake is orders of magnitude higher and I think hopefully users will care more about this and also we hope that this will help incentivize even the big players.
How do you think about the way that you're influencing the direction of the way of the agent market or the agent landscape especially now bringing in some of the stuff you've been working on recently like the Agent AI Summit and also AgentBeast, is that what...
AgentBeast, like a beat, like BAT, got it. Yeah, so one thing we can, I'm happy to share is we have this new agent evaluation paradigm called the AAA, identified the Agent Assessment for Open, Standardized, Reproducible Agent Evaluation. So basically we make the agent evaluation is open agent and then this way helps with the open standardized agent evaluation. And then we did this AgentBeast competition. So we actually had many thousands of participants that participated globally from over like 130 countries and over 800 universities and over a thousand different companies.
Wow. So you bootstrap the evaluations. It's almost like building a benchmark but you're building an agent that is a benchmark and then phase two go see who wants to contend in that benchmark.
Yes. So then in the end we yielded like hundreds of these judge agents and many hundreds of these subject agents and the judge agents cover very broad domains including, you know, coding agents, web agents, computer use agents and also financial domain, game domain, legal domain like very broad domains.
Sounds amazing. Congratulations.
Yes. Yes. And my dream is that in the future for, you know, like Europe's eClear if all the submissions can be done in this new paradigm that all of them can be used together for us to really, you know, continue to build out.
Sure. Yeah. The multi-agent...
Multi-agent and generalist agent evaluation benchmark that'll be...
Quite the vision. Well, thanks for sharing. I think that wraps us for this slot but I'll catch up with you tomorrow.
Great. Great. Yeah. Thank you. Thanks a lot.
Thanks. Yeah. Thank you.
Okay, everybody ready? All right, John Sad Falcon, thanks for coming to chat. We're going to talk about OpenJarvis, but before we do, let's get you introduced. You're a PhD student at Stanford. You're working on a project called OpenJarvis and tell me a little bit about the project.
Yeah, absolutely. So yeah, OpenJarvis started with this just general sentiment that like so much of what we do with AI today could just be done on device. There's a huge amount of consumer devices available from Apple, Nvidia, AMD. And while the, you know, cloud space and the cloud models are, you know, maturing and growing very rapidly, there's this underutilization of local compute and local models. And so we just wanted to see like how much of the current AI stack could be done on device. And so we proposed OpenJarvis, name is obvious inspiration. And we wanted to basically see like what were the limitations of existing hardware, existing models and just how much you could like push like these queries and these workflows on device and a surprising amount could just be moved on device. I think in a perfect world everybody would want their AI to be in their pocket at the rate at which local models are getting better, at the rate at which local hardware is getting better. We can imagine, you know, a very near future in which you have AGI in your pocket.
Oh, okay. The architecture started there and it's shifted to a little bit more of a generalization where you have these architectural blocks or elements that allow somebody to come in and specify the intelligence and make it keep the agent that's in your pocket pluggable so that until we get kind of mythos level agents actually running on my iPhone, I have the option to do some things locally, some things to do with a proprietary cloud provider. Is that right?
Yeah, I think that's right. Like the way we imagine the AI stack is like local first but like hybrid by design. Like I don't think everything will always be on device, but I think the vast majority of the things that you'll want could be possible on device eventually. Eventually and you'll want it to be on device because it'll be free, it'll be private, it'll be completely owned by you.
With OpenJarvis users, do you see the population being mostly Apple devices or a spread?
I'd say there's definitely a majority of users who are on Apple devices, but we're seeing a lot of Nvidia devices, a lot of AMD devices, even like a lot of like Google like Android or Chromebook being used towards this. And I think that's really exciting because it just means like depending on the hardware available, depending on whatever is, you know, the best open source models available, you can pivot and move the whole AI stack very, yeah.
The project has gotten a lot of adoption in open source so far. Tell me about that, like can you give some stories of use cases that people are applying it to solve?
Yeah, absolutely. So some of the use cases that we've seen that have been like the most popular is just like all of these daily and like mundane tasks that like you just like want to run on device for privacy reasons, for cost reasons, for just speed reasons. People are using it, you know, for daily messaging triage, for deep research queries, even for like different stuff around like code review or around just managing how they interact with, you know, co-workers or how they interact with, you know, their families. And so it's exciting to see all of these use cases that like some of which you envisioned, some of which you didn't. We're really excited to keep building systems...
That can, you know, help that span that gap between local on-device intelligence and cloud intelligence. So we're building all sorts of collaboration techniques for that.
So where are you going to take it from here?
Yeah, I think from here we'd really like just making the whole experience as good as it can be. Not just for people who are tech-savvy or like hobbyists, but people who just want to have their first experience with AI that isn't just up in the cloud, but it's actually AI that's on-device, so that you actually own the model and actually own the whole stack, as opposed to always leasing your intelligence by paying to OpenAI or Google or Anthropic. That's the main direction we want to go, is just making it more and more accessible and more and more enjoyable. Right now we're just focused on building the community and building the research work around it. Down the line, who knows? But we're mostly excited to just keep making it better and better and keep studying a space that we think is going to be really important in coming years.
Well, congratulations on betting on that early. Cool. I think you got to go.
Yeah, I got to head out, but this was great chatting.
Yeah, thanks for taking the time.
Okay, Parthasawa, thanks for stopping by and chatting here at Lounge at Case. You're a second year, just finished your second year in your PhD at Berkeley, and you recently launched Continual Learning Bench. So, let's start with Continual Learning Bench.
Yeah. Wow. Where should I start? I feel like this is just something that was kind of like a very obvious gaping hole in the discourse around continual learning. Maybe taking a step back as to the motivation, like how I even got here. When I started my PhD, I think I was very interested in these problems around sample-efficient learning algorithms or ways to update models after they've been trained, right? Like the way we train language models today, they're trained for months and these offline training processes kind of hidden in a box, and then we take them out of the box and we present them as a frozen topic to the real world. But a lot of people want to adapt language models. They want models that are learning online with their workflows, improving over time. And so I got really interested in this question, like what does it look like to actually get models that are learning in the real world. If you look at a lot of the discourse that's happening around continual learning, it felt like there was a missing piece where people aren't talking about the same thing when they define continual learning. And two, there are not actually any evaluations for continual learning. Everyone is maybe constructing their own synthetic set of emails that if you pick any three papers at random, I think with very high probability the evaluations are completely different. And I think that in the history of machine learning, one of the big things that pushed the field forward is we had these standardized sets of evaluations that people could get together as a committee and like, yeah, this is the definition of a particular capability that we're trying to optimize for. Or maybe not the definition, but it's a good goalpost or signpost along the way that we can all work together to optimize on this, and that just didn't exist for continual learning.
Well, let's shift topics and talk a little bit about your experience after your second year in academia.
