Welcome back, everyone. Hello. How many energy drinks did everyone consume? Four? Anyone consume more than four? Anyone consume five? Okay, nice. All right. Hope you're excited to hear the winners for this hackathon.
Before we keep going, our agenda for today will first have an incredible keynote speaker spread his wisdom about his experiences, then go on to the incredible nine finalists who have worked really hard to be here and, you know, get to this final stage, and finally get to the awards.
So, recapping this weekend, here are some incredible photos that we have snapped. Many more will be released at the end of this event. See if you can spot yourself. Give yourselves all a big round of applause. Well done this weekend.
So, let's recap who was at AI Hackathon. We had over 1,300 hackers join us for this incredible weekend, 400 of which were first-time hackers. So, give them a big round of applause for taking this big leap.
This event also would not have been possible without our almost 180 volunteers, 100 judges, 100 mentors, 30 directors, 20 sponsors, and of course our big bear. And special thank you to our sponsors, all listed up here. This event would not have been possible without their support. So please give them a round of applause. Thank you.
All righty. I'm very excited to introduce our keynote speaker for today, Anastasios, who is the co-founder and CEO of LM Arena. So, give it up for him. And also a UC Berkeley alumni. So, go Bears.
Anastasios Angelopoulos
2:04 ↗
Hello, everyone. You can hear me? Yeah. Good. Well, thanks so much to the organizers for inviting me. It's great to meet all of you. First thing I'd like to say is it smells amazing in here. It's just wonderful, you know, it's the combination of body odor and cologne that you only expect from a hackathon. So wonderful to meet all of you and exciting, you know, to be speaking to this crowd.
So, yeah, first of all, you know, my speech is like the least interesting thing that you guys are going to be seeing today. Congratulations to all of you on an incredible event, and to the organizers for a wonderful hackathon. I know that this is a huge amount of work to put on. So, let's have a hand for them. Yeah, really incredible work.
So, getting into it, let me tell you a little bit about me before we go into the content of the talk. So, I'm a bear. Go Bears. I was born in Berkeley, then I moved to LA later, grew up there, and then came back. Came back for my PhD in machine learning. I fell in love in Berkeley. My future wife, my fiancée, sitting in the audience right here. We met as part of the PhD program. And I started my company as a, you know, quintessential Berkeley company with a group of students and collaborators and professors here. So Berkeley has given me a lot. It's an incredible place. I have a lot of love for it. And I'm excited to be here speaking.
So now let me tell you a little bit about LMArena, which is the company that I've been working on, you know, as a research project through Berkeley for the past couple of years, and now newly as a startup.
So I'll show it to you via demo. If you go to lmarena.ai, this is what you're going to see. It's just a chat interface. I'm sure you guys are familiar with these, given it's the AI hackathon, right? And if you type something, you can say, "Hey, what are some good ideas for an AI hackathon? Keep it concise and give me five." We'll see if any of them parallel things that you guys came up with. Okay, so it says an accessibility tool, you know, a medical symptom journal, a code review, and then I'll say, "Which one would win the real Berkeley AI hackathon?"
Okay, we'll see what happens here. It's thinking and it says, okay, the AI-powered accessibility tool. So, you'll see these are two anonymous LLMs sampled from a pool, Assistant A and Assistant B. And then for Assistant B, which I'm going to choose because it's a little more concise and I like the mental health tool, that was Claude 4. Now, there's three valuable pieces of data there that we use to construct a leaderboard based on this. There's the prompt, there's the two responses, and then there's the actual human vote. And you'll see what we do is we use regressions like Bradley-Terry regressions and so on to turn this into a leaderboard. The number one model right now is Gemini 2.5 Pro. Then you get into o3, the last version of Pro. The number six model is Claude Opus at 1419. The score, higher is better. It's basically a pair. It's like a reward model if you guys use those. And then there's also different categories. There's creative writing, there's coding, and so on.
And then we also run certain, like, controlled regressions on it. So if you want to understand, hey, which model's best for me and I don't care about style, you can remove style control. Style control is a filter that we put on to say, can we disentangle style and substance and only show the style? And you'll see it has a big effect on the leaderboard. Like when you take away the effect of style, Claude goes up quite a bit.
Now, it's not just text that we do. It's also image generation, image editing, and other modalities. I'll do a fun one here. So, I'm going to go on the internet and find a picture of the Chancellor Lion. I'm going to put it in there and say, "Put his tongue out." And we'll see what happens. We'll see what it generates. It's thinking here. I've sped it up.
Oh, nice. Okay. So, on the left, it's wonderful. It's beautiful. Scooby-Doo tongue. Okay. Or taco tongue. Taco tongue is also good. But I'm going to choose the one on the left. Okay. And then the identities are again revealed. So you can see it's like a blind forced-choice testing platform for the purpose of evaluating AIs.
Now another thing, because you guys are hackers, you guys must be familiar with tools like Replit, things that are going to, you know, allow you to build an app in like, you know, however many seconds instead of coding it yourself. Now this is a Replit arena. We call it WebDev Arena that we built here, and it was actually built by, I think, an alum of this program, Arian Varizadeh. They're saying, yep, yep. And so it's like it'll pop up two, and then you can choose between them. I said, "Make a hackathon submission." This one's a little weird. It's not exactly a submission. It's more like a website. Okay. It's got the prizes. Okay. I can't find my talk in there, but that's all right.
Alex Johnson, CTO of DeepMind. I don't know who that is. And the buttons don't exactly work. They'll take you back to the top, but it's okay. At least it compiled. So, I'll click right is better. That's where we're at. So, if this tells you anything, your jobs at the moment are safe because neither of these websites or submissions is really good. So, can an AI win Berkeley Hackathon? Probably not.
Now, the platform that I just showed you, you know, it's a bunch of AI evaluations across different modalities: text, image, image generation, web development. We also have red teaming. We have in-context coding like a Copilot Arena. And it's been run, it started as an open-source project at Berkeley. It started just as a bunch of students working together, and after two years it's grown into a pretty large platform. It's got more than a million monthly users. Over 150 million conversations have happened on the platform. About one to two million more conversations every week happen, and it's a trusted leaderboard for basically all major model providers. So we work with all major model providers to evaluate their models, sometimes pre-release, sometimes at the moment of release. That includes the OpenAIs and Googles, the Anthropics, the Mistrals, the Metas, the XAIs of the world. And you might have seen LMArena online. It's kind of a, you know, we're active on Twitter and a lot of people are, you know, generating conversation about it.
Now, the question is, how did we get here? You know, how did LMArena become what it is? And how did it grow out of the open-source project that it was, a bunch of PhD students working together, into the platform that it is today? And, you know, where is it going? And the first-order answer to that question is it started in a hackathon.
Okay, so what happened is that some students at Berkeley and Berkeley SkyLab were working on this open-source chatbot, Vicuna, that was one of the first open-source chat-tuned LLMs. And in the process of that, they realized, like, hey, how are we going to prove that this LLM is actually any good? How are we going to prove that it's better than, you know, other offerings that are out there, or even non-chat-tuned things? And what they realized is that a lot of it is subjective. The LLM is interacting with the human, and so you need human evaluations in order to tell what's happening. And so that's the birthplace of LMArena. And you can see that's the first tweet about it on April 24th, 2023. It says, "Introducing Chatbot Arena. You know, we've released Vicuna, and there's all these other LLMs. Come try them on one interface."
And, you know, I initially approached the project, you know, I was doing machine learning, I was doing like statistical theory and stuff like that. And so when I saw this project, I was like, hey, what a cool opportunity to like develop methodology, new statistical methods in order to try and address the problem of evaluation in the world of LLMs. You know, it was exciting to me, and also I liked the people. And it was like, hey, I have a little bit of time on my hands. It was referred through my PhD advisor. Why not spend some time doing this like wacky side project? And so I first met with my co-founder Wei-Lin and Ion Stoica, and we filled up three whiteboards filled with, you know, just math problems, statistics problems that, you know, we weren't necessarily developing new things, but how are we going to take all these votes and turn them into a regression? How are we going to construct the leaderboard? How are we going to form the confidence intervals? And, you know, two years since now, I spend like all my time running this platform along with a wonderful team.
Now, a hackathon is not enough. After the hackathon, the project went and, you know, there weren't as many votes. There was some initial interest, and then look at the plot. It's like you start in May, you go to July and September, and the curve is down, guys. That's not like, this is not the hockey stick that we were, you know, necessarily anticipating. But credit to Wei-Lin, you know, he had enormous consistency in making sure that this platform continued developing. At some point in the middle of the summer, he said, "Hey, I want this to be, you know, I want to work on this full-time." And he started adding models, improving the platform, tweeting about it, increasing the distribution, you know, every day little improvements. And I, you know, I joined him around that time too.
And then, you know, that led to a little bit of growth, growth, growth, growth, growth. Then we were like, hey, let's take it to the next level. And we started living together. So we moved in. This is Wei-Lin's birthday like last year. And, you know, along the way, it wasn't just us. It was a huge group of Berkeley PhD students and undergrads and master's students that were working on this platform as it continues to grow. And, you know, you can see the spikes there. These often have to do with competition. So we were at the right place, right time. As, you know, maybe it was just ChatGPT at the beginning, but once Meta got in the game and Anthropic got in the game and so on and so forth, interest in comparisons and evaluations skyrocketed. So that's the way, you know, the curve grew and grew and grew and grew.
And we also started pre-release testing models. So our community loves seeing new models before anyone else, and so they come onto our platform with code names. I don't know if you remember, like, you know, "I'm a little GPT" and that whole debacle, but that happened on our platform and it was amazing. That's the other spike in traffic that you see there. And since last year, we've, you know, continued to grow it and improve it.
Now, another question that you might have is why did LMArena even resonate? You know, I showed you the polished version of the platform, but the early version of the platform that we built was a research prototype. It was a Gradio app. And, you know, credit to Gradio. Gradio, I'm sure a lot of you guys are way better at interface design than we are. And the Gradio app that we had allowed us, from a research perspective, to take the platform from just an idea we had in our head directly into code and scale to like millions of users. And then only recently have we transitioned into like Next.js and all these frameworks that you guys are probably using for your hackathon.
But why did it resonate? Why did, you know, despite the fact that we're not exactly the world's best consumer product people, why did this turn into, you know, such a phenomenon? And I think that for that you need to zoom out. We are in a new age of software. Software is changing. Unit tests are not enough for us to evaluate our software anymore because the way that our software works, you have humans inputting these super subjective prompts, asking you to do things that maybe don't even make sense. They're not precisely defined. And so ultimately, at the end of the day, because they're interacting with humans, you need humans at the other end evaluating them. The space of things that you're doing is too broad. It can't be just ImageNet-like classify or MMLU, I'm going to look at multiple-choice question answering. You need to have something that goes beyond and captures the diversity of uses, you know, that LLMs are being subjected to on a day-to-day basis.
So, this leads us into LMArena. You know, with static benchmarks, the way they work is you look at a vertical, I'm going to collect a dataset, and then like I'm going to test the model on the test that I come up with. But the issue with, there's just differences. Okay, these static benchmarks are useful for evaluating these narrow verticals, but at the end of the day, live benchmarks may also be useful. And in fact, I would say that these are the benchmarks of the future, not just ours, but many.
So what are the ways that they differ? Static benchmarks, for example, lend themselves to overfitting. If you create one test and you test the LLM over and over and over again on the same test, what ends up happening is it memorizes the answers, and then the test doesn't become a good, you know, a good judge of whether or not the LLM is good at the underlying task distribution. Live benchmarks, though, like LMArena, are different. We have millions of people coming to the platform giving fresh prompts all the time. And because the data is fresh, that guards against this overfitting because you're constantly getting new questions from the distribution of test questions that you're supposed to be good at.
Now, another issue with static benchmarks that we're starting to see is contamination risk. You know, as soon as you create a static benchmark, you publish it on the internet, chances are that because the LLMs are trained on the whole internet, that somebody's seeing it and training on it. But with live benchmarks that are created via, you know, people like us going to the website and asking whatever questions are on our mind, there's no contamination based on those, right? Because the questions are always new.
Now, one benefit of static benchmarks is that they're targeted, whereas live benchmarks are organic, right? You're getting whatever people want to ask. It might be something that you're not even interested in, or it might be something that you didn't know that you were interested in, or that somebody else in the world is interested in, someone else is not. Static benchmarks also lend themselves well to rubrics, right? People don't come to LMArena to say, "I want to, you know, give data." People are coming to LMArena because they love using the best AI and comparing them and looking at the leaderboard and getting multiple opinions from all the best AIs in the world. So natural usage data often does not lend itself well towards asking people to do things they don't already automatically want to do.
And static benchmarks, you know, often they're constructed via paid annotators, whether that's a graduate student that's sitting for days and days labeling images of dogs and cats, or that's somebody that you actually hire to say, hey, you know, you're a radiologist, to come in and label images. Live benchmarks are different, though. They're just like random people on the internet.
So, in other words, like this is where we are with the platform today. But there's a number of challenges that we need to resolve for the future. How are we going to take these features of live benchmarks, the fact that they're organic, the fact that they're determined by, you know, just random people on the internet doing whatever they want to do, and turn those into features? And I'm going to dig in a little bit on one of them, which is targeted evaluations. How can we take all this organic data that we have and turn it actually, you know, basically invert the problem of evaluation, turn it into a problem of...
I'm going to go target a task and I'm going to collect, you know, spend a lot of money to collect a dataset for it. Instead of doing that, maybe what I want to do is say, hey, go use the LLMs however you want. I'm going to generate a bunch of data from this organic usage and I'm going to analyze it. I'm going to basically mine it for insights as to which LLMs are good at different parts of the space. And then you start getting into the questions that are really interesting to people like us, developers and scientists, which is how do we create very personalized evaluations that are telling you which model is best for you? Which model is best for your use case?
For people like me, how are we supposed to answer this question? And it's a tough question. You know, one way that you can ask the question would be to say, you give me a prompt. You give me, "I want my hackathon project about, you know, a personal assistant that's going to help me schedule things in my calendar," and then I'm going to tell you which model is best exactly for that use case. But the problem is you might be the first person doing what you're doing. That might be the only time I've ever seen that prompt ever. And so the standard paradigm of regression won't work because regressions don't work on data points that I've never seen before, right? You need to like run empirical risk minimization. You need to calculate, you know, you need to have your loss function, you need to minimize, and so on. That's not the way that this is going to work.
So, it seems like a hard problem, but it turns out that there's a way to solve it. And this is where it's useful to have, you know, Berkeley people on the team that are great scientists, because what our team has done is we've developed a way of training a large language model to take prompts as input and output leaderboards, even if those prompts have never been seen before. So the idea basically is parameterizing the Bradley-Terry coefficients, these regression coefficients, as a language model. Can we train language models to intrinsically output leaderboards by nature? And the way that it works is basically a creative way of supervising via your loss function.
Okay, so the only thing we observe, remember, is pairwise preference data from Chatbot Arena, left-right votes. Is model A better or is model B? And what we end up doing is we say, hey, let's call the prompt P. Let's put it through an LLM. You'll get theta of P. Theta representing the leaderboard. It's a vector of length M, where M is the number of models in your leaderboard. And then what we're going to do is we're going to supervise it via binary cross-entropy in order to try to predict the votes that we see. So if theta of P is good, this model, which is like sigma of theta of P transpose x, that model is going to be good at predicting whether or not you voted left or right. And if the leaderboard is bad, it won't be good.
And so the loss function is going to tell you that. Loss function is high when sigmoid of blah blah blah does not agree with y, right? That's how the loss function works. And it turns out that that loss function is differentiable, so you can apply the standard, you know, toolkit of deep learning in order to make sure that loss goes down and you're learning leaderboards that are good at predicting. Okay, so what we end up doing is we actually just train a language model end-to-end, supervise it in order to do this task, and you train it with backprop just as normal, and theta of P then becomes a prompt-specific leaderboard.
Now, what's interesting about this? Why did I just tell you this technical story? It's because as we're developing these AI apps, there's a lot of interesting and important research questions to do along the way that are really going to change the way that we build our software. So, one of the insights that we have while doing this is to say, hey, by doing this, by turning evaluation into a problem of learning instead of a problem of averaging, as you would normally think about it, we benefit from scaling laws. We can have scaling laws for evaluations. And that's exactly what you see.
If you train, you know, a half-billion-parameter model to do this leaderboard prediction task, it'll do well. It'll do better than the standard regression, but not nearly as well as a 7-billion-parameter model, and so on and so on. And the more data you get, the better it gets. So, we're benefiting from ever, ever, ever more granular evaluations that are better at telling you which model is best for you and your app via this, like, new scientific methodology that we developed open-source at Berkeley.
So, last thing I'll say, you know, takeaways from this is this was just a hackathon project at first, and now it's become a real company, right? We have, you know, more than a dozen employees. We've raised money. More money, more problems, of course, but things are off to the races. Now we have to demonstrate that this platform is useful. We have our million users, our community that we're digging into.
The first thing I would say is you have to stay consistent. If you looked at the plot I showed you earlier, the platform almost died, and it would have died if not for, you know, Wei-Lin and others supporting it along the way. So if you don't, if you come out of this and you build your hackathon project and you say, "Hey, I'm not going to maintain this," it will die, and then nothing will come of it. But if you do maintain it, who knows? It might go dip down and then come back up, just like ours did.
The second thing I'll say is just find the best team. You know, one of the things that we're really proud of is that our team is built of, like, pretty low-ego people that are looking for people better than them at every task. Wei-Lin and I know that we're not the best consumer product people. We know that we're not necessarily the best at UI. We know that we're not the best at backend, but we need to find people that are better than us at this. And in fact, you want to find other people that continue to create this low-ego environment where you guys are building a team serving a greater mission than yourself, right? So, go take your hackathon project, go find the best people in the room, work together on something that's going to become greater than the project that you built over the last couple of days.
And the last thing I'd say is keep adapting. The world's going to teach you how to change, right? You have to listen to the feedback the universe is giving you in order to continue improving the application that you guys are building. So, those are my three takeaways from my project, and, you know, your mileage may vary. It's not like we've had massive success yet, but we'll see where it goes. And looking forward to hearing from any of you on how your projects, you know, go to the moon.
Appreciate it.
Thank you so much. Alrighty.
And now for the moment you all have been waiting for. We have our three grand prize tracks: the creativity track, the developer tools and productivity track, and the social impact track coming up. We'll have nine finalists pitching to you all to a panel of judges who will be seated over here. And they each stand a chance to win 5K in each of these tracks, which totals up to 15K. All righty.
Cool. So, as I mentioned, the pitch process is going to be nine finalists coming up on stage. They'll pitch for about three minutes, and then they'll have three minutes to talk to our judges. And speaking of our judges, please meet your grand judges if we can have them come out. We have Carl Choi from Alumni Ventures. Let's give them a big round of applause. James Alhorn from Lightspeed, Mark Astor, the dean from the College of Engineering, David Bloom from the House Fund, Liz Wessel from First Round Capital, and lastly but not least, Alex Demarcus from UC Berkeley Inc.
Awesome. And now I believe it is time to introduce our first team. As their slides come on, right up. Our first team is Chip Chat. Let's give it up for Chip Chat.
Hi everyone. Thank you for joining us today. My name is Yashna.
And we are Chip Chat.
So, as you know, hardware serves as the backbone for nearly everything in society today: medicine, telecommunications, memory, energy, and it's only going to be playing a more important role. I've been in a microchip lab for the last two years at Berkeley, and I've realized that PhD students have spent six, seven years iterating over the same microchips again and again. And sorry about that. And students in 61C have taken months to build that intuitive circuit design knowledge, and companies like Meta have spent over 46 billion dollars on the metaverse and over five years to develop chip technology for glasses like that.
Now let's do it in 10 seconds. Let's take the example of a 4-bit adder circuit. We begin that by inputting the circuit we want to build. Can you play the video, please? We begin by inputting the circuit we want to build and wait for Gemini to produce a result. First, Gemini produces a Verilog file summarizing the circuit logic.
Using a custom pipeline, we process the Verilog file to produce an interactive logic circuit. In this 4-bit adder circuit, you can select the inputs, simulate the circuit, and verify the output. 2 plus 2 is indeed 4.
Next, we synthesize our Verilog files to include information about the logic gates.
We continue our chip generation process by creating a DEF file to include information about the position of individual circuit elements. Lastly, we produce a GDS2 file that represents the circuits in 2D and 3D, planning the real estate of the circuit. This GDS2 file is now ready for fabrication.
You can play the video. So within Chip Chat, we have a circuit assistant where you can chat for educational purposes to understand the circuit, and it has the multimodal agent file where you can upload the schematic drawing and it will generate from very low code to the physical chip design. So you can go from this to this.
So to summarize, our proprietary workflow starts by using Letta with Gemini as its model to generate the Verilog code, and then we use a bunch of libraries to generate the GDS file, the interactive circuit, and the physical layout file.
Today, going from circuit idea to GDS2 file takes about two or more years. We need to cut that time to two months. So we use AI agents for that. There's higher demand but less supply, as there's a massive shortage of chip designers and hardware engineers. We don't want to replace them, but we want to help them create faster tools to focus more on the architecture of the GDS2 file ready for fabrication. We strive to take Chip Chat to the next level by incorporating verification, higher compute, and higher processing to bring this to a full-scale end-to-end CPU and GPU development pipeline for engineers everywhere. Currently, they spend over 50% of their time on verification and development, and with Chip Chat, we see that students, engineers, and everyone interested in hardware can enter the gateway and turn their ideation into fabrication. Thank you.
Can you talk about how you think about the monetization of this business? Who would you sell to, and where would budgets come from?
So currently, I think it's really important to make this tool open source, particularly for students. Right now, with the state of our current models, they don't necessarily support the ability to make large CPU or GPU files. So we might focus on education for now, and later on branch to GPU, CPU, and potentially machine learning hardware accelerators.
I can add on to that as well, really quick. All of that, in order to build the full-scale end-to-end pipeline, requires money and data. So that comes with time, and as soon as we have that, we would love to take this further and help build that. Yeah.
Can you tell us a little bit about the stack that you used? What tools you used in the technical AI stack?
Yeah. So to generate the initial Verilog code, we used Letta with Gemini's model. And the reason for this is that Letta makes it really easy to incorporate things like chain-of-thought reasoning, first of all, to help it tackle, like, more complex problems, kind of like how Cursor will think and reason before actually generating a solution. It can do something similar and split into smaller steps to generate larger circuits, and also makes it easy to connect it to external tools. So in the future, if you want to connect to documentation, that's something we could do. And the rest of our stack, we used React for our front end. We used Auto, you can see on that workflow slide, to kind of go through that pipeline. And yeah.
Do you see this working alongside or replacing some of the other kind of more classic EDA tools, design tools that are used today?
I think it's interesting to put this in parallel to Cursor. If you were to use Cursor, you need to have some level of software engineering knowledge. If you were to use something like this, you need to have a fundamental circuit design knowledge as well to use it effectively. Of course, you could fiddle around and make something, but you wouldn't be able to use it to its full potential.
So, thank you. Really appreciate it.
Thank you so much. And next up, our team, Athena. Let's give it up for Athena.
Hi everyone, my name is Lucas, and on behalf of my team, I am thrilled to share with you guys Athena, our multimodal AI tutor. Let's dive in.
First, I want to show you some headlines. AI has promised great strides in productivity, yet at the same time, education outcomes are decreasing across the nation. So, how do we reconcile this? Well, it's not all bad. AI promises huge productivity boosts for education as well. In fact, a Harvard study found that students using AI tutors doubled their learning outcomes while requiring shorter sessions. So, why isn't that currently happening? Well, let's dive into the current solutions and maybe where they're going wrong. If we look at providers like OpenAI and Anthropic, they've started exploring education spaces, but it's simple chatbot interfaces. It's not quite immersive. Even more tailored solutions like NotebookLM with some multimodal capabilities just don't quite hit the target yet. We think that we need a big breakthrough to take AI education to everyone, and that's Athena.
How can we utilize AI learning capabilities to reimagine human-computer interaction when it comes to learning with Athena? So let's dive in. Athena is a super customizable, interactive, modular canvas powered by our voice assistant. The first step is uploading your context. You go ahead and upload lectures, you upload papers, you upload anything you need to provide context. And after that, our voice assistant takes over. And I will show you our demo now.
Can we have some volume?
Okay. Well, you'll just have to believe us that we're actually talking to the voice agent. I promise we are. It's pretty quick, and in the background, it is connected to many tool calls that allow it to generate different types of content. Like the first thing it's going to generate here is a study guide. Once you've gone over that study guide and maybe have some grasp over the knowledge, you can ask it to generate a quiz. And in fact, that's what Daniel is asking. And in addition to that, we have so many more tools that it's connected to. We have the ability to generate quizzes, interactive artifacts, we can generate visualizations, diagrams, we can generate animations. So, we have video capabilities, voice capabilities, text capabilities.
But that's really only the start. A really cool feature we have is the desktop view, which connects your, let's say you're working on your math problem set on a piece of paper or a tablet, it connects that directly into Athena via a video feed and can provide live feedback and correction.
Here's a basic overview of our architecture. We make extensive use of the Vapi platform to power a voice assistant, equipping it with the tool calls we need to generate context and manage the layout. Furthermore, those agents are powered by Gemini's native voice capabilities. And Gemini also powers the rest of the stack, including writing code to generate a lot of those visuals.
All right, so this is the team. We're all from Davis: Daniel, Ain, Neil, it's his birthday, so he's not here, and myself. We met at Hack Night, this cool event, and we've been friends ever since. But let me tell you a little bit more about the vision of Athena.
Our visions are grand, but rooted in real feedback. And in fact, already by talking to judges, we found use cases that extend well past education. Our judges expressed interest in using this for their enterprise applications, like onboarding, anything where you have to collect knowledge and provide immersive, personalizable experiences. And let me tell you a little bit more about personalization. Using metrics, by showing our users different modalities, we can learn which modalities are uniquely tailored to each user. And without relying on expensive methods like fine-tuning or things like that, we can nevertheless provide gradually...
increasing personalized, customized, tailored experiences. We believe that student and everyone's engagement is critical to boosting productivity and we believe that Athena will provide just that. Thank you.
Yeah. What sort of preference data can you collect to improve the application experience over time?
Yeah, so there's a few things we can point to. One is just time spent engaged with different modalities. Since this is a really natural discussion with an agent, it's super easy to seamlessly switch between different interaction types and we can track which interactions users spend a longer time with. But it really doesn't stop there. Another important feature is the use of quizzes to verify that this learning is actually working. Right? So you can imagine a system where we, you know, teach with different methods, collect the information in the quizzes, see how well the students are doing and use that to tailor the system further.
Just like adding on to that, like you mentioned, like the quiz accuracy score, we can see which forms of learning tended to actually create higher accuracy on those quizzes, creating benchmarks and like kind of like optimizing what learning modalities would be best suited for like the general public but also just for the individual user. And from there we can start tailoring user experiences from there and collect a lot of data on learning habits, even helping schools potentially.
Good job on the presentation. There are a lot of noise in this space currently with the AI and education. So how are you thinking about going distribution and making the first adoption for your initial customers?
Yeah, I think that's a great question and I think the key is playing to our strengths, right? We're college students. We attend UC Davis. We're often in lecture halls that are filled with 200, even 300 students. And in those spaces, we see firsthand the sort of challenges that students face when it comes to education these days. So I think our perspective as students will mean that it makes sense for us to really focus on that demographic first. In terms of distribution, our teammate Daniel here, I don't know if you guys know him. He's a content creator actually. So that's definitely going to help us out. I think we've really seen the rise of these sort of viral distribution strategies and I think that's something that we want to tap into.
Yeah. And just adding on to it, like you mentioned, like we are like the target audience. So we know what the users are looking for, what our audience is looking for, and so we can tailor our own go-to-market and like our own like social media marketing or like ambassador programs on campuses. Like we know our market. So I think this will help us like know the approach for how to our go-to-market strategy better.
One more thing I wanted to add was the sky's the limit when it comes to the domain. We can work with all sorts of ages. We can work with children in elementary school. We can expand it so that it's usable in enterprise locations. Like we spoke to some judges. So yeah, I just want to say this like the domain is just so big for this kind of product.
Can I ask how can your tool be used to help the instructors of the courses too?
That's a great question and honestly I don't think it's something that we've given a lot of thought to, but let me take you back to those metrics. I think as we start collecting more data about the type of interactions that students really engage with, we can relay that very same data back to instructors to help them make their own lectures more engaging. And I think we're already seeing instructors use a lot of like supplemental materials that they connect to their courses. And I think this we can provide basically a constant stream of data between instructor and students to boost everyone's experience.
Thank you, Athena. Thank you very much.
Hi guys. Hi judges. So this is T mockmouse and we present for you mockmouse, which is Cursor for QA. So I just have an honest question. So in today's hackathon, how many of you guys just did like accept, accept, accept, accept and then just hoped let's hope it works and it worked. But what if it didn't? Like what if there was some scenarios where it was breaking and stuff like that. So for that we present Cursor for quality analysis testing.
So what is the problem? Like why to build an automated QA tool? So around 60 to 70% of QA time is spent by manual QAs in all the startups, in all mid-size teams and 30% higher error rate due to people doing it manually. So obviously human interactions, right? We are going to, you know, like miss out some things. Those things will turn out to grow and, you know, become some like big problem for us in the future. And third was 50% slower release cycles. So think of it like you build pretty fast, like shipping fast is a thing in SF I hear. So you build pretty fast but then it breaks but, you know, let's ship it and let's make sure let's not break it. So we will make sure that all the release cycles will be pretty fast if we set up automated QAs.
So like we have Cursor like for code, we have like Cursor for sales marketing, then why not for the most important part having it work properly for we should do all the sales going ahead. So what do we offer? So we have AI powered test generation. So in simple words you can write your tests in simple plain English. So you write your test in simple plain English and our intelligent orchestrator will break those down into proper agentic steps which will go down like go to browser use and then browser will implement all the test for you and then provide you with proper results. We also have performance analytics like when a human or a QA does antitesting like that human being cannot really give you performance analytics and then LCP rankings and you know how does performance work, what was the load time, etc. etc. We also provide you with those like those kind of analysis. On top of that we have realtime monitoring so when the test is running you can also get a recording of that session of that and also live streaming of like seeing what your agent is actually dealing with and then full application context. So obviously we are not building like SPAs, right? We are building something really big. It has to carry a lot of context and it can't just be clicking off the browser and going ahead. So what we have is application context which will carry the complete portable memory of the application all throughout all the tests and do comprehensive analysis before going ahead and running all these tests.
So how did we build this? So the major part in this product was played by browser use. So we solved for social browser use and used it for our main interaction with the web applications. Then Next.js, Express for front end and back end. We wrote complete infrastructure in Python because that was the most well suited thing to go with browser use. Supabase for our database and then we used Claude for all of our agent orchestrations, deductions and also with browser use.
Meet our team. So myself, I'm J Salony, T and Kushi. Excellent team to build with. We really built it in past 24 hours and we were able to deliver it to you guys.
So yeah, this was mockmouse. This was the demo and if you guys really like the product and are interested about it, you can check the demo live on this YouTube link and if you guys are more interested in discussing how did we pull it through or if you want to use it then you can schedule a call with us on this link. Thank you so much.
Could you describe how tests are composed and executed?
So to give like an overview of this, okay, you write the test like okay test flow or test login flow, test search functionality. Now this line is taken, this plain simple prompt is taken along with the entire context of the application. The context is driven through various ways like there are screenshots taken, you know, browser use takes screenshots and analyzes them with like GPT or Claude. In this case we're using Claude. And then along with all of this context and the prompt that we passed, it comes up with a proper orchestrated steps, 7 to 8 or 8 to 10 steps of how to actually go around the browser and click that particular things and see if it's working. Keep in mind around like the entire memory of this thing. Okay. Like in some places this button would work in this way but for this application context we want to work it in this way. It also holds that for us.
Is there a specific type of user you guys have built this for maybe or type of team? Maybe smaller QA teams or people without QA teams?
So we would really like to focus on your all sort of teams going ahead in the future but obviously like we don't want to like start rush and go fast. So starting we want to focus on fast-moving teams because you know I have seen a lot of teams, a lot of SF teams or you know they believe in building fast and while they build fast they somewhere miss out on the fact of building perfectly. So why not provide them with a tool because let them build fast but we provide them with a tool as well.
There's a bunch of automated QA products like Spur is one that's pretty similar to this. I'm curious when you looked at the competitive landscape, what did they miss?
So I think the most important thing is like a conversational flow. So what we are trying to offer is not like okay, like you put a test, you get results and it's done. What we want to offer is like a natural agentic conversation. We really don't want to make it feel like you're talking to a tool. We want to make it feel like you're literally talking to a person who is on your team. So that brings me actually to my future scope which is like we have three things in our future scope. First is Slack integration of this. So you could literally tag @mockmouse and then ask it like, like recently Cursor launched there @cursor to you know directly run background from Cursor like Slack. So we'll do the similar thing like @mockmouse run this test for me and it will give you all the test results. Secondly, it was uptime checker. This is something a lot of keyword testing apps don't have according to me. It's like every 12 hours, every 24 hours. A lot of big businesses miss out on millions of dollars of revenue if they are down for even 10 minutes. So how about like we have an uptime checker every 12 hours like any custom duration, any custom period the user wants to. And the third thing is running this on all kind of CI/CD and a push or anything. So like giving every layer of flexibility starting from PM on the team to the developer on the team.
Thank you, Mock Mouse. Give them a round of applause.
Next up we'll have the three projects for the creativity track. For these tracks we'll have the judges present three questions after their pitch. Thank you. Next up is Doazy.
Hello everybody.
Raise your hand if you like music. Yeah, let's go. Yeah. All right. We are Dozy, your AI assistant for music production.
So if for anybody who doesn't know a DAW, a DAW is a digital audio workstation. It's how beginners to professional creators produce music. We noticed in existing DAWs that they have too many complex menus. Plugins are hard to navigate especially for beginners. There's a very steep learning curve and we aim to eliminate that with our product here. So there are existing AI music solutions. However, we noticed they're way too automated and in my opinion they take away the fun, the ability to edit and tweak these tracks that are autonomously generating. So introducing Dozy, this is what we came up with. It's a middle ground. We use AI to assist, not to automate. We're aiming to have efficient communication in real time while you're working on your music. So, here's a quick two-minute demo where we cover most of our main functionality.
Our integrated AI assistant within our music production software. We offer multimodal communication with our Dozy agent. For example, let's say you've been hearing a melody in your head the whole day, but you don't have the technical experience to get it into the production software. You can simply hit record.
You can see that Dozy automatically populates all of these converted MIDI tracks directly into our music production software.
Hey, can you generate me an 80s jazz fusion track with a trumpet solo, please?
Generated music track beat.wave. Parameters used genre jazz.
This live generation is possible through our open-source API Beethoven AI which takes OpenAI generated specific prompts and turns them into live AI music generation. Next, can you edit that same track and give it both reverb and delay as FX?
Perfect. I've successfully added both reverb and delay effects to your Jazz Fusion track.
Dozy doesn't replace your creativity for jazz, especially for that trumpet song. Thanks. I decided I actually don't like the delay. Can you remove the delay, please? Now, your Jazz Fusion track only has the reverb effect remaining, which will give it that nice reverberant space without the delay interfering with the natural flow of the trumpet.
Can you go through every single knob and adjustments in the reverb menu and tell me what each of them do?
Complete reverb parameter guide. Every single control. Here's what every single knob and adjustment in your reverb on the Beethoven Jazz Energetic track does.
Here are a few quick and effective techniques you can try to spice up your jazz track. One, add chord extensions.
Dozy doesn't replace your creativity, it amplifies it. Please feel free to ask any questions about how it's made.
Great demo. I guess that elephant in the room, how are you thinking about IP and copyrights?
IP and copyrights of the generated music.
Well, we are, well the user is generating the music. We aren't copying it from like anywhere.
Right? But you're leveraging Claude or OpenAI. So some data and...
Oh yeah, we are using OpenZeppelin and Claude. So we haven't really thought about that yet. But this could be, we can use an open source model, Llama, it can perform the same way because for the music we're using a separate open source model that could be used commercially.
Got it. So user is going to own the music generated.
Yes. Because the user is generating the music. It just like, it just like removes the barrier to talking with that complicated interface. That's the main goal.
Yeah, so we implemented conversational memory. So as you continue to use this software, it's going to remember your preferences. It's going to remember what you did generate, what you liked, what effects you put onto it. And we store a lot of that locally. In the future, right, it would be on a server or in the cloud or however this product would work. But right now, we have it locally and that allows for future creations. It's pulling a large mesh of drum lines, different instruments that it knows you enjoyed. So I don't think we'd ever run into issues with copyright since most of it is largely being generated on the fly.
So I guess related question which you say here OpenAI controls beta. So can you explain a little bit, is your app prompting OpenAI which controls another generative model which actually produces the audio?
Yeah. So since our main Dozy integrated agent here is controlled with the conversational memory and by the user, we'd like to keep that by itself so that it has all that memory and then we have that internally in our backend creating a list of very specific instructions, everything from BPM, key, what instruments to use, just from the like one sentence 'can you generate me a jazz track' it takes all that information and makes a custom prompt that is designed to feed into Beethoven AI.
So Beethoven AI. So you write prompts to Beethoven AI and Beethoven AI gives you wave like action.
Yeah, we do, you know, we're simple. We don't know what we're talking about. Sometimes we give it a simple prompt. Our agents will turn it into a complicated prompt which is fed into Beethoven AI which is then integrated to upload it directly to the Dozy software.
And then you generate multiple tracks with Beethoven AI that are overlapped or it's always a single track?
Yeah, they overlap or it depends. In this case, I asked for a drum line with a trumpet solo. So like as you heard, it added drums and a trumpet. It does overlay them in one stereo wave file here. But we can, we don't have proof of it, I guess, but we have generated like just, you know, consistent bass notes and then a guitar and you can have layers and do it as you normally would produce music.
As an Ableton power user myself, I love that you're building in the category. My question is to use your product do I have to use your specific DAW or can you integrate with, be it Ableton or the others?
That's a great question. That's actually honestly the major thing that we were thinking about when building this the past 24 hours. So right now as an Ableton user, I don't know if you're familiar with Reaper, but it's another existing DAW. I also use Ableton myself. Reaper is what we decided to use for this project because it is highly scriptable. So for a prototype we can have Claude directly generate these, or Reaper provides tools through an API. So Claude can generate these dynamic scripts that, you know, create music and add effects and delete tracks and anything we need it to do. In the future we definitely hope to implement this widget style into every existing music production software. But as a long-term goal, our final product would be hopefully a DAW itself which is designed to have a low barrier of entry and, you know, coach people along the way and not be bombarded with all these dials and buttons. Yeah, so we'd hopefully have a simplistic UI in the DAW and then in the chat it would pull up like dynamic sliders and say, 'I noticed you're making a jazz song. Here's a reverb slider for as if it was playing in a jazz concert hall. Feel free to adjust it on the fly.' So that's definitely the long-term goal.
Thank you. Yeah. Thank you guys.
All right. So, I have a question for the audience. Have you guys, how like show of hands, how many of you guys have tried to read a research paper and couldn't get beyond the abstract or... Yeah. Yeah. Okay. Or like barely understood the abstract. All right. All right. So I see the hands. All right. So basically that was the inspiration behind readme.ai. So basically we want to make research more accessible. So next slide. So here are the pain points, right? Like scientists, especially researchers, but people in general will spend a lot of time reviewing these research papers. But these research papers or literature, academic literature aren't really made with novices in mind. That creates a huge comprehension barrier that often causes a lot of people to give up before they're even done with the abstract, right? And so that's why we created ReadMay AI, built by us student researchers, for anyone who wants to learn. What makes us unique is that we focus on video generation that's grounded in scientific research papers that we're trying to understand and not... And we're going to have that.
So here's a look at our UI. Obviously there's a lot of components that we'll get into later, but the meat and potatoes is our video generation stack.
Who here has heard of 3Blue1Brown? How about Khan Academy? Yeah. So, we generate AI generated videos in much in the style of 3Blue1Brown and/or Khan Academy. So, let's take a look at a clip from one of these videos.
Can we get a click on that? In 1998, Larry Page and Sergey Brin faced an original paper that powered their original search engine.
We can also do much more complicated animations. So here's a sped up version of this without any voice. Completely AI generated and all with upload of a research paper. And then if we run out of or if it's too hard to generate animations like this, we fall back to Google's VO API for more complicated, more realistic animations.
Can we get a click on this? Yeah. The key insight is that accurate binding affinity prediction requires understanding...
Here's our video generation pipeline. We take a PDF, we put it into Claude, then it goes into Manim and VO and a voice API. It also goes into FFmpeg and MoviePy and it gets all stitched together into one nice, beautiful video for your consumption.
But wait, there's more. What if you want to learn it other ways? We have a key concept summary for key concepts that you want to understand.
We also have if there's code on the web on the paper, we can extract it and give it in an easy way where you can interact with it with the readme file.
And lastly, we even have a quiz that allows you to test your understanding. This quiz was generated from a biology paper. I'm a math and CS guy. I know nothing about biology and in just a five minute video I was able to learn quite a bit about this biology topic which is sort of an open-source AlphaFold. Yeah.
So here are some of our next steps. These include open source deployment, creating a crowdsource library of videos as well as creating a chatbot to help our users understand their papers better. Thank you for listening. And who's ready to go beyond the abstract?
I'm curious. Do you think that this will save people time or really more just actually engage them enough to get through it? Or is it both?
