CEOInterviews.AI
Start App
Jason Mars
Cofounder, Clinc

Mercon Keynote by Prof Jason Mars

📅 Aug 18, 2025 Center for Instructional Technology - University of Moratuwa 76 MIN 63 SEGMENTS · 4 SPEAKERS
MERCon 2025 University of Moratuwa.

Questions asked in this interview

8
  1. 52:07I'll come to you. Any questions?
  2. 52:45Maybe some kind of a direct embedding exchange or some kind of thing?
  3. 56:13What if we can provide a way to solve a problem with less number of codes?
  4. 59:12And my question is, what's your opinion on incorporating machine vision language models for this, like to unlock the new era of AI-powered software building?
  5. 1:02:10Because this is a very new research area, right?
  6. 1:06:00So my concern is about this new approach you talked about that is rather than doing fine-tuning, so you are trying to augment the prompt, right?
  7. 1:06:27So how do you propose that we get that same knowledge using prompt engineering?
  8. 1:09:55So does it prompt several times and then does it change the prompt based on the input and the output or it has a predefined prompt?
Host 0:48 ↗
Ladies and gentlemen, a very splendid morning to all of you. It is often said that the right words at the right time can spark the right ideas, shift perspectives, and inspire action. On behalf of University of Moratuwa, I warmly welcome today's keynote speaker, Professor Jason Mars. A special welcome to Professor Linga Tan, all academic staff, well-wishers, and other participants for this session. It's time for one of the most awaited highlights of the day, the keynote session. A space where knowledge, experience, and vision come together. Now, it is my pleasure to invite a distinguished member of our academic community. This is none other than our very own Dr. Loishan Wilmanikum. Dr. Wilmanikum serves as the local organizing chair of Merkon 2025 and is a senior lecturer in the department of electrical engineering at the University of Moratuwa. We are especially grateful to him for connecting us with today's keynote speaker, renowned for his experience and contributions to the field. He will be introducing our keynote speaker. Ladies and gentlemen, please join me in warmly welcoming Dr. Loishan to the stage.
Loishan Wilmanikum 2:44 ↗
Good morning ladies and gentlemen, distinguished faculty members, researchers, and students. Welcome to Merkon. It's my privilege to introduce Professor Jason Mars, professor of computer science and engineering at the University of Michigan, president of Jeeves Labs, co-founder of Clinc, and honorary associate professor in the department of electrical engineering, University of Moratuwa. Jason's work sits at the intersection of science, technology, and entrepreneurship with a clear mission: translate AI breakthroughs into meaningful real-world impact. He pioneered Jeeves and the J programming language, which is an agentic AI stack that raises the level of abstraction for building sophisticated production-grade systems. It already powers platforms at Clinc, Toby Live, and Zero Shortbot, proof that this is industrial infrastructure, not just a research toy. He's also a founder with successes: more than 62 million raised in venture capital, business grown to 200 million plus valuations, and AI solutions delivered to 20 plus million users across banking, healthcare, automotive, and food services. Along the way, he earned recognition including Bank Innovation's Most Innovative CEO in Banking in 2017, Voicebot's top technologies in voice AI ranked number four in 2019, Crane's Detroit Business 40 Under 40 in 2019, and Kauffman Vanguard Award in 2021. There are a lot more to go, but I will shorten this. He's the best-selling author of Breaking Bots: Inventing a New Voice in the AI Revolution, a candid take on where AI is headed and what it really takes to build world-changing technology. Before Michigan, Jason served on the faculty at University of California, San Diego, and worked at Google and Intel, bringing a rare mix of research depth, system building, and company creation to everything he does. This morning, he'll cut through the noise: where AI is actually headed, the open problems that matter, and the engineering principles that will define the next decade. Merkon, please join me in giving a warm welcome to Professor Jason Mars to deliver the keynote speech.
Jason Mars 5:30 ↗
That was a phenomenal, phenomenal introduction and I appreciate it very much. Hi. So, let me know when the slides are connected and you're good to go. Oh, I got to log in. Okay, let me do this. And I'll set it in presentation. All right, we should be good. So today, while the slides are coming up, this is probably one of the most exciting times I've had to be able to present a new talk. And this new talk is a very special one because I'm going to do a couple things. I'm going to say something very controversial. Controversial enough perhaps to put folks on their heels a bit. And you but bear with me as we go through this talk. I believe I'll convince you of a perspective that's incredibly new to me now as of the last year of work and the last year of observing what's been happening in the market. So what I'm going to do today, make sure the clicker is good. Yes. What I'm going to do today is I'm going to present three parts of a story and I believe that this story represents what's happening right now worldwide from an economic perspective and an academic perspective.
Now the first part is I'm going to present the struggle and I think that this is the struggle that ties together everything that we've observed the world doing in the last decade. Right? There's one unifying thing that's incredibly shocking and surprising that explains what's happened in the last 10 years. Then we're going to talk about the noise that we're experiencing today. It's a very noisy world. And in this kind of noisy world, it's likely that most people will be led in weird directions and believe things that aren't true. So, we're going to talk about the noise and then we're going to talk about our attack. How do we attack this world and specifically some of the work we're doing given the insights as to what the struggle is and what the noise is we're experiencing today. So, first let's talk about the struggle. Now I'm about to present the controversial statement that will sound not true and then we'll investigate and untangle and examine this statement. So what I would say is from observing the last 10 years of work that's been going on both from an economic perspective and an academic perspective, I submit to you guys that we don't actually care about models or training them and likely never will. A slightly more bombastic challenge would be perhaps we shouldn't work on fine-tuning.
Let's understand why. So, we're going to look at the last 10 years and some key innovations. I'm going to give you guys a very careful timeline and we're going to see why each one of these innovations is evidence toward a reality that we actually don't care about the models and we do not want to train them. And it starts with one of the most seminal works that created this trend, right? And so this is a paper if you've heard any of my talks before you've seen me talk about this paper. I think it's one of the most important papers of our time. In 200 years when archaeologists come to understand why did humanity create AI, this is the starting point and it was in 2013 about a decade ago. And what this paper did was get us to accidentally understand, somewhat surprisingly, they say it in the paper, it got us to accidentally understand that you can give a model a very simple task like predict the next word in a sentence. Super simple. A neural network can read Wikipedia trying to predict the next word. And then the model would come to understand something about the world that isn't that task predicting the next word. So if you train a model to predict the next word, what was found is you can use algebra on the neurons in that model.
So that in the space of the activations of that model in the embedding, and an embedding is just for everyone to understand it, just think of it as a multi-dimensional number in Cartesian space, right? So in two-dimensional space, it would be x y. That's an embedding. But in that embedding space, you can do algebra on those vectors. And that algebra could allow the model to tell you if you take a king, you subtract a man, add a woman, you'll get a queen. The model will tell you that's a queen. So it understood something about royalty. It understood something about gender and the relationship of gender with royalty in the world and all that model did was read Wikipedia predicting the next word. This is a simplified view that's going on but this was exactly the accident and I think this is evidence that we are trying not to understand how to train models. I'll explain more but this is where we got our first buzzword. We can get this for free. This knowledge, we didn't try to train the model this thing. Our goal wasn't to train it royalty and gender, but we got that for free when we let the model do a simple silly task on Wikipedia. No training required specifically for this kind of knowledge. Interesting. Okay. Now, what happens next?
This was still when AI was not that interesting on a global scale like it is today. This was Google doing some research. They had a lot of multi-core CPU. They had these data centers. They were working on AI because Google was doing research in everything. Kind of like quantum computing today. Google's doing some of the most advanced quantum computing work. So in the same kind of basic research spirit, they worked on this and they had that interesting result. But that result exploded with the academics still not in industry yet. I'm going to teleport a couple years into the future to 2018. The explosion in academia that happened with this work from Google on their multi-core led us to Google inventing a new kind of model, one that can be trained faster and better doing the same pre-training work as the prior work and that's a transformer neural network. This was the first transformer. Remember in GPT the T stands for transformer. Right now, every AI experience and AI excitement that you're having today is based on this kind of neural network model.
Okay? Everything you see out there, all of the GPTs, Anthropic, OpenAI, Google has their ecosystem, Meta, they're all using this model. And Google invented this model because they wanted to take advantage of GPUs, right? So there's a hardware story here, the enablement of the experience that we're having today from this model. But we'll teleport there. And the interesting thing about this model and that model is using something very simple, they were able to get intelligence without training the model. Okay. It's interesting. Now, there's a couple problems. The way that you, why was it not enough? Why didn't that blow up? Why was that not the GPT moment? It's a massive discovery. Well, the problem is it was hard to use because you still had to be an expert, right? The economy didn't react in a big way at that point in 2018 because you still had to be an expert. Yes, the model can learn things for free, but you had to use algebra on embeddings to get the knowledge. That's a little weird.
Or you had to fine-tune it, transfer learning. The paper I just showed was the BERT paper, the bidirectional encoder transformer that required fine-tuning. So you had to just do a little bit more training which means getting the data and figuring out the parameters of the model and all of this expert stuff required. So it didn't take off but methodologies for training were abundant in academia. Nobody in industry cared. And then there was another innovation. Right? So we discovered we can get this knowledge for free. Okay. We can't really use it. Nobody can pick it up. All of the students here, you know, weren't able to say build the next big products. So the work continued and then we had an interesting question. Well, we don't want to use algebra on embeddings on numbers and we don't want to have to then train the model because that's hard. Why can't we just use natural language to engage the model? So in 2020, this paper is credited with another big buzzword. The buzzword we're going to talk about is agentic AI today. But we're seeing the journey of the buzzword: pre-training, interesting, zero-shot learning. This was big. This was big enough for Microsoft to pay $1 billion to OpenAI. And I'm not talking about the chat GPT $10 billion. I'm talking about the $1 billion that nobody really pays attention to because the $10 billion overshadows that $1 billion of course.
But it's when OpenAI coined this term and showed that oh well you can use natural language prompts to get the model to do multiple things. There was a problem though. You might be asking well that sounds like prompt engineering. Isn't that where we are today? Not quite. So there was something that looks like prompt engineering that predates prompt engineering called zero-shot learning. Few-shot prompting. And I have to give a shout out to Google because one thing you should observe is Google is driving all of these innovations. There was a paper that predates this paper that coined the buzzword that explains the idea even better as to how you can use natural language to get the model to do some work. The problem is you had to use a very contorted version of natural language and you had to train the model to understand natural language that way. So for the things you wanted the model to know, you had to add a particular kind of prompt with colons in the right places and then you had to train the model with those colons. Yes, it was language, but it was unusable. Unusable from the perspective that we can't build software with AI easily. So, we're continuing to work. And then two years later, OpenAI gave us something very special, right? OpenAI gave us chat GPT.
And what was really special about this model? Well, I actually want to talk about why this model worked. So well, it's very similar to the training method that we're doing all along, but this was the first time that we took advantage of models training themselves, right? This used a reinforcement learning algorithm where you had models working with other models to challenge each other. Right? So you would have one model, and this is from the original paper, right? One model just scores how well you're doing. It's trained similarly as the model that's trying to learn the GPT-3. And then you have another model with the same pre-training, but it scores goodness. And that model would score the performance of your GPT-3 continuously as long as you want. Right? So this got us to a point where the model could understand natural language really well. And that's what you used the first time you heard about and played with AI. You used an AI that was trained with this technique or something very similar to it. Of course, there's tweaks to the technique, parameters being tuned. Keep in mind all of that work is happening by a very few people in the world, often a tower. We can call that tower OpenAI. And they're using incredible expertise to figure out how to train the model one time and then have it not need to be trained again and solve problems generally.
Now, this is the moment when chat GPT came out. This is the moment where oh my goodness, stock markets change. The top 25 cap companies in the world, the top companies in the world are all now fighting for the AI dominance. Right? We just saw that you can talk to it and it can talk back. And now we want to figure out how this is going to change the world and be the one to do the innovation for it. This is where we got the emergence of a buzzword that you probably found, you probably have heard of, right? And the reason you've heard about this buzzword, but you didn't hear about zero-shot learning. Maybe you heard about pre-training now because everybody's, it's in the news, but you heard about this buzzword because this is when everybody was trying to figure out what now the AI is here. Let's figure out how to do technology with that as part of the ecosystem, right? But there's still a problem. Just like there's problems, these problems just don't go away. Now, we have to figure out how to use prompt engineering to build software that uses AI to solve problems in a big way.
All right. Why is this a problem? Well, you want to get the model to do something interesting and then you have to figure out what to say to the model to get it to do that thing properly. We haven't solved this yet. I'll make this point in a couple slides, but what I'll say now is everyone knows the biggest use case that everyone's working on getting these models to be applied for. Does anyone want to scream out what it is? I heard. Yeah. Yeah. Yeah. Yeah. We want these models to build software. Vibe coding, right? It should write our software for us. And by show of hands, how many people have tried vibe coding in the room? Using a model to generate code for you. It's okay. You can say it. I know like the professors are here and you don't want them to know that you're using the models to do the coding, but a lot of us are. However, in production, when you're building real programs to launch to millions, it's well known that this isn't working. Indeed, there was a paper a week ago that showed that developers that are using vibe coding are about 25% slower at completing tasks. Isn't that interesting?
How did that happen? Well, how do we get it to do the vibe coding? Well, it turns out those are heavily prompted systems. Now, I said I was going to come back to this in a couple slides and we're going to talk a little bit more in a couple slides because you have to understand how Cursor, the most famous company trying to solve this problem, is approaching it and we'll talk about it again when we get there. But what then happens next, right? Because now we're up to today. So prompt engineering was oh my gosh, we have the chat GPT. Let's prompt it to get it to build our software. Well, today we have agentic. And this is an agentic system, this paper, and it's actually very hard to find papers that will be the seminal papers for agentic AI as a concept because the academic community isn't using that term yet. How many people have heard of agentic AI, agentic, you know, agents, bots and so forth? Awesome. It's not a term that we're using when we're writing papers, but this is a nice paper. It's Coder. It's issue resolving with multi-agent and task graphs. Interesting. Well, and we'll talk about agentic when I tell you where did this term come from, right? Where it didn't come from academia. All the other terms came from academia. This term came from Wall Street and our good friend Andrew, right? You know, he's promoting LangChain. LangChain is a company he's promoting and he gives talks. He's a brilliant academic. He's kind of that academia entrepreneur mix. And then he started using agentic around the time that these papers were being written. So then agentic AI became the term of our time. And what does it mean? It means you have separate AI prompt contexts working together to solve a problem.
Now I was recently on a one, two-hour debate on a call with Logi, a couple other colleagues, Ranga was there if you know Ranga, I know Ranga's in, yes Ranga was there, Linga of course, defining agentic because it's also not very well defined. There's many different ways you can kind of look at it, but I think every agentic system has this property: if you have the notion of a role and many roles working together to solve a problem, there's a system that isn't necessarily using that concept of different roles, at least the concept. So here in this Coder paper, you've got manager, reproducer, fault localizer, editor, verifier. And what these are are separate AI brains working with each other and together to solve a coding issue in GitHub, let's say, right? And this is how they interact. First, the reproducer tries to reproduce the bug and then if it can, it's going to go to a fault localizer. And these are different kind of AI specialists. And how they're specialized. Are they trained to do each one of these roles? Are we training the models? Do we even really care at this moment what model we use? I'm sure you're hot swapping out models left and right. OpenAI, no, it didn't work so good. Let's go to Google's Gemini. Nah, it's not as good. Let's go to Claude. Maybe it's Claude. I don't know. I just want the generalized brain that some expert built so I can build the software to solve my problem. Right? So this is the state of agentic AI. This is what we're doing today. And this is what Cursor does. Indeed, there's separate prompt contexts. It's all prompts. And the different prompt contexts where the memory is in each context is actually hitting the same model. It's not even separate models in practice, but they're working together to solve a problem.
Now, but what really is agentic? How are we using agentic approaches, which is what we're doing today, to build software with AI, AI-powered software? Well, Cursor is a big company that's using an agentic approach and they have one of the savviest agentic systems in existence. But is it solving the problem? It's very well known that it's not. It doesn't actually build real production-grade software yet. So we're still working. So I'd return to this question, right? And you've seen a story of where humans' energy has focused and what has been successful. We focused on getting these models to a place where we don't have to touch their weights. We don't have to train them. And the reason we're doing this is because we cannot scale the expertise, right? That's the problem. If we can have every person that comes out of that we graduate be equal to Ilya at OpenAI or one of the top AI engineers that are training GPT. If we could do that at scale, maybe we would care about the models and care about the training when it comes to building the software for the future. But we are striving very hard to build an approach with AI that doesn't require expertise in these things. And so the market doesn't care about it, right?
So that's kind of crazy. And the controversial lesson that might come from this is maybe building tools, and this is where I was telling Loy he's not going to like this part. Maybe building tools that make it easier to fine-tune models is not what the market will ultimately want. Maybe we have to solve how do we build AI-powered software. And the trend has been sprinting away from the models. It's sprinting away from the training. It's sprinting away from the hyperparameters. That's too hard. Our society can't support that requirement of expertise. So that has informed my new perspective on AI and that's why I say my new perspective is we don't care about the models or training them and we likely never will. Okay. So if that is true, what should we work on? That's a big question and that's a question that I think I'm not going to answer that for you today. I'll tell you what I think, but I have no clue if I'm right. But that is perhaps the right question for you to answer. And whoever answers this question properly is going to be able to do the innovation that's going to take us into the future. Right? Another question is are we done? We saw a journey with pre-training, then zero-shot learning, then prompt engineering, and now agentic AI. Is that it? Is that the right answer? How many people think we're, Yep, that's it. Agentic is the answer. How many people think unlikely? All right, that's interesting. We're not done, but we just want to be able to build something cool with AI for crying out loud. This is the struggle. That's the struggle we've been working on for the last decade and we're not there yet. And Agentic is unlikely to be the answer. Crazy, right? Because if you go out there in the world, you're not going to hear that. What you hear is everybody has the answer. It's just very curious that everybody's answer is different. That's just very curious. But everybody knows what the solution is. So be wary of that, right?
So that takes us to part two. That was the longest part too. The next few parts, that was the most important part. That was the hardest part because I'm substantiating a perspective and a claim that could be jarring if you just say it right. It's not obviously true. So you have to small-step the logic behind it. And I actually look forward to hearing from you guys what the gaps may be in that logic because I care about getting to the right answer. As I think we all should. Okay. So, the next part, the noise. This is what's going on today. Noisy. You go out there, people are going to be like, we know how to use AI to solve everything. Let's understand that. But one thing I would hope that you can kind of tease out from what I've been saying so far, right, is that these statements may be true. The first true statement is Nvidia gave us the hardware. Thank you. That's an answered question. How do we solve problems in general with AI? Hardware check. Maybe we can improve upon it, but we have the feasibility. Researchers gave us the model. I would say this is mostly a check. We have a model or at least we should assume this is true. It's very interesting. We have the model and it's incredibly intelligent and we have methods to make it more intelligent and we haven't seen a bound on how much more intelligent this kind of model can be, this transformer neural network. But now we have to invent how to build the software and I think that's true because empirically I am not aware of anyone who can competently build AI-powered systems on the planet effectively because Cursor is actually a failed product so far. It's impressive, it may be the best we can do, the closest, but in general it's not there. And there's research to support it, peer-reviewed research, as well as you hear this all the time when you hear people talk about using these systems to build production code. And the ones that say that they are successful in using it are spending 25% more time to produce that production code by using the tool. So if Cursor hasn't solved it and we've looked at a number, I'm searching for any competence in doing this well in the market, then I would say that that's still an open problem.
But let's look at what people are doing. Let's look at the noise. These are the folks that are saying they solved it. This is the things people are trying, right? Prompting is programming. Okay, a query language for large language models. And the question I want you to ask yourself as I present each one of these is does this look like the solution? That's the question. Gut check your gut. Let your gut tell you is this the solution? Top tier published paper, they have a method that they think is the future. It's a way to contort your prompts in a certain kind of way to get the model to do the right thing. Right? And so this is a language idea for how you can create prompts to help you do your job well. Okay, that's interesting. Ask yourself the question, does this look like what you're going to use to build software? It's a declarative language. It's very academic. Is this what's going to actually work in the world? Maybe not. This one's also really interesting. We've been watching these guys because we think they're interesting. DSPy. You know, we actually know some of the folks here. Matei Zaharia just a couple weeks ago, I saw a talk, I was there in person for a talk that he delivered on DSPy's future. Interesting. I don't know. But ask yourself the question if you've coded. Is this the answer? What this is trying to do is get the model to search for an answer online and then answer a question. This is how you have to build it. That simple task. Simple task. Do you know what the heck's going on here? We have a forward. This forward is inherited from the model architects, the expertise glass ceiling that I just talked about. This is what they use. So they're contorting the journey of building prompt-engineered solutions into something that looks like model inference or model backpropagation training software. Hmm. Okay, that's another try built by folks that are of that Ilya competence, very smart people. But is that the right answer for the world?
Okay, those are two interesting academic, those are probably the two most interesting academic papers until the one that's coming out very soon. We'll talk about that in a second. Let's look at industry. AutoGen. This is one of the most popular frameworks for agentic AI building. So here you can see your prompting here: planner, coder, tester, and critic. So you're going to have a planner then he's going to talk to the coder. You are a planning agent. You are a coding agent. You execute code and report results. Different roles, different prompt contexts to create an AI solution. Okay. Now, what is it that they're adding? Because I still see prompt engineering challenge. And I mean, is this syntax sugar? Coder, planner equals agent, planner, coder, system, plan, string. Okay. All right. Well, thank you, AutoGen, but I don't know. Ask yourself the question, is this the solution? Who knows? We're gonna come back. I have some early results. Let's call it armchair scientist results. We'll talk about it. Let's look at LangChain. Okay, LangChain's another approach out there. Ooh, they have LangGraph stuff. Graphs are interesting. But here we're prompting as usual and it kind of looks similar. These things are all very similar. They're kind of presuming, you know, okay, it has this...
Agentic theory, right? This prompt engineering first approach. Now let's look at Crew AI. And we like Crew AI. Crew AI is probably a lot of people's favorites. All right. This is another thing to help us build AI software better.
Okay, same goal. And then that's interesting. Oh, wait. Oh, sorry. And what's nice about this is you can put your agents as planner, coder, tester, critic together. And you see this kind of they give you the tasks, the things the agents can do. So you can articulate their agentic approach kind of nicely there.
All right. Interesting. So, this I just wanted to show it to you and have you ask yourself the question. And now I'm going to tell you as an armchair. This is armchair science because this isn't rigorous enough to be peer reviewed because I don't even think anyone's asked the question yet. I don't know if the question's been asked. So, is it working?
Spoiler alert, probably not as much the noise would indicate because each one of those very popular technologies say that they are the solution for how we should build software into the future. Very interesting.
So I'll give you two anecdotal armchair science reports. I went to this talk and William Horton is in production. He's working for a big company building agentic solutions. That's his whole job to architect and implement and build agentic solutions. And he gave this talk in person and I spoke to him for at least an hour after. And the first question I was asking his talk was about Crew AI and how awesome Crew AI is and how he uses Crew AI to do his work. It's his favorite.
So then I talked for like an hour. The first question I asked him was, 'Oh, so have you actually used it for production code?' He was like, 'Oh, absolutely not. I do it all by hand manually. I try to figure out the prompts myself. I'm not using it.' And I was like, 'But you just gave a talk about it.' Like, 'What's going on?' He's like, 'Oh, no, but that's what you do, right? Like you talk about the hot new thing so that you're relevant.' And so he was invited to give a talk. So he couldn't talk about 'I'm not using agentic frameworks,' right? So that was an interesting anecdotal situation. He's not using it in production.
Then we looked at the most popular open-source agentic systems to ask the question, are any of them using the frameworks? We have these beautiful frameworks. One thing I forgot to mention, he also said he tried LangChain and it didn't work. So he's been trying a lot of things, but these are Aider, very interesting, Claude Code, there is a war from OpenAI, Google, and Anthropic. There's a war to build a coding agent, an agentic coding system. Cursor became a $6 billion company. They're like, this is the future. If we're not there, we're going to lose our standing in the market. So we have to build an agent. That's what AI needs to solve first. This has actually been decided. I'm going to make sure I include that the evidence that that's actually the market decision as to what to do first with AI and at scale. So they've open-sourced the agentic systems. You can go read the prompts. We've looked at the code of all these systems. The best agentic systems out there.
None of them are using any of the frameworks up to today. So is it working? I don't know. This is my armchair science. And we've literally inspected the code and we've built diagrams inside Jaki of how each of these agentic systems look. There's a guy named Udit. Oh, I was going to say Udit, this talk is for you, right? Who's actually diagrammed out analyzing this stuff and he's also building stuff.
Okay. So interesting, interesting one thing interesting to note here is there's one kind of technology that has taken off and that is MCP and tool use. Tool use is going to be involved, right? So I wanted to mention that for the nerds out there that I think that that one's here to stay as we continue to try to figure things out.
Okay, so it's not being used. Okay, so maybe at this point in the talk you're like, okay, the next part is only about 10 minutes. If I can borrow that time from the next agenda item. But maybe I point this talk. It's like, oh man, we're not as far ahead in AI that I thought. At least that perspective says we aren't where I feel like we are with AI because the marketing dollars are there. There's trillions of dollars invested in companies saying that they know what's happening with AI, but really when you analyze it, it's not clear. So maybe you're like, 'Oh, wow. That's not clear.' So now I'm going to talk about and do with that what you will, right? But now I'm going to talk about what we're doing because we're on this journey.
And we have a couple principles that we're trailing down. We don't know what the answer is, but we're investigating it and I think we're doing better than what's out there in the market. But I don't know if we have the answer yet. But what's our preconceived notions? Well, I don't know if we should include touching model weights in the answer. And I think we also know that prompting is hard. Okay, at least the way we're doing prompt engineering today. Agentic is another approach and we're getting results from that, but that's still not there yet. Well, let me talk about a couple things that we're doing that we think are interesting, but we don't know.
There's a paper coming out. Oh, I circled it. It's animated in whatever. There's a paper coming out accepted last week. Gianica, some of you guys know Gianica. He's working with me, right? So, he came to University of Michigan. He's one of my PhD students. There's actually a couple other folks here. So Kougation's there. Kougation's in the room, I think. Yes. Yes. Yes. Kougation's on here. This is our people. Ceini. Ceini is also in Michigan working on research. This is a top tier paper. And I could go on. This is a big win so soon, right? And it's about an idea that we have for how to do coding into the future for model integrated applications. It's not even out yet. So hopefully it'll be on the list. This is every academic's dream, right? Hopefully Gianica's paper is going to be on the list of the seminal works. And so here we're saying maybe we shouldn't do prompt engineering, right?
Dude. All right. All right. Cool. I got so many anecdotes to say, I gotta stay focused. So, let's look at this example, right? So, this is just a typical code that you're used to. It's for a game. You got levels in a map. There's walls and you have positions of characters. This is the data structure. And that's all that's there. And we have a feature where you can say now I just want to get the next level and just do it by LLM. No prompt engineering as a core principle. And Gianica built this when he was in my seminar here. I was doing a seminar with Moratua on my AI course that I love to do here. It's like most fun thing I do as a professor. He just came and tried this idea that we kind of conjured together and it worked.
Without doing anything. Putting a by LLM here, the AI just generates levels and returns it back to the function's return. You don't do anything else. It's wild and it's crazy why it works. The hypothesis behind it, and I'll just say it's magic, but here's the hypothesis. If your runtime can see your code, but not just your code, can see the values in all of your variables in a snapshot at a time while your program's running. It can see your semantics of your code. It can see the semantics of the runtime variables. It can see everything. It should be intelligent to know exactly what to do. And here's why. When we write programs, we imbue those programs with enough semantics so that your buddy can read your code. The AI next to you that's biological can read your program and know what to do. If they also had the power to see what data is in memory, maybe it's going to know even more.
Here's what's wild. So, this paper is written and we're ongoing work. It's going to be published. We're working on another paper now. I just had a group meeting with a group we discovered this. Me and Udit discovered this and then I had a meeting with a group yesterday and they were like, you know, I was using this by LLM and it works better than prompt engineering. And I was like, that's what we're discovering last week. It's actually better results than prompt engineering and that's not what we were aiming for. We were aiming for, oh, I hope it's as good. That's what we were shooting for. How close can we get to human level prompt engineering? It does better. And here's why. I think this does better because a human has to anticipate all of the right things a model should see to do the right thing. And then you have to statically bake that in as one go. And you'll never anticipate the interesting information that's already present in the functional program. So maybe there's richer information here than prompt engineering. We're working on that.
Anyway, this is one idea. There's a few. And there's a repo. This is our repo. It's all open source. By the way, the design of the whole Jaki vision is to do research that matters. And the way you do research that matters, if you want to understand, the way you do research that matters is you do research in situ. You do research sitting in the field with the problems at hand. That's why Google has been dominating the innovation because they came up with the transformer because they were in situ of GPUs are here, we don't know how to do it. No one thought about it in academia and they do it over and over again. And with that observation we created Jaki so we could be in situ of the problem to do the research that determines what the future is. That's actually the hypothesis. Everything we do is open source. MTP is open source. Paper's open source. Jaki is open source. We're working hard on making it better. We showed you one paper. There's a lot of ideas in here.
So, this is the same idea by LLM. If the LLM can see it gets personality and they can see you passed in Albert Einstein. There's enough semantics here for it to know exactly what to do. It's an enumeration. It knows. And this is what code looks like. It's so by LLM is there. There's some other ideas, a graph view of the world and an agentic view of an object that we're integrating in. We're getting some really interesting results. We've written a Twitter clone in 200 lines of code. So you don't have to understand full stack. Everyone's a full stack engineer is one of the missions that we have here. So we're working really hard. We're doing a lot. And with that, I'd love to have you guys all join my next seminar. Engage with us. We're a very open group. We just got a big grant from the US government from the NSF, the National Science Foundation to make Jaki bigger. And that's really exciting and we can get an even bigger one next year. So, there's some National Science Foundation in the US money that's to make this more successful. So, we're happy to engage with anyone with ideas and with energy, to connect with us. And with that, open it up for questions.
Host 51:28 ↗
Hey, thank you Professor Jason Mars for that remarkable session. The floor is now open for a Q&A session and if you have any questions for Professor Mars you can raise it now. I also invite Dr. Loishan Wilmanikum to accompany the Q&A session.
If you have any questions, please raise your hand. I'll come to you. Any questions?
Audience Member 52:45 ↗
Yeah, so first of all thank you for the excellent speech, Doctor. So my concern is that most of agentic frameworks are actually still using natural language. Like when we are sending a message from one agent to another, we are still using natural language or some sort of a variation of that, maybe a JSON file or an XML file. But the thing is natural language actually contains some sort of ambiguity. So when an agent is interpreting a request from some agent, it might not be getting the correct knowledge. So what my question is, like, is there any potential to create an agent-only language or some sort of a communication medium that will be properly interpreted by those agents? Maybe some kind of a direct embedding exchange or some kind of thing?
Jason Mars 53:34 ↗
Yeah, this is, oh, that's such a beautiful question because perhaps you're on to the right question that can lead to the next area of research. Now, this is actually really interesting because it dovetails very nicely into the conversation we're having. So, one thing the person asked was problem with natural language is free flowing. There's ambiguity. You know, that's what makes this hard in a lot of ways. Is there an agent language that you think might make sense or some way to address that right ambiguity that makes this hard? You want to know what's really interesting? What happens if you have ambiguous code? I would say my quick answer is perhaps we do have something close and maybe it's code because when we write code, if it's ambiguous your code's going to break. So good code removes ambiguity. And maybe that's in part why, maybe that's part of the answer. You should collaborate with Udit so we can figure out well why is it better than prompt engineering? Maybe it's because humans are ambiguous by nature but the code isn't. That's actually really interesting. But I think you're right. I think that's an area of study that I know that there's some folks researching, starting to research it more from the perspective of NLP, natural languages, and how can you create encodings for models to communicate with each other in a more strict way, let's say, and more precise way. So I think it's a very important question and perhaps that's the next leap. And maybe it also explains why moving away from prompt engineering is fundamentally the right step, which is a bombastic thing to say right now. If you go online and say prompt engineering is dead, oh my gosh, you're going to get a lot of clicks anyway. Good question. That was a good question.
Cool. Any others? Did I offend any of the lecturers or professors in the room by saying maybe fine-tuning isn't the future? I think I must have. At least I said maybe. Maybe.
Host 56:03 ↗
Oh, we have one down here. Loy raising. I'm not lucky. I'm sorry. Tommy.
Audience Member 56:13 ↗
So actually currently agents are really working well with less number of codes. They are struggling when we input a lot of tons of code. What if we can provide a way to solve a problem with less number of codes? Like initially we had assembly, so now we've grown up with programming languages, high-level languages. So what if we can provide to...
Jason Mars 56:41 ↗
Yeah, that's a great question and it's a nice observation, right? And I think Tommy is really on to something here. It turns out that as we raise abstraction levels, complexity and bugs and misunderstandings go down. So if you tried to write your favorite game Halo but in assembly language, you're more likely to not be able to achieve that. But if you raise the level of abstraction to say C, maybe it's more achievable. You raise the level of abstraction further to maybe Python, you might be able to solve the functionality bit even though you sacrifice some amount of performance as you raise the level of abstraction. And so I think what Tommy's asking is, okay, is that an answer to getting models to solve the AI coding problem? Maybe they have to create too much code at too low a level. And if you can raise abstraction, so we can have an assembly language to high-level language leap, or maybe we have a high-level language to an AI-level language. Oh my gosh, paper title. Maybe that can be a step in the right direction to getting AI to perform better at articulating software, right? And this is in the code generation space, right? That's actually a really interesting idea. And I think telling that story, and the thing is these are beautiful stories because these stories really kind of challenge the status quo thinking. But I think that's an interesting one. I really love the understanding how communication languages and encodings. I think these are the right questions for the next generation. You guys are probably going to solve the problem, right? Like because you see how that decade went. You know what I mean? That decade, we're not there yet. But it's frustrating that more of the industry don't function like scientists. They all claim to have the goose that lays golden eggs and it confuses everyone. It's because there's money involved. Money can really corrupt, you know, meritocratic systems. Anyway, yeah.
Cool. Any other?
Audience Member 59:12 ↗
Thank you for the interesting talk. And my question is, what's your opinion on incorporating machine vision language models for this, like to unlock the new era of AI-powered software building? Like when we are learning the world, we take visual inputs as well as the language. So what's your opinion about the visual language models on...
Jason Mars 59:43 ↗
Yeah. No. So we love that. So all of the work we do actually generalizes to multimodal models. So actually the MTP I presented works with vision language models as well, right? So at the end of the day everything is turned into tokens that these models are seeing and vision is also a language, right? It's colors and well it's pixels when it comes to what's on the computer. And it's a representation and at the end of the day all of these things turn into activations. So like there's some floating-point values that are inputs to these models and I think that we should understand AI through that lens and in a lot of ways that's what multimodal models do. So when you look at say Llama 3.1 or 3.2 or whatever it is, the architecture of that, they can, they in the same latent space which means in the same neural network they are having neurons that's dedicated to audio speech, textual speech, pixel spaces, and all of these are kind of consumed into one latent space which means the activations that are moving through the neurons. So we look at the world through that lens and just to make this even more crisp so you can understand how it fits in here, an object in a language is a representation. So if you have an object, a class for image, that class represents an image at least the data that's in an object of that class. And so what MTP, what the by keyword sees are just objects, right? It'll take advantage of the information space. It's a small matter of engineering to then take that principle and build a system that can support multimodal models. And so that's what we do. We actually do the engineering to be able to have objects that are images hit the image part of that token ingestion space and then the language hits the language part and that's just wiring in the infrastructure to do that. Yeah. And that's available today and it's actually you can just go use it. You know what I mean? We're using it in the seminar class as we speak.
Audience Member 1:02:10 ↗
I have a little question extending what he's asked. It's about now images, videos, these things also represent with embeddings, right? So usually images divide into several patches and we use them as a sequence, right? Text embeddings also like that. So when we mix those things, are there confusions in agents? Because this is a very new research area, right?

12 more exchanges in this transcript

Sign in free to read the rest of this interview. No card required.

Sign in to read the full transcript

Cite this transcript

APA, MLA, BibTeX
APA

Mars, J. (2025, August 18). Mercon Keynote by Prof Jason Mars [Interview transcript]. Center for Instructional Technology - University of Moratuwa. CEOInterviews.AI. https://ceointerviews.ai/interview/1338870/

MLA

Jason Mars. "Mercon Keynote by Prof Jason Mars." Center for Instructional Technology - University of Moratuwa, 18 Aug. 2025. Transcript, CEOInterviews.AI, https://ceointerviews.ai/interview/1338870/.

BibTeX
@misc{mars2025_1338870,
  author       = {Jason Mars},
  title        = {Mercon Keynote by Prof Jason Mars},
  howpublished = {Interview transcript, Center for Instructional Technology - University of Moratuwa. CEOInterviews.AI},
  year         = {2025},
  month        = {aug},
  url          = {https://ceointerviews.ai/interview/1338870/},
  note         = {Speaker-attributed transcript with timestamps}
}