Ep 21 - Powering the Future of XR: How Qualcomm is Driving Spatial Computing & AI-Powered Immersion

Thank you for having me.

Yeah, I mean just think about it. We've transitioned. We used to have personal computing PCs and laptops, and then smartphones came along and we started to do more of our tasks on smartphones. It's really an integral part of our life. We didn't stop using the PCs; we used them for certain tasks, but now we moved on to using the mobile phones. I think that there is a similar huge interrupt that's going to be coming where we are moving towards spatial computing devices such as smart glasses or immersive AR devices or even MR devices, getting us to this next world where you basically can merge the virtual and the real worlds in a real amazing fashion that adds to the experience in a very impressive way. And I think that's why people will start to use these devices more, and in a way, some of those use cases that we may have on the mobile start to now move to these devices. We still hold dear to all the other devices that we have. So as I've thought about it, I've said, hey, what are the key technologies, what are the key experiences that we want? One thing that comes across is, I always call it internally, it's a very Qualcomm problem, right? Because you got to get to a point where you can do all this processing but at the lowest power possible. And this is really our pedigree. We've been doing this for the longest time. We don't come from devices that are plugged in; we come from devices that have tiny batteries. And those learnings set us up extremely well. And then some of the changes and really the acquisitions and the implementation and the R&D that we have done over more than a decade today specifically in XR sets us extremely well because we own all the perception, we own all the technologies, and of course the broadest roadmap in the industry for XR products. So I was like, hey, this is a great place. And I think especially now the inclusion of AI makes certain things possible that were just not possible in the past. So great inflection point for our industry in my way.

Yeah, but on your previous comment, I'd say, hey, I really look forward to the time when the Gen Betas make fun of the Gen Z's with their spatial computing devices. So, you know, that's I know. Yeah, get rid of that old cell phone. There you go. But coming back to your question, I think the key part over here is that what is a device that can actually in time really start to take some of the functionality of a smartphone? And with Qualcomm, mobile has been so central to us even though we have diversified in a massive way. We're in automotive, we are in XR products, we are in IoT, we are in computing, we are in all those different product lines. But I think the next-gen consumer device is really a spatial computing device. And I think to a great extent some of the popularity that we are seeing with Meta Ray-Bans shows it extremely well, that even with a device that doesn't have full display capabilities today, it really shows the promise of what a device like this can do and people are really flocking to it. It looks good. It also adds to what you want to do in a given day. And I always say probably the best device ever to do agentic AI is basically a smart glass. And I think great credit to Meta and team how they've been able to create this good product, but I think it shows people what a good spatial computing device can do. And that's why from a Qualcomm perspective, we've been very focused on making sure that we are leaders in this space and we continue to make the right investments to be able to make this space successful.

Yeah, you know, I've had this great privilege of basically driving Qualcomm AI strategy for the last six years before I took over the role in the XR business. But the real problem is, I think some of the key challenges that we had with MR have been, right? MR for example has been, there's been a problem with the availability of content. Well, with the inclusion of AI, you can actually start to create that content. The techniques like for example NeRFs and even Gaussian splatting are allowing you to be able to create 3D content quite easily from 2D sources. And I think that in itself really starts to make this happen in a very good way. That's one. Secondly, I think generative AI again allows you to be able to personalize these experiences in an amazing fashion. So imagine what happens today, right? There's a graphics designer, he creates a world which is a virtual world, and you see that, I see that, my 12-year-old daughter sees this too, right? But our likes and dislikes might be very, very different. But with GenAI, if I could basically tap into the likes and dislikes of that consumer, I could figure out what is interesting to them, what textures, what colors, what themes, and then if I could create a world that's much more suited to that person, now you got a personalized virtual world. And I think that really changes the way we interact with it. And it continues to evolve and change as your likes and dislikes evolve. So some of the challenges of MR, I think they're starting to get solved by the inclusion of AI. Another thing has been the ease of being able to, for example, navigate within a virtual world, to have for example an LLM that's integrated into it. You can do a massive job of being able to really just talk to it and be able to navigate through that part of the world. For example, we're working with a company and they do gaming, and we said, hey, these NPCs that you have in the games, they're typically standing over there and they have these lines that they will just say, but I can have a full LLM session with one of these NPCs while the LLM is actually running on my chip. You don't even need the cloud because we have so much on-device AI capability. So you see, if you put in the graphics capability, the LLMs coming in, and the GenAI part, MR gets much, much better. There's a similar analog on the AR side and on the smart glass side, right? The ability for you to have a camera up top that's able to see what you see means that the glass actually is seeing what you see. And that changes the way it can actually do multimodal AI because for multimodal AI, it needs to understand what are you looking at. The context that I'm able to get, if I can feed that into an LLM, the fact that it's daylight right now, the fact that I'm in the office environment, it really changes that response because see, today you go and you ask an LLM a question, it gives the same answer whether I'm asking or an 80-year-old boy is asking that question. That should not be the case, right? So really, I think those challenges start to get solved. Now, like you mentioned, a great use case, right? Translation. I was actually with one of our partners in China and they have come up with these glasses and the whole meeting was in Mandarin and basically the displays were showing me the English translation on the display while I'm wearing those glasses and I didn't have to stop them every 5 minutes to translate for example. It's an amazing use case or you go out to a restaurant in a country where you don't speak the language and you can hold your menu and basically it puts in the translated words on that menu and superimposes it for you. Right? There are so many such very good use cases that take away the friction for a lot of these scenarios. And I think similarly navigation and then I always look at verticals. So if you think from a medical perspective or education perspective or retail perspective, even enterprise perspective, right? There's so many use cases on remote help and remote assistance. Just a massive amount of things that become possible because of the addition of XR. And I think these two are truly symbiotic technologies and they come together in a way that basically is going to really set apart the experiences that consumers really, really want to have.

I think so and I think XR is a huge beneficiary of this AI advancement that we are seeing and actually AI too, right? When you, like I said earlier, right? I mean, if you're talking with an agentic AI, the best device is a smart glass because you're driving or you're talking, it should be able to do those tasks for you and you can just talk to it. There is no need for a keyboard. There is no need for holding something up. I mean the equivalent of this is that you got to hold your cell phone all day long like this to be able for it to be able to see what you see. But especially enterprise use cases, we have multiple partners, you mentioned Siemens, there are others as well, where I think it's really, there's so much low-hanging fruit that we can just tap into very easily.

Yeah. So, we're working actually with a partner that has a lot of oil exploration and all, and you can just imagine that you are able to actually put on these MR or HMD-like device and you can go in and essentially send a person over to this oil rig for example and they need to replace this part. Can you imagine the amount of time that it takes to train that person? But now you can actually have them walk them through step by step and it tells you, hey, take these two screws out first and then you do the next step and the next step and it's really extremely easy. Or an equivalent of this is also we have a company where you could basically be wearing smart glasses, for example, a foreman on the workshop floor and can identify that there are these five security or safety violations at this point in time on the workshop floor that nobody might have noticed, right? So the applications are really many. We have a partner also that does this for aircraft management and aircraft maintenance and they basically send out people where they're able to actually help them replace parts and maintain it by leveraging AR technology and really reduces the amount of training time for them and also allows them to send in many more people than who can actually typically go because they don't need to spend all that time getting trained first.

Fewer errors for example, right? That's a huge benefit for them because this assistance is there and many of times there are mistakes that are made that can be prevented in a very good way by using these techniques.

Yeah, I think what we have been working on for quite some time has been what we call split rendering, right? So now in a way what you can do is there is a lot of rendering for example in some scenarios that may be happening remotely but then with the engines and the compute that we are packing into our MR products now we can actually do massive amount of that on the device. And as you can do that on the device it actually starts to bridge a lot of that gap that you have with latency especially when all of the rendering is happening remotely. Now in this case you could have for example LSR-like functions on the device in the end to be able to give you seamless experience while still being able to stream the data down from a server. But the other thing that people are also looking at is to continue to do more and more processing itself on the device, more rendering on the device. We have the best-in-class graphics engines. We have the best-in-class camera technology for video see-through experiences. We have a very dedicated block inside for computer vision to be able to identify objects to be able to track things and all. And right with that, what you can do is you can actually do more on the device which again helps with the latency and in scenarios where you might have poor connectivity.

Yeah, I think it is. It is quite diverse for sure at this point in time. And that comes from the fact that this area is relatively new. Still somewhat new, I guess it has been around for quite some time. But I think what we need to work together is to work with alliances where we can actually create a way forward where we are not reinventing the wheel for each and every different use case that we might be thinking of. So for example, we are involved with a few of these different alliances. There is AR/VR Alliance. There are other ones also that we are involved with to start to make progress towards being able to standardize some of these interfaces and some of these file formats perhaps such that we can actually do it in a way that different devices can seamlessly work just from the go rather than having to do anything specific. So there is more work needed but I think it's starting to get to a point where people now see the fact that with these different providers, us being central in the sense that we are really enabling the full ecosystem. But I think having the perception piece and all the different technologies we can actually play a very critical role in aligning everybody together to a single standard such that we are actually able to make this easy for everybody to use.

So yeah, we are working very closely with Google on that. We're their partners with Android XR and we're providing not just the hardware but also working with them on the perception and other aspects. But in a way what we see is you have Horizon OS coming up from Meta. You also have Android XR coming up from Google. I think they're becoming good strong ecosystems that we can actually start to scale based on. We may have yet another one perhaps in China but I think it's not going to be very different than where perhaps smartphones stopped, right? If you recall at one point in time we used to have Symbian and we used to have other ones but really it's come down to two at this point in time. I think XR will take a similar path in my opinion as well. So I think it's very, very positive and we're getting to that good point. I think we worked of course with our partners like Samsung as well. So we talked about the first Android XR-based device coming out that's actually based on our XR2 Gen 2 platform. So yeah, a lot of these things are new moves but very, very positive at this point in time because now we see the ecosystems really stabilizing but at the same time a whole new form factor like the smart glass part coming up in a very strong fashion as well. And especially we're in touch with all the different display vendors and we think it's getting to a point also where now you have the ability to be able to manufacture good waveguide-based displays and others that can go into pretty thin devices. And I think that's the other big thing that has been a little bit of a damper on that segment to grow but I think that problem is getting resolved too. And then on our end we're pushing on all cylinders, right? So every product that we come out with it has a better transistor, it has a much better IP that we use inside for lower power for being able to be much more performant and at the same time completely new architectures as well to be able to reduce power consumption yet to be able to perform at the highest performance levels.

I think in the one-year time horizon, we will start to see glasses that basically have displays in them. And then, of course, one of the challenges with glasses really is the fact that there isn't enough space to put a battery. But you can imagine that on the enterprise side actually a bigger device is much more acceptable because it has to be ruggedized and it has to be used in very tough environments. So it doesn't necessarily have to be that fashion statement. So I think it will happen fairly quickly in enterprise and then even on the commercial side we actually have partners that are looking at launching within this about a year time horizon where people will be able to wear glasses that are about 50 grams or in that weight group that basically have fully capable display subsystems built into them. And then you can imagine the scenarios that we have talked about where people could wear this all day long, right? Where as they're going through and they're navigating the device is able to just bring up a simple navigation, not too intrusive or anything like that, hey you need to turn right over here for example. And I think those are the kinds of use cases that are really going to excite consumers to move into this form factor. Other interesting part is that I think what Meta Ray-Bans have shown is the device needs to look good, it needs to be something that's a fashion statement as well. And I think as people are looking at that angle too, I think there are so many options and as we make it easier for people to be able to launch these devices, I think many of those fashion brands and companies will come up with these products too. And then the inclusion of AI whether on the device or on the cloud, and by the way we have these kind of like three paradigms. You can do the AI in the smart glass case on the cloud where you use the phone to just be able to connect back to the cloud and the AI processing happens over there. The second scenario is basically you could essentially have that AI processing happen on a smartphone. So most of us still carry a smartphone. You can offload that processing over there. The third scenario is that you can just like you mentioned the Department of Defense-like scenarios. You can actually have an AI co-processor that sits on the body of a person that can basically do a lot of the AI processing on the device as well. So we see all three of these paradigms coming together as we move towards smart glass plus AI-like use cases.

Absolutely. Right. So 5G becomes, and in fact the way I put it is it's basically all the different connectivity technologies, 5G, Wi-Fi, Bluetooth, again one of the key areas where Qualcomm has been investing for many, many years and we really excel in creating these technologies with the lowest power possible but at the same time being able to receive the signal in the most complex of scenarios. Now the other part that's coming together along with this is that we're truly moving to this hybrid AI-like configuration as well because I think the advent that we're seeing on the AI model side is smaller models continue to become more and more capable. So for on our devices for example we can run a, on a smartphone you can run an 8 billion parameter model that's pretty capable and actually a model like a Llama or for example Qwen and Baichuan in China, these different models essentially can run on the device and as you're able to do that you can offload to the device. And I usually explain it this way, it's almost like we'll get to a point where there's a constellation of devices around the person. You have your smart glass, you have your smartphone, you have your earbuds, you have your smartwatch. And really the device should be able to decide where to do that processing, which particular device is the most optimal for it. So let's say you're at home, your phone is available. So you do the processing on the phone, but then you go for a run or you go for a bike ride and now it starts to do that same processing just on the smart glass. It goes to a smaller model and continues that experience in one way or another. So I think that continuity of experience will make it even more attractive to consumers. And it's actually something that we can do quite easily at this point in time.

Right, so I think we basically have the, like I mentioned, the widest roadmap when it comes to devices. What we want to get to is some degree of an ability to be able to have multiple tiers of products depending on the need and then you can have these digital twins for example of an airport or of a hospital and there are even actually countries that are talking about country-level digital twins because there is so much benefit that they can accrue in terms of being able to provide services and first responder-like scenarios. So we have actually enabled all those different classes of products and like I said if we can get to a certain degree of standardization where you can have file formats and all that are consistent between these different devices we can actually start to offer many different tiers and categories of devices that all are able to leverage that experience. It's a little bit restrictive at this point in time, but people are doing these digital twins like devices with multiple different partners of ours. And we're working with partners that of course have this ability on the cloud side also to be able to stitch this together. But we see this advent becoming much, much more popular in coming days.

Yeah. So, I'm going to make this bold forecast over here that within five years, all of us will be carrying an additional device with us, which is a spatial computing device. It might be a light smart glass-like product. It might be a fully capable HMD that allows you to have a fully functioning digital twin or fully immersive sort of gaming experiences. But I think we will all have a device like this and I think those of you who have tried some of these devices realize the amazing benefit it produces. And like I said, AI continues to merge with XR in a very big way and I think that's what's going to make these devices far more valuable and useful to our consumer partners.

Yeah, I think this is such an amazing field, right? I mean, when I look at all the different products that Qualcomm works on, I feel that XR is probably among the most, what should I say, most blue ocean sort of an area, right? There's so much innovation that still needs to happen at the device level for example, right? We got to get these smart glasses that last through the whole day. We got to basically have perception that is pristine with full amazing eye tracking and hand tracking and to be able to understand full 3D reconstruction. So a lot more work needs to happen in the algorithms on the next level. We need to get to better video see-through experiences, much better imaging quality. So at the core IP and technology level all those things need to get better. So there's room for improvements at every one of these levels. Then on the level above that it's basically on the experience level, right? Can you stitch together and create something that's not available today. For example, on the education side, I always love the kind of integration of smart glasses and AI on the education side in a massive way. You can imagine that a student that's actually learning something and their smart glass or their VR device is actually able to answer their questions. It's able to address the difficulty that they might have. A simple camera that looks towards the person can actually even gauge if the student is able to understand the material that's being delivered to him. And if not they can change the pace of that using GenAI, right? So the amount of amazing work we can do in medicine, in education, in enterprise, in retail, right, in entertainment, there is so much opportunity for innovation. And then of course even on the form factors itself people are developing different kinds of devices that are able to do this whether a smart glass that you take it into your car and it's able to connect with your car so the intermingling of these devices is going to be extremely interesting as well. We of course have the Snapdragon Spaces platform also that we're going to continue to enhance as time goes by to be able to make it even easier for people to be able to come onto the platform, create these experiences and just wow the consumers.

Sure. So, please follow me on LinkedIn. I'm also on Twitter at the same time. We put out a lot of blogs and white papers on the Qualcomm website. So please follow me and you'll see a lot more pretty exciting stuff coming up in the days ahead.

Thank you for having me. It was awesome. Thank you.