AR Insider Innovators: Qualcomm

Thank you for having me.

Yeah. So, you know, I do a lot of very interesting things and I get paid for it. But what that really means is that basically I drive the technology, the road map and really the ecosystem from our AR, VR and MR product lines. And it's a super exciting time in this domain especially with what we're seeing today with XR and AI coming together. And I think there's a massive amount of opportunity there and I'm sure we'll talk a lot more about it in the next few minutes.

Yeah. So I think we are very focused and if you picked up any device today that's basically for VR, MR or AR, there's a good chance that it will be using a Snapdragon solution. And the reason for that really is we have been investing in this space for more than a decade. We have basically invested in creating all the perception technologies. We have acquired multiple teams for example for hand tracking and for 3D reconstruction, other core perception technologies. And what we've really done is in the process of acquiring them, we've looked at those portions that are executing multiple times a frame or multiple times a second. We've taken that goodness and intelligence and actually hardened it in the silicon. And in the process of doing that, we are able to do certain things at power that nobody else is able to do. And with that approach, what we have done is we actually created two distinct product families. There's a product family for MR and VR product line and there's a product family for our AR and smart glasses product lines. And we call it the AR products. And essentially the way to think about it is these two product lines are actually quite different. The requirements that you have on an MR headset with the level of rendering capability with all the concurrent AI networks that are running at any given point in time, all the neural networks for hand tracking and eye tracking and all that are running, it's a very very compute device. But what makes it even more challenging is that an ML device requires you to work at that level but to sustain that performance level. And that is what we are very good at at Qualcomm, right? We've done this in devices that are not plugged in, but they run off of a battery and we really like what we think we can take this technology towards. Similarly, on the AR side, what might be a 10, 15 or 20 W power envelope on the MR side becomes a fraction of that, logarithmically smaller, almost like a watt at most. So now you got to be able to do best-in-class camera technology, best-in-class audio, video, microphone, all of that running at a power envelope that very few companies can do. And again, I call this a Qualcomm size problem because we know how to be able to do more for a given amount of power than anybody else. This is really our pedigree. And then to put it all together, what we really focus on is the right transistor technology. Then on top of that, we take our best-in-class core processing technologies. We have best-in-class CPUs that we do in-house. These are custom CPUs. We do best-in-class graphics with our Adreno family of products. We have the highest performance camera technology and all of these are really relevant and absolutely critical to make best-in-class XR products. We built those on top. On top of that, we bring in the perception algorithms. On top of that, we have our developer platforms like Spaces and others. And of course, we support Android XR and we support Horizon OS as well. So those are kind of all the pieces how we're driving the space forward.

Absolutely. Right. I mean, this is fairly specialized, right? Like you alluded to, right? Even if we just talk about graphics and rendering, we are not just doing normal graphics and rendering that we do in a smartphone form factor. What we do over here is foveated rendering for example. And the whole pipeline has to be foveated. And fovea being of course the center of your retina. So you're really doing much more resolution at the center with lesser resolution as you go further out. And that's how you're able to do this amazingly high resolution displays and be able to support them in a sustained fashion. And those specializations that we're talking about, in addition to that we have built in a very specialized computer vision block and that block has all of these hardenings for the perception algorithms in addition to that. And then what we have done is and that's why these products are all custom built like you alluded to. So XR products for MR/VR, AR products much more for smart glasses and immersive AR and see-through like products, they are specialized in the sense that now you're much more focused on power. I'd say for the latter category it's really power, power and then it's any other factors and that's why they're so distinct. And you know we're very glad that we've built a pretty large portfolio and connecting to that actually we just announced the latest member of the AR family that we are calling the AR1 Plus. So you know we've been hearing our customers and what's very important is what is critical when you want to build a nice form factor smart glass. It's basically number one the size so we're really focused on reducing the size of our solution by about 20% such that it can fit into more compact smaller devices on the leg of the smart glass. That's one key thing we've done. The second thing we have done is really focused on improving power consumption because again these devices are very light devices with very little to no room for a battery. So as you are able to use that battery in a more efficient manner, you are actually able to make that use case last for much longer. So that's been the second major area of focus. The third one has been camera and image quality. Right. I mean, you can imagine that you're sitting in a dark restaurant having dinner and you're holding the menu in front of you and your smart glass needs to understand all of the text that's actually written on that menu. Well, you can do that with the great multi-frame sort of processing that we have built into our products, but we have enhanced that with AR1 Plus. We have also enhanced the image stabilization capability because what we're realizing is the way people move their head is quite different than the way they move their hands when they're holding a smartphone in their hand. So we're actually making that a lot more capable. And then lastly, we're really pushing hard on the AI capability in these devices because I want to say really that the ideal device for agentic sort of AI flow or for a world where it's an appless world where AI is the UI and that device is really a smart glass. So what I was able to show on stage today and very exciting was essentially a 1 billion parameter language model running just on the smart glass. And this smart glass is in airplane mode which means it's not using the internet, it's not using your phone for offload, it's not using the cloud for offload and it's able to run that full foundational language model entirely on the smart glass. I feel that really opens up a whole set of possibilities of this device form factor existing and working independently of anything else. And I think that's where I see this technology going forward in the future.

Yeah, I think that's a great point. Right. So by being able to do more on-device AI processing we are able to take away some of those privacy concerns that people have because now that information that's very very critical, very private never leaves the device and in being able to do that you keep that information private. Secondly, if you want to do certain excessive amount of cloud processing there is a cost associated with that. So if you're able to do it on the device you're actually able to do it pretty much free right and free is of course a great price. And then the other aspect that comes in with this also is the personalized aspect of AI. So what I mean by that is if you really think about the constellation of devices around a user, you know, you have your smart glass, you have your smartwatch, you basically have your smartphone, you have your earbuds. The context information that you're able to get from all these devices is uniquely different. The smartphone typically knows your location. The smart glass is able to tell you about all the environment outside. It's able to tell you it's daylight. It's able to tell you this is, you know, you're talking on a video conference call at this point in time. Your smartwatch is able to tell you what's happening inside, you know, the heart rate, the activity level, the stress level, all those things. Now can you imagine that with this additional context how much of a better job can you do with generative AI when you have those additional pieces of information and I think that's the personalized aspect of AI that we are able to bring when we do it on device that makes it quite distinctive as well.

Yeah, I think we are very fortunate that we're working with really the key companies in the space that are launching these amazing devices. We have had a very long-standing relationship with Meta and we've launched these amazing products out there. You can look at the Meta Quest devices. If you look at Meta Ray-Ban smart glasses which have really established this as a category and people are really flocking towards it. I think you see celebrities wearing them. You see all the people wearing them because of the ease of use that comes with a device like this. So we're very excited about working with them as we go forward as well. And of course I had one of our partners up on the stage as well today. So you saw quite a lot of the new use cases that they are looking at doing. And I think that's how this space moves forward with us collaborating with all of our OEM partners. Similarly, Snap is very unique that they have a very different form factor. So when you think about Meta, they of course have the all-in-one great device which is the Quest-like product. They have the very light smart glasses. Snap has been focused on an all-in-one AR device, a fully immersive AR device. And it's a very unique, a different form factor. And I think that's the key point with our platforms. We're actually able to enable very well differentiated end products for our OEMs. They're able to differentiate on the form factor, on the use cases, on the capabilities, on power, on battery, all of those things together. And Snap is actually taking a dual SOC architecture where I mean if you talk to them, they'll explain how they're actually able to do more of the apps on one of the application processor and then much more of the perception on the other application processor. And this gives them some very unique benefits from a power and other aspects as well. And then of course we've just launched our Android XR product lines and we also had our Google colleague on the stage today at AWE just to talk about what they're thinking of doing. We're very excited, you know, devices with that we're working with our partner we worked with for years Samsung will be launching the Muhan device that they have shown and really a lot more. But I feel that these are all very different swim lanes and that's the real crazy, amazing part about this space because with Android XR it's attached to a smartphone, with what we are working with Meta and with some of our other partners it's attached almost to the cloud and these companies have amazing cloud assets and the AI assets that gives them a very unique advantage in that space. And then we have a third set of partners. They're actually talking about connecting the smart glass to a puck. Think of a puck as a, you know, like a headless smartphone almost. And you're able to offload from the smart glass onto that device. And in doing that, you're actually able to get even longer life on the product, a longer battery life by being able to say that all the work is not being done on the glass. So really different form factors, very different use cases, swim lanes, but it allows the consumers to be able to really experience some amazing devices that have launched, but there is some really, really impressive stuff that's coming in the pipeline as well.

Yeah, I think this really like I said it opens up this unique new aspect right. In a way what we are able to do for example with the device that we're working with Samsung on which is the Muhan device now it's able to actually bring a lot of those assets together. It's able to bring in Gemini which is of course the LLM that Google has and be able to bring it together such that the interactions in XR become a lot more seamless right. You're able to navigate if you look at some of the demonstrations and the use cases that they have shown, instead of having to try to, you know, figure out what particular part of a Google map you want to see, you're actually able to just speak to the device and it's able to take you there, right? To be able to see that you can go to a restaurant that you're about to go. They are able to bring in the ability such that you can even understand the ambiance and walk through that restaurant before you go over there. And then of course some of the work that they showed at Google IO which is super impressive work which is on smart glasses. But they like I mentioned this is a smart glass that's working with the smartphone which is an asset that many of our customers have and at the same time this is one that they have and I think it's a natural extension from there also to go towards smart glasses. So I think both are very unique. Both play to the strengths of these companies that are making these great investments in AI and in agentic AI flows to be able to bring that to a device. And I come back to this that basically what other device can make this AI agent the most frictionless experience than a smart glass. It's already on your face. You're talking to it. You're not typing. You're not basically entering anything through a mouse. It's all extremely frictionless. And I always say this, but you know, technology is much more interesting when it's not in your face. It's really much more transparent. And that's pretty much what smart glasses do for AI. And I think you will see this trend continue where all the people who are making these big investments in AI, they want to take their AI assets and LLMs and agents and bring them to the consumers and you will see them work and launch these smart glasses to be able to exactly do that to bring that asset right to the consumer.

Correct. Yeah. And I think this point that you're making is absolutely a critical one. This sort of transition always in the industry is what allows completely new people to come into the marketplace. It allows real amazing amount of innovation. Right? Llama for example is a great differentiator for Meta. I mean this is open source. It's a great model. What I showed today on stage was actually a Llama model right it's available. It's great. Its quality keeps on improving. The trends that we see for example and that's why I showed this particular use case today at AWE which was that as models become more capable they're becoming smaller as well. So if you look every year while maintaining the quality at the same level the model is a tenth of the size. Yeah. Well, that's exactly what allows them to be able to run on device. But I think that's going to be the differentiation how well you can do it. How seamless of an experience can you make it? And that's where you give credit to Meta Ray-Bans, right? They have essentially been able to create an amazing experience in a very seamless fashion. And that's why we see this amazing traction. So I think you know smart glasses are unique in this sense. This is not just about hardware. This is not just about core software. It's also a lot about getting the AI part right. And like you said, that part is not easy. As we can see from Apple's example, right? At the time when they announced it to where they are today, it has been a little bit of a challenge for them to be able to bring some of those assets to the consumers.

Yeah. So I spend a lot of time looking into my crystal ball actually. No, what I really mean is that basically these new verticals are emerging and I think we need to make sure that we're addressing all of them and that's why you mentioned the swim lanes example also right people are looking at one aspect or another but what I want to do is as I'm defining a chip when I define that platform I define it almost two years before it really comes in the commercial product. So I think much more from the perspective of verticals and that's what I showed today with our work that we are doing with XR Sports Alliance. We announced actually our second set of our second cohort which includes actually some really really great partners and we want to be able to show why an MR device is the best device to be able to experience sports. How you're able to see and view angles that you're just never able to do that on a traditional flat screen TV. Right? So that's one vertical I talked about. I also talked about our engagement with the VA and how they have been able to utilize MR devices to really be able to do an amazing job of patient care. I also talked about education our partner there Prisms as to how they have seen the efficacy of the use of MR in education because it's very different when you tell a student about something versus if you show the student about that same thing. It really comes across in a very very strong manner. So we also talked a little bit about all of these and I think that's how it comes together. But with the XRSA the sports alliance I want to be able to get to a point where fairly quickly we'll have a proof of concept. We'll be able to show how we can bring these unique aspects to sports and to sporting events that people are able to see and such that they can experience these unique things and then the vision that we put together when we define those parts starts to come to fruition in these very different verticals and the very different requirements that come with them.

Yeah, this is really a form of almost like embodied AI or physical AI because AI is present with you in a way, right? So what we were able to show recently, put it on our blog post as well is a very great use case of multimodal AI agent and what we showed over there was that a person can come up to the smart glasses and essentially there is a set of equipment that's right in front of them and it picks a profile first of all whether the person is a younger user, older user, a pregnant person and then essentially it takes in all the exercise equipment that it has via the camera that's available on the smart glass uses a LLM model in the background based on the requirements of the user as to what their fitness goals is. It's able to recommend a regimen and a routine that they can use to be able to hit those goals. And you know, we showed it of course in the fitness context, but you can see that you can actually extend this to very many different areas. For example, in the productivity space, you could look at this as being, you know, the information that you're able to give back to a worker depending on whether he's a trainee or whether he's an advanced user. It's able to give him a very different perspective. In the medical field, right? You can just imagine that a doctor could be wearing this and it can transcribe all of the discussion that they have with the patient because which is by the one of the biggest pain points that doctors have because they have to spend so much time afterwards in trying to basically transcribe everything. It can translate for them. It can bring an X-ray report in front of them and even point out, hey, these three might be areas that you really want to focus on or it can bring up a blood chemistry report that says, hey, these three particular readings are out of line. So, the just the ability to be able to utilize multimodality is going to be very powerful. But what I really envision is, you know, I'm coming into work every day and I'm like, you know, I need to meet these three guys on these five topics and I better be able to do it such that I can get my work done. I'd like to be able to talk to an agent. It's able to go in, look at my calendar, look at the calendars of those people, put in the title, get everything scheduled by the time I get into work. Or you can imagine, right, you can tell your glasses, hey, I want to meet a friend and this is the friend at 5:00 p.m. or for dinner tomorrow. It's able to choose a restaurant for me, looking at Yelp, looking at the map as to what is close to me and basically take care of all of those steps. That's what you want. You want AI to simplify your life. It's a tool. You know, I keep saying this. It's really a tool that's meant to reduce the amount of time that you spend doing mundane and routine tasks. And I think that's what agents allow you to do.

Yeah actually I showed the demo that I showed today the outputs from the LLM were showing up on my display and we're actually mirroring it onto the screen. So yeah absolutely very doable.

Yeah. So, we actually are really taking a very full and complete look at this, right? What we are seeing is, for example, one of the key things that I talked about was how the device form factors are evolving, right? In the MR scenario, we have had all-in-one MR devices, but now we're moving to optical see-through MR products. We are also moving to disaggregated sort of MR products where you might have a light visor that you're wearing, but then there is a puck that's offloading some of that such that you can wear a device like this for much longer period of time. On the AR side, we talked about the combination of attaching this of course to a puck or to the cloud or to the smartphone, but even attaching it to perhaps a watch or also attaching to a car or attaching it to a PC, right? That's one thing we talked about. But additionally, what we talked about was when we think about the realm of smart glasses, you don't have a keyboard, you don't have a mouse, you have no way to interact with it. So, we actually launched a ring form factor which you can use to be able to control a smart glass device with the IMUs and with the sensors that you have inside while it's connected to you via Bluetooth. And it can be very discreet because your hand could be down and you can still be controlling your device, not something that's very obvious to other people around you. And it becomes a very good way to be able to interact with your device. So, we talked about that too. And of course AR1 Plus platform that we're very excited about. So those are some of the additional things in addition to XRSA and some other aspects.

Thank you for having me. Great to discuss all these great topics with you.