MWC26: Ziad Asghar on Unlocking the Potential of XR

In every era, there is a need, a spark, an instinct that brings us together to imagine and to innovate. This hasn't changed. What's possible has. We create technology that makes our world better. Ideas that move us, empower, and connect us. But great ideas cannot thrive alone. They need to be shared, challenged, refined. Through our insight and collective curiosity, we advance and grow. New solutions, new directions, new answers. We do that here together. This is the age of connected intelligence. Welcome to the IQ era.

Good morning everyone and welcome to day two of MWC. We've got a fantastic panel to kick us off this morning talking about what is the X factor in the next reality. So I'm really excited to introduce our fantastic panel.

Lovely. Welcome everyone. So, we're just going to do some quick introductions and then we'll get into the questions for our panel. So, I'm Lisa Kelvy. I'm the global head of infrastructure and transportation at KPMG. I'm personally really excited by this topic because the potential for the industries I work with is absolutely enormous and we have a fantastic panel here with the real experts this morning.

Good morning everyone. I'm Rafa Camargo. I work at Reality Labs at Meta and I lead a lot of the engineering work to develop our wearable products.

I'm Rob Morgan. I'm a storyteller and an experienced designer. I wrote the first manual for storytelling in augmented reality. I started life at PlayStation where I was a lead game writer working on augmented reality and virtual reality games. And now I work across museums and games and theme parks integrating augmented reality technology.

Lovely. Thank you. So, we've got in safe hands this morning. We've got a fantastic panel for this topic. So, we're going to kick off with a question about the current landscape for XR. It feels like we're at a real tipping point now. As you just said, moving from the smartphones into other devices. Really quite exciting blend of human intelligence and then the extended reality. Let's start at this end. What excites you most about where we are in the next few months? What's coming up that's really exciting?

Yeah, as you said, I think we're like in a very interesting point and an inflection point. If you look back at the history of personal computing, you know, we've been giving ourselves more flexibility, more economics, more mobility to it. Mobile has been a big leap forward on that. But the one thing we haven't been able to disconnect from is like we are tied to screens, you know. So you're either on your virtual world or you're in the real world. I think with XR that's what we are changing. Suddenly digital and physical come together which is a lot more natural for us. You cannot say like oh leave aside your digital world, leave aside your physical world. They really need to come together. So we are at the point where finally the technology is maturing on that and we're not talking about like even next year the following year I'm wearing these glasses and they already have AI that is contextual you know I have audio and all that. Really augmenting my senses already and I don't need to be basically heads down onto my screen to do a lot of things as an example. So that's really the exciting part here is really the logical natural evolution to what is basically personal computing.

Fantastic. Rob, I know that you're really passionate about the content and the experience. What's exciting you?

That's the thing like as an artist and a content person right now we're also going through this transition from pass through virtual reality and pass through mixed reality into proper see through which has been the dream for so long that people like me have been talking about it for well over a decade and I think what's exciting for me is being able to put dimensions of adventure and gameplay into people's normal lives into people's ordinary walk to the shops because I think it's telling that still the most popular and successful and recognizable augmented reality XR experience is Pokémon Go. Not because it necessarily augments really sophisticated graphics into space, but because it adds this dimension of adventure and objective into people's lives. I'm really excited as we move away from pass through and into see through to be able to put experiences out onto the street that really add something to people's everyday and to create that kind of context of content in their lives.

Fantastic. And for you

Fantastic. And for me personally, you know, my background is infrastructure and construction. So my whole life has been about taking 2D things and making them 3D in reality. And I can see enormous potential for these technologies in terms of how we do that much more effectively. I think picking up on some of those themes there. I know Meta talks a lot about super intelligence. Could you bring that to life a bit with some sort of real world case studies of what that means for you?

Yeah, I mean Mark and Meta, we've been talking about super intelligence. It's a way to kind of like maybe express the AI in a more clear way to consumers. And what we mean by that is essentially it's an AI that is contextual aware that is always on with you and then it anticipates your needs and reacts to those and proactively works on those versus just reactively working on those. You're seeing just a beginning of that at this point. That's starting right now mostly reactive. You're still like querying it and all that, but very quickly you're going to start seeing things where it's like proactive and basically being one step ahead of you, always helping you because it really understands your environment. We're talking about like you cannot do that with your phone. Your phone is in your pocket. It doesn't know what's around you. It doesn't understand what you need in this moment. You know, you may get a message but it may not be the right thing to bring it up to you right now because you're in a very important meeting as an example and so on. So all these things are going to change significantly the utility of the product. And it's not just for consumers. You can also look at enterprise. There are a lot of use cases that we're seeing emerging as well on this. For example, there is a lot of field operators you know that need a lot of skills to like is there a repair or an installation and all that we're seeing those emerge. Suddenly you cannot have like your top tier support everywhere. So there are two parts to that. Like one you're like there dealing with something you're hands-free but AI is with you they're seeing what you see it can whisper to your ear it can guide you through that when with the AR display it can also literally digitally guide you overlaying a digital components to the physical things and then even when the AI is not ready for that you already are connected to maybe a tier three support that essentially now can scale because they can support in real time someone that is basically gone beyond the complexity of what they need normally do in the field. Yeah. So that's one example. We're also seeing other examples like for example in training you know they basically you're wearing this you're training on a new activity and all that they is really both guiding you but also seeing what you're doing where you're stumbling where you're doing it well and really coaching you through it. That will be another good example where you're going to see also in the enterprise doing these things. Same thing you can translate to consumers. Yeah. So, we're really really excited about this. We're literally at the tipping point of this thing and like actually the examples I just provided they're real. They're happening today and then you can imagine now extending this especially as the AR displays come on to be more accessible on these products as well which is really coming now together. We already launched last year first display glasses in market that are truly wearable but very light not like things you saw before that were honestly from a form factor were not there even if the technology already provided some of those experiences but the form factor is also fundamental so you're also going to see that in the consumers so we're really really excited about it.

And I can really see that because I know in the sectors that I work in one of the biggest challenges is time to competence so typically engineers field staff to be fully competent can take seven eight years and actually if we can use this technology to really get that down to enable them to do more and be supported I think there's a huge productivity potential absolutely with that. So coming on to you Rob in terms of the experience I know you've mentioned museums education what do you think are the things that are exciting there?

I mean I think that AI as Ziad was saying like AI is a force multiplier for what we were already trying to do on your smartphone in terms of encountering the real world. That's what's really exciting about it, being able to seamlessly give you like information that's fed into your field of view as you're encountering like real things rather than things just mediated through a screen. So from a game perspective, AI is not just improving the pipeline of being able to make stuff, but also it's unlocking the capability that's easy to imagine, but which has been so impractical for so long of reskinning a real environment to make it look like a Victorian fantasy or make it look like you're living in a Nintendo world. Like that kind of game and ambient fictional experience is really exciting to me as a storyteller. But then if you think about a cultural experience like visiting a museum, we've been looking for an interface that allows you to kind of curate your journey around a museum for many many years. And like apps can only get you partway there because ultimately you don't want to be looking down at a screen. You want to be out and in the environment. And so it's easy now to think about being able to curate something that is designed for you in terms of a route or even an explanation or a curation of a single artwork or artifact. But I also do want to kind of I think we need to be conscious as developers and creators that ultimately going out in the real world is an encounter with the unexpected and the unusual and things that we weren't necessarily ready for. And that's a good thing because that helps us grow. I think that at times we've verged on kind of the danger of only seeing what we expect or what is being curated for us on the internet. And I think that causes people's minds to close somewhat. So I'm also very conscious that as we're creating these kinds of interfaces, it's really important that we maintain that encounter with the unexpected that the real world represents.

Yeah. And I think that's a really fascinating point, isn't it? And I certainly could do with the AI guidance for this event here. It's something this enormous finding your way around. You can see that being really powerful. So we'll come back to the topic of the unexpected and how do you sort of avoid getting into a bubble but let's hear first what are you excited about?

Yeah. Yeah, I can see that. And you mentioned in your introduction about the engineering of these things and you know the real challenge of getting wearables that are also things that people want to wear. They've got to fit with comfort and all of those things. They've got to fit with accessibility. They've got to fit with fashion and looks and the image you want to present to the world. What do you see as the biggest challenges in getting wearables that people really want to wear?

Oh, lots of them. First I think so far we found actually the glasses form factor is the most ubiquitous from with regards to enabling these kind of experiences. A simple reason you know literally glasses are close to your senses your more important senses your vision your hearing they can talk to you they can see the same thing that you see they can hear what you're hearing in exactly the same egocentric way that you do it. There'll be other form factors that will emerge and all that Ziad was mentioning a bunch of them right now but this is the one we have invested the most because it's the most natural one but it's also one of the most challenging one first our face is our identity I don't know how many neurons in our brains are just dedicated to basically distinct people I cannot remember people names I remember everyone's faces isn't it and nature really designed us for that and which means it's very different than the current things that we see in consumer electronics or your personal computing even a smartphone probably put it out here we all have the same one or they all look the same suddenly you know you want to choose style and I cannot do like three styles for all the world like even the portfolio we have out today and it's the beginning we had like I don't know like over 50 different SKUs you know and it feels short. So that's already a challenge. It's just a style. So you get acceptability because essentially you really really want to personalize it. The second one is you were mentioning is ergonomics. It is essentially you want to wear this all day. So it's useful to you. It's always with you as an assistant. It really need to feel light and it needs to feel like you're wearing it all the time. So and the third thing you were mentioning it at the same time it needs to be very powerful from a compute perspective. Indeed. So it can really do this contextual understanding. It can be basically even when we bring the AR displays you know we have literally single digit count with one hand milliseconds to do what we call photon to photon. So like we see what's going on in the world. I want to put something digital on top of the world and I have four five milliseconds to draw everything and place it there. Which means it needs to happen here. I don't even can go wireless somewhere else. I don't have that amount of time. So when you combine all these things, they're going to be fighting the ergonomics. You want long battery life. You want to use it every day. That creates weight. Batteries are not improving at Moore's law or the way silicon is improving. Even silicon is not improving now fast enough for the amount of compute we want to put here. And so that's like a significant dimension just like on the device itself. And then we've been talking about the AI and that's another challenge. AI needs to become truly multimodal. You mentioned a bit that before it really needs to be models that are basically native multimodal video, audio, speech and it's not like just what you're saying, how you're saying it and all that. So it feels natural. It feels really contextual aware including to your intonation, isn't it? And then the other part, it needs to be super trustable. We talked about that. And in addition, it needs to find the right balance. It's not a good assistant that is nagging on you because now it's capable on you all the time. Don't do this, don't go there, like do that. It really that contextual awareness that we talked about needs to be very sophisticated. It needs to be at the right time that is interrupting you or proactively going to you at the right time receding and saying like I don't need to do anything for you right now. You're taking care of it. Those are the challenges and they're going to be for quite a while. We're making a lot of progress in form factor. We already launched our second generation. It's much better. You'll see new generations from us launching in glasses both styles and comfort. But the AI is probably the one that well it's evolving super fast because actually we're already doing things that we thought like we'll be doing two years from now not today. So I do think that also is going to converge quickly but all these things need to come together to have a mass adoption and then the last piece is going to be essentially price which I think as it grows we have economies of scale so it's accessible to everyone.

Yeah. So coming to you Rob and maybe this is a bit of an unfair question, but you've talked a lot about creativity, a lot about storytelling. You know, I think we're quite fixed on things we know. We're fixed on watches, we're fixed on glasses. Is there something else we need to be inventing for wearables?

I think it's worth talking now about interoperability because for good reason. It's kind of an old-fashioned buzz word, but the spatial internet, which I think is still a useful concept, that's not going to be built necessarily on TCP IP and the same kind of inherently interoperable protocols that the original internet was built on because it's being built out of proprietary hardware. It's being built out of proprietary data sets and 3D meshes of the world. And a lot of it is going to run on proprietary AIs that have been trained on proprietary data sets about the world. Like there's a data race on now to get that data to train those geospatial AIs to be really good at being able to interface with the world to be most useful to users. But equally, I think we do need to think about interoperability because if you look at again kind of an old-fashioned buzzword in some ways, all of the metaverses that have been successful and been compelling for users, things like Roblox, things like Fortnite, places where people, particularly young users, choose to dwell and spend their time, those are built on platform agnostic interoperability. And so I think when we're thinking about when I think about the kind of creative experiences that I want to be building in the glasses, if I want to sell or I want to provide a partner the opportunity to sell a cool skin to somebody in augmented reality so they can walk down the street with like, you know, golden flashes going off around them or wearing like cool augmented reality Nikes over their real shoes or be part of a game world in that way. You sell that kind of product, the user needs to know that a decent proportion of other people are going to see it. And so I think as we're developing these ecosystems, we also need to be thinking about the kinds of experiences that are going to run across hardware in these ecosystems and developing towards that so that we can create experiences that are competitive and compelling for people and that you know allow people to dwell within an augmented version of reality that isn't really kind of closed down and private and sort of encrypted.

Yeah. Brilliant. Ziad, for you, where are the biggest challenges with wearables? Maybe the biggest opportunities.

Yeah. And just, you know, open to any of you really who wants to come in on this point, what about the connectivity point? Because obviously the quantities of data that we're going to need to be transferring to and from the devices is going to be enormous. And you know, frankly, last night in my hotel, I couldn't even get 5G to work. So, how big a roadblock is that?

Yeah. And again, I can imagine that then conflicts with the weight and the wearability and the battery. So, uh,

Really interesting. The other thing that also we're going to need which has been traditionally in wireless you have a trade-off between bandwidth and latency.

And these applications actually need both.

Yeah. Okay. Well, maybe staying with you, if we zoomed our thoughts forward a decade and we're sitting here in the 2030s, where do you think these products are going to get to? What will we be experiencing?

Fantastic. Rob, for you, where are we going to be?

I mean, that's where it gets really exciting, right? Because that's where we can start talking about those kinds of ambient narrative or gameplay experiences that kind of remake or reskin your environment to be a fantasy environment or to be a gameplay environment or whatever kind of environment that the user wants to dwell in to share a game with their friends or to just make their walk to the shop a bit more interesting. It's also worth pointing out that, you know, we're at a conference and we're surrounded by big, really, really high-draw screens that are drawing a lot of power. And I think if we kind of transfer a lot of the display capabilities that we currently have to put into LEDs and put up on a wall with a large amount of power draw, if we transfer that into people's own vision onto a wearable in their own eyes, then suddenly we're able to think about reducing the power draw of like things that we currently need in order to, you know, sell products or to communicate to people or communicate traffic signals. But I think that does also raise this question of what happens if we begin living in customized realities. Maybe we'll get on to this in a second, but it's worth thinking about whether we're all looking at the same billboard, which is currently a broadcast piece of media and it's showing the same thing to everybody. I think if we take some of the power draw and we take some of the responsibility for curation of that onto the individual user's wearable, that's really exciting for the user because reality becomes more customizable and personalizable. But then are we really even living in the same reality anymore? And that raises other really interesting questions about the usability of the real world and how this technology is going to affect that that are too big to fully get into today. But yeah, it's an exciting time.

Okay. Well, let's hear your vision for the future. I know you've talked a lot about a coach and a companion that's always with you. Is that the vision or is there something else?

It's much more than that. I think as the technology converges, so we've been talking a bit like the things that need to get solved or like improved on the technology to make this a true wearable that essentially is always on with you. I think there are like three things that I'm excited about and I'm envisioning you're talking 10 years from now. It might actually come sooner than that. And it's one, we're going to give you what we call superpowers. We're going to augment your senses and I can talk in a bit about like that's already happening in some cases. Two, you're going to have super memory because now your device not in your pocket, it's with you all the time. It's basically going to help you remember everything. We'll have to adapt to it. Names. That would be fantastic. And then the third thing is every person in the world is going to have an amazing personal assistant. What some of us are lucky have maybe at work, you know, and you cannot live without or like some people even can afford it in their personal life and all that, but most of us cannot. Now you're going to have it, you know, and that could be contextual to you, you know, if for everyone it's like your personal assistant. It's always going to be like what I said one step ahead, contextual aware, helping you, guiding you, doing things for you in the background but also in the real world not just the digital, you know, but if you're a student, is this completely going to change the way learning is. You're going to be learning by doing, you're going to be coached all the time, every, and I think it's going to really basically scale up the human, you know, power and growth because right now, not everyone has the same access to education. Not everyone has the same access to being coached and teach and all that. That's going to like level the field. I think it's just that the human opportunity and the human power there is going to be just multiplied. And then if I go back to like what I said at the beginning on like getting you like super senses, we're already seeing it today. People with disabilities are already using our devices as mission-critical. If you're vision impaired right now with just the glasses I'm wearing right now and the AI is changing from like you depend on people for almost every step most of the time or everything to be super sorted in your life where now you can live with the basically what you were saying the real world when not everything is exactly as you put it and you can navigate it with like getting examples like every day we get someone saying like this has changed my life, you know, with our display glasses we're already seeing it, people with hearing impairment we're having a conversation and now I'm actually giving subtitles to what you're saying here. Now imagine that like for everyone like the superpowers that you get with regards to like now we're having here each one of us in a different language and it doesn't matter it's in real time. You're just getting us basically literally with your voice but in your own language if you don't understand that one. So you're going to get all that and it's going to be accessible and affordable. And then I was talking like super memory. I'll give you like simple examples, you know, but you need to compound them like every minute of your day. And right now people do like I don't know like shopping list when you go to the supermarket and then you have to remember it and people imagine I like which is how it is today. I put a reminder in my glasses and then I check what was what I needed to buy in the supermarket. It's not going to be like that. You're in the supermarket, you literally when you're walking by the eggs on the aisle, this is going to tell you, hey, there are only three eggs in your fridge. Yeah, you should probably pick more.

Brilliant. So, it's a very exciting sort of future that you've painted there, but Rob, you've mentioned a couple of times the ethical questions that we've got. So, what do you think those biggest ethical questions are and do you see a difference in different parts of the world around how they're perceived?

So, I think a lot of the conversation around AI right now is about privacy. I'm not an expert on privacy and that is not I mean I kind of want to project forward into a more an issue that's been harder to imagine for a long time but I think that is now coming into focus which is this issue of being able to customize reality because privacy on the internet, you know, that conversation has been going rumbling on for a long time. Identity on the internet is incredibly important. And what especially young users really care about, what they use the internet for so much of the time is being able to define and project their identity out into the world. And that the internet and social media has become this incredibly defining tool for letting people show off who they are and explore who they are and discover that sort of thing. That's fantastic. But I think as we move into a space where we can imagine lots of people walking down the street wearing glasses that can superimpose graphics into space, we can already see now demos that do AI reskinning of real places. If you want to see an example, there's an example. You can sideload onto a Quest 3 called Daycart XR. It can also recognize the human body and AI reskin the human body in real time. It's not realistic looking, but you can tell exactly what it's trying to do. And it could put me into a tuxedo right now despite whatever I chose to put on my body this morning. And so we're raising the issue now that as you walk down the street, we might move into a space where you aren't sure how other people see you, no matter what clothes you decided to put on this morning. So all the such exciting uses for XR. When I put an XR headset onto a student and I show them an application like this, this is what they pick up right away. Because if you imagine a classroom, there's so many classroom benefits to wearing glasses that could model screens. But if you're the teacher and you're standing at the front of that class and you can't tell how your students are seeing you or how they're seeing the blackboard, then suddenly we live in a slightly more fragmented version of reality at a time when I think we need to be able to get better at being able to work out what's true and what isn't. So I raise this issue of customized realities because it's a few years down the pipeline obviously. But the minute we start giving people the ability to curate their reality, we need to start thinking about what happens when they can live within custom realities because that has big socio-cultural implications.

Yeah. And I think we're already seeing, you know, in the real world at the moment the challenges of fragmentation. So it's a really interesting one. So we've just got two minutes left. So perhaps give you each one minute to give your view on the ethics and then we'll come to some questions from the audience. What's your one minute take on ethics?

Yeah. In my experience it starts from a utility. I think if it's very useful, people will adopt it and then the second part that will come with that is basically going to be social norms will develop around it in terms of what's the right way to use it or not. You were mentioning camera so I always give this example which actually dates me then. So basically when we did the first camera phone, I worked on like the second one to market, our friends at Nokia beat us by a few months. And camera phones started to come out and it was going to be the end of the world. They started passing laws. Yeah. In many countries, that was like, well, you need to flash, you need to make a sound, you need to basically warn people and all that, which is somewhat important, but it's also a kludge in terms of there's always ways to hide it and all that. And I bet you those some of those laws are still in the books and not enforced. Because what needed was like basically is really our social contract to evolve to basically create the norms. What are the right ways to do it? What are the not right ways to use it? Obviously this one is a new challenge because it's sort of like more always on, but I think we need to develop that and then also it's important as we develop these devices in addition to a lot of just by physics we're talking about the challenges, a lot of that processing needs to happen locally. If you're sending like a huge amount of data to the cloud, the uplink is your most expensive power thing so this thing is never going to shrink as a result. So there is physics also that going to take care of a lot of that thing. The other part is we need to give agency to the users. And also to the bystanders. That's what we're doing with indicators and all that where it helps create that social contract where you know what your device is doing also and the people around you also know what your device is doing.

Well, I'm sure we could do a whole other session on this topic and I imagine there probably are some elsewhere in the conference. But let's give maybe one question from the audience. Who's got a burning question for the panel? Okay. Yeah.

Real quick question. So, with the advent of AI, we're going to have a lot of folks looking for new jobs, transitioning from being a stenographer to a new profession. Please provide your perspective on how this type of technology you're working with will help humanity get educated in new ways.

Great question. Who wants to start with that one?

Brilliant. Um anybody else have a question here at the back?

Hello. I saw on the show where also contact lenses as a new human interface. How do you look about that and nobody knows you're wearing glasses and nobody knows how you are seeing the world? The customized reality you're talking about. Is it a problem?

That's not going to be a problem for a while. Because it's just the physics are going to get in the way. I think what you can do with contact lenses, we're talking still like more on the science fiction part of it. It's not going to be in the next 10 years. You know, we've been investing a lot on display technologies. One of the teams I built at Meta. I think we're the most advanced in terms of like AR both in VR, but also on the AR display technologies. AR is more challenging because it's more form factors and the see-through. There are interesting things you could do with contact lenses, but it tends to be very small like indicators and all that that are going to have utility maybe in some vertical applications, but in terms of like a general consuming and a general computer interface, it's still a while way out there. It's just and it's just physics, you know, battery, power, compute, everything, you know. Plus, it's only even when we get there and as I said, there's a lot of hypothetical that will only be just one component of the system. You still need audio, you still need input methods, and all that. I should have mentioned actually before, if you're talking about like things that feel more like science fiction, it's one of the things we've done, you were talking earlier, an important thing is sort of like what is the input modality? Obviously, these are going to be more natural input modalities for devices. We're already seeing it like with voice, with hands and interaction and all that, but our view especially for AR, for mobility, for what comes after mobile, you know, if you're wearing glasses, if you have a UI in front of you and all that, how do you interact with it? If it's just putting your hands in front, you're using like big muscles. So, that's like a very tiring. You know, for some things, it's a natural thing to do. For many things, it's a navigating UX and all that. Our bet and what we have invested on is essentially a neural sensing in your wrist and then you micro gestures with your hand. Which honestly as we advanced technology you just have to think about it and not even just do them. You can basically control the UX and that's an example of where now you were talking about wearables and the network of wearables and all that really coming together is there. I think the contact lenses are going to be very limited in all those fashions. You're going to need all the things around and as I said like the resolution you can put on displays, the power and all that. It's going to take a while. So, it's going to be more glasses. If you want to put augmented reality, it's going to be glasses.

So, we've got less than a minute left. Anyone else want to have the final word on that question?

Brilliant. Well, thank you massively to the panel. It's been a hugely exciting conversation. I am personally really excited about the potential of some of this and perhaps I don't have too much concern about the jobs because we've still got a lot of work to do to get this where it needs to go with the hardware, the infrastructure and all the exciting creative things we can do with it. So massive thank you to the panel and thank you to the audience for listening.