Regents' Lecture: Paul Jacobs, "Digital Sixth Sense"

We've got to get costs down. And I think one of the things we look at is how do we go and disrupt healthcare by coming at it from a continuously connected point of view? And I think many of you work in these areas, you get the notion. This thing that's on the screen right now is sort of a first pass of what we've been doing. It's basically a hub system. Medical devices generally have some kind of communications capability in them, but they don't rev that very often. We tried to put cellular connectivity into many of these things, but the problem is that you have to go through both an FDA regulatory cycle and an FCC regulatory cycle and that didn't match up. So what we did was we just built a hub that these different devices can talk to. And then on the back end it goes into a database that's all HIPAA compliant and so forth. So privacy is respected. But that information now can be delivered. Instead of being siloed one manufacturer at a time, it can be delivered and mashed up and goes to you, to your family, to your healthcare provider, to your insurer if you decide to do that. And we've put it both in a standalone device, so that somebody comes home from the hospital, they have their medical device and this hub, they just plug it in the wall and everything just works. But we also now have ported it into the phone and so these medical devices talk to the phone and the phone keeps track. And there's a lot of local computation that gets done. 'Cause you can imagine you put all these sensors out there, they can overwhelm the network. And so some local computation gets done, in other cases data just flows directly through depending on what the regulatory requirements are. But that system now allows information there. So if I'm wearing a continuous glucometer on my waist and my blood glucose level goes up or down, I can immediately get information. And say, 'Okay that's kind of interesting.' But I actually think it'll go far beyond that, that you'll have sensors inside your body. Pacemakers that lower mortality rates by 50% by being connected. But even more, we're doing a clinical trial right now of a sensor that's intended to be injected into your bloodstream. And what happens is when you're about to have a heart attack, cells start to die and the DNA and the various biomarkers start flowing through your body. And you can actually detect that two weeks ahead of time, that's what we're looking for. So imagine your phone rings and tells you that two weeks from now you're gonna have a heart attack. (audience laughs) Which is actually a good thing 'cause you can do something about it and not have your heart muscle die. Voicemail. Yeah, yeah, you get voicemail. (audience laughs) That's why you want to get it in real time. So seriously though there will be stuff that you will agree to put inside your body to monitor your health, that will happen. And you can see that example right there is one that many people can relate to. And I think it's something that will go on even more in the future. Another thing that we did to try and drive this vision, trying to create sort of a catalyzing effect where mobile health went from being something that's sort of science fiction to being reality. And we actually worked with the XPrize Foundation, which hopefully you know that XPrize, where the first one was launching the Virgin something or other thing up twice into lower, not orbit, but suborbital. Anyway so we did an XPrize with them on the tricorder. And so the idea is that there's 40 teams now that are competing to build a device that's five pounds or less, that can diagnose 15 disease states that are equal to or better than a panel of board-certified physicians. And the competition's gonna happen in basically three plus years from now. And I personally think it's gonna get won. And this will be a device that uses sensor technologies that are being created, both invasive and noninvasive, preferentially the less invasive the better. It's gonna use various kinds of software systems for decision support and diagnosis. It'll use back end systems, communication systems. So the idea is that in developed worlds you won't necessarily send people to the emergency room because they'll be able to figure stuff out themselves. And in developing worlds you can use less skilled healthcare professionals to go out and use this to do initial diagnosis and see whether people need to actually go for more care. The early days of this stuff is already happening. We see some of the things happening. But by doing this prize, and it's a 10 million dollar prize, we thought, 'Okay we're really gonna catalyze people, and we're gonna make this a reality.' And so the tricorder that we all saw on Star Trek, I think in three and a half years we're gonna have a tricorder. Okay the other thing for this, in this notion of internet of everything, is obviously connecting everything together. So we have all sorts of radios, cellular radios and Wi-Fi and Bluetooth and Zigbee and there's just a broad range of them. But what we want to do is make sure that we can connect up these things as much as possible so they can actually interact with each other. But what's really important is not just the fact that they can have radio communications with each other, but we want them to be able to connect with each other. And one of the problems with the internet of everything and machine-to-machine communications is often these different devices are built by companies that specialize in a particular vertical. And so they're sitting in their own silo. And even in the same companies you see that the guys that work on energy systems and displays, they're not talking to each other. They're far apart in their companies. So we wanted to create a system that actually talked across all these verticals, a horizontal layer. And it's an open source project called AllJoyn, in fact the Linux Foundation today and tomorrow is running a session with a bunch of companies that have come in from all sorts of areas. Like I said, white goods manufacturers and consumer electronics guys and obviously communications and automotive and energy and just broad ranges of companies coming in to talk about how we create some layer that cuts across all of these. And what the layer does is it allows for discovery, it allows for control, it allows for notifications, it allows for ad hoc authentication. It allows you to do these different kinds of mashups of stuff. We did the initial work because we wanted this to be out there just to cause the tide to rise so that the internet of everything can actually work and these various things can all talk to each other. But we made it an open source project 'cause we didn't want anybody saying, 'Oh well now Qualcomm's gonna come in and like try and tightly control this thing.' So it's a Linux Foundation effort. I encourage you to go check it out, it's pretty cool stuff and there's people here who are already working on this notion. And one of the things that's cool also is that there's a framework on the phone such that if you build an AllJoyn capable device, it can actually send notifications and controls to the watch. So I can control stuff from the watch which is kind of cool. The other thing obviously is these sensors are gonna create tremendous amounts of big data. And so the notion that you'll want to be able to analyze that data and do things that people want at the time, the right information at the right place at the right time. And that, in this picture we're talking about retail, but I just talked about healthcare. It's the same notion. You want to be able to look at that information that's coming in, you want to be able to compress that down. You've got to care about the power consumption of these sensors, you've got to care about the network load of these sensors. But still a lot of information is going to get crunched. Whether it's crunched at the edges or crunched in some centralized places, both will happen. But the notion that we will actually have a lot more information about the world around us, I think is probably pretty natural to this audience. But that will become a much, much more mainstream concept in the next few years. And one of the things that we did with this actually is we built a system that uses the sensors in the phones. And this is actually an image of an app that we did for Paramount for the Star Trek Into Darkness movie where you ran an app on the phone and it had various sensors. It detected when you were watching a certain video, it detected when you went to a certain place. It detected when you saw a certain poster. And as you went around and did these missions and went certain places you unlocked more content. And it was very highly downloaded. Another thing that we've done is that we've worked with some of the biggest ad agencies in Japan and they've actually used this system and found that once you give information that is localized and personalized, then they were getting three times better click-through rates on information. 'Cause it actually was something that the people cared about. And then beyond this, beyond what's in the phone, we've actually built a system of very inexpensive Bluetooth beacons that beacon out rolling code. So, if I carry one of these around I'm sending out a beacon but not a beacon that can be traced to me, because the codes roll all the time. And it's only after I give my authorization that then I can go back and correlate that sequence of codes was this person. And if I say, 'Up to this time you can know it's me, but after that you can't,' as soon as the code rolls, nobody knows who I am anymore. So we did it to make sure that we could preserve anonymity and still get the benefits that you get of saying, 'Okay if I'm willing to say who I am, I can get this offer. If I'm not willing to say who I am, but I'm willing to say I'm interested in X, I can get that offer. And if I'm willing not to do anything at all, just drop the bits on the floor and nobody knows I was there.' And this came from the time when we worked and put GPS into the phone. And people said, 'Oh it will be great. The restaurant will know when you walk by to send you an alert.' And I said, 'I don't want the restaurant knowing all the time that I'm walking around the city that I'm right next to them, so that they can send me an alert.' So this is the response to that. At very, very low power and very low cost, we can beacon out information, the phone walks through and it gets the information and it can respond how you as the user decide to respond. And if it's a notification it comes here and if I'm not interested in it I just flick it away, so it's less than a second for me to deal with these kinds of things. And of course with machine learning over time you'll filter better and better. So a lot of really interesting things going on. We're actually doing a trial with the Dolphins in Sun Life Stadium where you'll walk through the stadium, when you come in various things will happen. When you leave that stadium after the game it will actually give you information about traffic conditions and any other information about the game that you might be interested in. Obviously offers to go various places and so forth. So this stuff's actually rolling out now and it's real and it's very, very inexpensive to use these technologies because the sensor was built into the phone. And the phone was the expensive component that made it all work. And then so finally I was saying earlier, we'll have this seamless interactivity, we'll have the device on our wrist, we'll have the alerts, but we'll also have the control. So here this is an AllJoyn alarm that says, I'm an alarm in the room, do you want to control me, turn me on, turn me off, set the time, whatever it is, turn the lights on and off. These are not the real user interfaces obviously, but anyway so you can hit the button, on, off, dim whatever. Coffee pot, we've actually showed these things off at trade shows and they're kind of fun, 'cause everybody walks up and they sort of play with all the devices in the world around them. But the notion is that I don't have to pull my phone out of my pocket, unlock the screen, bring up the app and do it, stuff will actually get pushed to you in real time. So interactivity will happen in the far, far more seamless way. And so you can imagine now this notion of being able to see things around you, interact with things around you that are virtual things. But you'll actually have the notifications coming to you in a very seamless fashion. So in the end we're really focused on what's the evolution of all these different technologies? How do we drive new radios, new services, new processors, new devices that are out in the world, new protocol layers? I mean it's from top to bottom there are new opportunities to do things once you think about the world in a way where the physical world and the cyber world are combined together. And they're seamless between each other. So that's the vision that we're working on right now. I think it's been exciting for us. We've already started to do a lot of things along those lines. And like I said, because it's a vision that I articulated to the company, I'm not the one that's coming up with every single idea. I don't have to push things down. People in the company come up with all sorts of crazy, interesting ideas. So hopefully you will too. Thanks.

Thank you.

Nope, it's our responsibility to drive the costs down in the devices and to build networks and various systems that'll allow us to get connectivity to the most remote places that we can possibly get it to. So we're working on all sorts of technologies to do that. We're working on things to make the use of the technology, not just on the device side, but actually generating the bits over the air radically cheaper. We have an effort called 1,000X Challenge which is to increase the bandwidth by 1,000 times and reduce the cost by 1,000 times. And we pretty much know how we're gonna do that. So the network will be massively decentralized, very, very inexpensive and it will be able to do things like download content, be edges of a content distribution system. The thing that I was talking about earlier about this notion of physically carrying bits around, that's a thing that I think will get used in emerging markets dramatically so that the expensive part of getting from the cloud, the big network, to a device in an area, that'll happen once and be shared among many people. And then it will actually be distributed by super distribution, people will distribute to each other and sort of that's how content will go out. And that will radically decrease the costs and it will radically increase the coverage. But we're also building peer-to-peer networks so that if you're in an area where you don't have macro network coverage, you'll still be able to communicate with devices and send stuff, information back and forth. And so all these different techniques will allow us to give connectivity where maybe you wouldn't have had it normally. And then there's satellite systems that are getting built, I mean there's just a broad range of things that are going on. But yeah, we absolutely feel like it's our responsibility.

Thanks for being here.

Good they're doing their job, they're being very secret— no actually. We're not that secretive, actually.

You really don't want me to tell you the answer to that. (audience laughs) Okay, so I mean there's sort of the near term aspects of it of just people relating to each other. And I know that there's a lot of people who think that we're using these devices and being less connected to each other. But I don't believe that's true. I mean I find that using a device actually allows me to be connected to people that are far away in space. And it allows also the sense that, it's a sense of security, so it's not that I'm talking to my friends and family and colleagues all the time, but the fact that I have a phone and they have a phone. I'm not worried, I know that my son who's off traveling in Europe right now I can get in touch with him any time I want. In fact I can have a face-to-face video conference with him whenever I want and so that gives me a sense of security. And by the way that applies to healthcare, too. I mean in the future you won't think that your doctor is watching you at all time, but the fact that somebody can watch you in case something happens will give you that sense of security. So I think those notions of enhancing our ability to interact with each other, I think are incredible. And there's some amazing examples of things that happen. I mean look at the Arab Spring where people were posting pictures on Facebook of snipers on rooftops so people knew don't go there, it's not safe to go there, go somewhere else. Or they can organize and create social change. I mean all these kinds of things have created amazing opportunities for human expression and so forth. I mean in the long term, yeah I think we need to watch to make sure that privacy is maintained. Now maybe that's a generational thing that I care more about privacy than people who've grown up in an age of massive social networking. But I don't know, I mean it takes an Edward Snowden event to make people think about it. So those kinds of things, privacy and security, I think those things will be things that we all care about. And so there are upsides, there are always with any technology some downsides. I mean I think it's a Berkeley thing actually that we should be caring about what those downsides might be and trying to mitigate them and avoid them as much as possible. And focus on using these technologies for social benefit.

Yeah, no I mean it's a huge thing because, I mean the tricorder XPrize is all about artificial intelligence and bioinformatics and using sensors to do this kind of work. Because if you think about it you take your average doctor, they've been trained a certain way. But they clearly don't know everything. And even more importantly as time goes on medicine is increasingly personalized, so that there are drugs that are specifically targeted to a particular mutation of a cancer cell that you have. And then you peel the onion back. And all that stuff is done through bioinformatics at this point. And the ability to do sequencing is getting cheaper and cheaper and at some point that's gonna be a plug-in on your cellphone maybe, or at least an accessory that you might have. So I think those components are absolutely critical going forward and the artificial intelligence systems are getting better and better. I was telling some of the people earlier that we have a technology we just announced called Zeroth, which is new processor architecture which is spiking neuron models. So it's modeled after the way your brain works. And the thing does unbelievable stuff. You don't program it, you teach it. And it already today does tracking of objects better than dedicated algorithms, 'cause it can deal with different backgrounds, occlusions, objects that are very similar looking to each other, moving at high speed with other stuff coming across. And it keeps track of all that kind of stuff. And we can do various motion control systems with it already, so that notion of artificial intelligence which is different from a lot of traditional artificial intelligence. And I think that's going to be quite an interesting field going into the future. And we're building software tools and so forth for people to experiment with that as well. So there's sort of the more traditional kind of AI and then there's this new, very—I don't—new, that's probably the wrong way to say it. It's this notion of using bio-mimicry to build real systems that do sensing and interpretation. Kind of robots for your mind as opposed to robots for your body. That stuff's happening.

Yeah, I mean it really works. I mean it's a little frightening actually that it works so well. It's very biological looking.

I like PJ, that's good.

Oh cool.

All right, guy. You mean aside from getting out. Actually I can tell you my best memory actually was quite early on. I met my wife in the first week of school in the dorms and we stayed together through Cal. 30 years, 30-something years. But anyways, no it's—my brother played beach volleyball in La Jolla, and so I was only okay at volleyball, but when you come to college nobody really knows that. So I just went around to all the pretty girls I could find and tried to get them on my volleyball team. And then I found the two ugliest guys in the dorm. Sorry guys. One of them's nickname was Bear 'cause he was so hairy. Anyways, that's how I ended up with my wife.

Guarantees are hard when it comes to security of systems. But I would argue the systems aren't secure in that sense even today. So the issue is actually us putting more emphasis on—already today in medical systems you have to put redundancy and things like the different processors running completely different code. Voting and all sorts of stuff to make sure that the systems work. And I think we just need to put more and more emphasis on that. And the fact that they'll be upgradable, so there will be cat-and-mouse kind of attackers versus defenders going on all the time in all of these systems. I talk about this internet of everything. Once you start building things into buildings and the built environment, they're gonna be there for a long time, so they better be upgradable. And you're gonna have to deal with security. Obviously healthcare is an area that you care an awful lot about. But like I said, today pacemakers are connected and you get tremendous benefits from it. I mean 50% reduction in mortality rates and so forth by being connected. So there's a trade-off. And I think if somebody wants to do something bad, if they decide that they're gonna kill you by messing with your pacemaker it's because they're already a psychopath. And so we got to deal with those people and try to create societies where people don't do that. I mean those kinds of issues are incredibly serious and we put a lot of effort into it. And we're actually building technologies now to try and sort of be monitors. So they'll sit at trusted computing things that will sit aside and watch what's going on in the system and try and detect things that are happening based on the behaviors and not necessarily based on finding this particular bit of code that we've already detected as being a virus or malware and so forth. I think there's a lot of techniques. Of course every defensive technique somebody's gonna come up with the next way around it and yeah, so I mean it's cat and mouse and we're gonna have to deal with that.

I think it's good, but we'll see.

Yup, that's a great question. Okay so, first of all, one of the things that we're doing is just trying to get the cost down. So drive cost out of it so that more and more people around the world have access to these technologies. And we're doing a pretty good job so far. There's projected to be seven billion smartphones sold between 2012 and 2017. Half of those, more than half of those will go into what we'll call emerging markets. And then the question is what do you do with those? So we already have projects underway where we do things like, Egypt we have a teledermatology project where people can use the camera on the cellphone, take a picture of a skin lesion, send it back. The doctors analyze it, they send back the information and say, 'Okay here's what you have to go do.' We did agricultural projects. We had this fishing with 3G nets things where people get information, the fishermen get information on weather conditions, where the fish are, once they catch the fish they get information on which market to go to. So that they don't all go to the same place, drive the prices down. And that's actually been demonstrated, have returns. We're doing education things, we did a number of projects, actually John you did a project on teaching people English. And so we did an algebra thing, actually that was in the US, but it's applicable. Where you give kids the ability to actually teach each other how to solve the problems. And we got massive increases in standardized test scores, so education, entrepreneurship. We have a project we did with a professor at UCSD around voting fraud. There are actually patterns that you can detect when fraud is occurring in a voting place, so they could actually go in and figure out which voting places were fraudulent and deal with that. So doing governance kinds of issues. We have a program called Wireless Reach which does these kinds of projects for social benefits around the world. I think it's 200 projects or something like that in 40 countries. It's a large number of projects around the world. It's actually—we do it because it motivates the employees tremendously. I mean people want to know that they're part of something bigger than themselves and that they create this amazing technology sitting in—half of them sit in beautiful San Diego. I mean yeah, we have people all over the world, but the notion that you can take the advantages that you have and create opportunity for other people is just a very motivating thing. I think it's a lesson we all learn from being in Berkeley that we're part of a bigger world. And so using these technologies, driving the costs down as much as possible, working with partners who are in these markets so that we actually know what the needs of that market are instead of trying to use our first world notions to create solutions for that. And then the other thing that's very interesting, it's kind of like the boomerang effect, is that some of the technologies that we created for emerging markets, like how do you build a $20 cellphone? Well once you build the chip that will make a $20 cellphone for sale in India, that chip can come back to the United States and be something that goes into a wireless glucometer. The boomerang effect happens that developing solutions for price-sensitive markets actually is important as a way to get technologies back into the United States. And I would say on mobile health it is absolutely the case today that more mobile health work is being done outside the United States because of the regulatory burden of the United States and of many developed countries. You're seeing a lot of stuff going on in the Middle East and Southeast Asia and Latin America and Africa. I mean you see all this stuff happening because it can happen in those places. There aren't as many entrenched interests and so forth. So yeah, I mean we see that as a huge, huge responsibility of the company.

People can, they have connectivity and they have an amazing amount of compute capability. Even in the lowest-end cellphone you have an amazing amount of compute capability that you are now proliferating around for very inexpensive amounts of money. Yeah it's a lot of money relative to the income levels, but you actually find people prioritize communications above many other things in their lives. And there's a reason because it improves their economic condition. And then they can live a better life. And we've seen that already happen. I mean there are studies that show based off of some of these programs, that people's economic condition and their standard of living goes up. And so that's why we're excited about being part of what we say is humanity's biggest platform. Because it does allow you to have impact in all walks of life, in all areas of life.

That was a good question. That was a great question.

All right. (laughs) It's a rugby shirt.

Go Bears.