Live with Eben Upton from Raspberry Pi talking AI & Industrial IoT on Arm at Embedded World 2024

Yeah, we've been shipping on Arm-based platforms at Raspberry Pi for a little over 12 years now. And what's been interesting is all the way through that time, people have been finding ways to do things that you would recognize as AI, artificial intelligence, machine learning applications on Raspberry Pi. And as we've progressed from shipping a single core 700 MHz Arm 11 all the way up to the quad core 2.4 GHz Cortex A76 which is inside Raspberry Pi 5, that kind of watermark has moved up. The level of sophistication of ML workload that would fit onto the CPU of a Raspberry Pi has increased to the point now that we're getting kind of, I guess, that kind of Peta world now where 80%, we reckon roughly 80% of typical machine learning workloads will run very nicely on those Cortex A76 processors. So really, Raspberry Pi's approach to AI is kind of, I guess, as it has been probably certainly for the last five years, which is to try to capture a very substantial fraction of the workloads with the CPU. And then when you get to that 20%, that residue, the workloads which do require... So what fits well onto CPU: sparse networks, quantized networks, small networks, workloads often that are not continuous workloads. You know, frequently you find yourself in a situation where you have some other more crude detector which determines whether machine learning needs to happen, which will then trigger a more sophisticated model to do the classification, to do your high performance, high precision classification. So those kind of intermittent, quantized, small, sparse workloads fit very well on the CPU. That residue of larger models or workloads that need to run continuously, those very legitimately get pushed onto accelerator hardware. And really, our aspiration is to be the best possible host for third-party accelerators. We'll take a look at one example of that in a moment.

Okay, so this is, this is a, this is nice. What I'd probably classify as a medium, I guess, a medium difficulty continuous AI workload. So if you think that you can push maybe low hundreds of GOPS of workload on those Cortex A76s, this is a workload that would be more kind of one to two TOPS workload. Now what we're looking at here is the Sony IMX500 machine learning enabled image sensor platform, which will turn into, the fruits of our collaboration with a very long-term collaboration with Sony Semiconductor Solutions. This will turn into a Raspberry Pi product, a shippable Raspberry Pi product in the next few months. What we have is a 12 megapixel image sensor integrated into a module with a 2 TOPS machine learning accelerator actually bonded onto the back of the sensor die. So what this is doing is it's returning, it's a standard Raspberry Pi format camera module, but instead of simply returning visual data, high quality visual data, it's also performing inference on the module and then returning the results of that inference to the Raspberry Pi. And in fact, you see here we're using a Raspberry Pi 02W, so this is a 1 gigahertz Cortex A53, probably in the kind of middle of our, you know, a medium competence, a medium throughput Arm processor solution. With this relatively high performance ML accelerator attached to the front of it, here we have it running MobileNet classifier, person detector, over here PoseNet, pose extractor. Same silicon, different models uploaded from the Pi into the accelerator. As you can see, this is, you know, I think over time we'll see, we work with a number of accelerator vendors. Historically, of course, done a lot of work with Google with Google's Coral accelerators, which is USB integrated. Raspberry Pi 5 adds a PCI Express user port to the board, obviously opens up the opportunity to work with other higher performance accelerator hardware. And so I think what you're going to see from us is a, I guess, a tiered approach to AI: the idea that at the low end, low to medium end, you have CPU, then this will probably be a good example of a medium end accelerator, and then you can imagine higher performance accelerators often attached to PCI Express.

Well, the interesting thing is we've seen a couple of things going on. One, we've seen an increase in raw compute capability in the platform over time. You know, a quad core Cortex A76 at 2.4 GHz compared to a single core Arm 11 at 700 MHz, that's something like 150 to 200x increment in scalar, in conventional scalar throughput. But what's happened on top of that is you've seen the layering on first of NEON, so vastly improved vector performance, and then in the 8.4 architecture you see the addition of those 8-bit dot product instructions. I mentioned quantized models earlier. One of the interesting, I mean, I think in any of these eras of AI technology, whether it's the sort of the classic CNN image classifiers, I think we're starting to see it now actually in large language models as well, over time people start off in an FP32 world, they start off in a world where every number has to be full range, a full range 32-bit floating point number. And after a while, people realize that you can get really impressive, as long as the tools evolve, people find ways to get broadly equivalent performance out of much more heavily quantized models. The 8-bit dot product instructions in Arm v8.4, obviously the A76 in Raspberry Pi 5 is the first processor we have which has access to those instructions, gives you another kind of 4x on top of this. You know, vast increase in scalar performance, vast increase in vector performance, and then increase in performance on quantized models. And those are the things that are really stacking together to take you from, you know, back in the Raspberry Pi 1, Raspberry Pi 2 days, really very simple things, things which really only flirting with the notion of being AI/ML, to things which are really full, you know, for those intermittent workloads, really highly capable solutions.

Yeah, so, you know, where have we come from as Raspberry Pi? People know about the Raspberry Pi educational heritage, they know about our hobbyist heritage, they know how much work we've done to, I guess, get young people, we've had enormous success particularly in the UK in getting a new generation of young people excited about computing. But really, the journey that Raspberry Pi has been on over the 12 years we've been selling the products is from being primarily education and enthusiast focused to being really an industrial electronics company. So something like 70, well, that education market and that enthusiast market are still incredibly important to us, 70 to 80% of Raspberry Pis today are now selling into either what we call embedded applications, so those are OEMs who are outsourcing the intelligence component of their platforms to a Raspberry Pi solution, or industrial applications, people taking Raspberry Pis and using them in an industrial context. And this is just a lovely example of some of our, actually some of how some of our OEM customers are repackaging Raspberry Pi technology in form factors which are more suitable for deployment in an industrial context. So these are DIN rails, as you can see at the back. We have a huge number of people, people are either taking the classic Raspberry Pi form factor, the Raspberry Pi SBC form factor, and finding ways to mount that in a DIN form factor, or people who are taking compute modules and producing more customized solutions which often integrate some of the additional electronic components which are required to interface with industrial systems while retaining compatibility with the Raspberry Pi software ecosystem.

Um, as I said, I think that there is an enormous opportunity to deploy intelligence into the environment, particularly into the industrial environment. I think actually that there will always be a limit on human beings' ability to interact with intelligence systems, but the ceiling for deploying AI ambiently into the environment is very high, and it's really just driven by what has a positive return on investment. You can imagine a world in which every street light, every light switch has some amount of intelligence embedded in it. Our aspiration, particularly in the tinier ML world, because what we've not talked about here, all of the stuff we've looked at here is what you'd call big Raspberry Pi, a class Raspberry Pi running on Linux. There's a whole other world of, you know, we see an example up on the wall there, you see the Raspberry Pi Pico. There's now a whole other world of Raspberry Pi Arm-based solutions in the microcontroller class space. And I think what I'm most excited about is that those drive an opportunity in the sub-dollar space. Once you push the costs down low dollar, then the number of positive ROI opportunities to deploy intelligence in the world explodes. And so that's probably, you know, where do I hope we are in 10 years time? I hope we're looking at a world in which we're shipping not tens of millions, as we are today, tens of millions of Raspberry Pi solutions which are capable of running machine learning applications. My hope is that we are shipping hundreds of millions, billions of units which are running machine learning applications. And where, you know, if you look around, you'll see behind any objects in the world, you'll see a Raspberry Pi or a computer doing some sort of intelligent application. It's a super exciting future. I look forward to that for sure.