Chief Development Officer, Marvell Technology
Road to Chiplets: Architecture - Sandeep Bharathi: Data Infrastructure and Chiplets
Road to Chiplets: Architecture
Data Infrastructure and Chiplets
Sandeep Bharathi
Marvell
Explosion in data generation, transmission, computation and consumption has made hardware and Silicon “hot” again. This has coincided with another inflection point of slow down in Performance, Power & Area (PPA) gains with decreasing process dimensions. As a result of these two diverging phenomenon, there is an urgent need to look at architectures and implementation strategies to scale up to the market needs of designing HW for the data infrastructure market. In this talk, we will look at why “monolithic...
About Sandeep Bharathi
Sandeep Bharathi, Chief Development Officer at Marvell Technology, discussed the company's approach to AI-driven data infrastructure and semiconductor innovation during a December 2023 webcast. He stated that AI depends on compute infrastructure and data movement, and that Marvell is addressing bandwidth challenges through evolving signaling mechanisms, such as moving from NRZ to PAM4 and coherent signaling. Bharathi noted that Marvell announced the 1.6 Tera Optical DSP Nova and the 800-gig coherent DSP Orion, and described a 51-terabit switch with 512 lanes of 112-gig SERDES. He characterized Moore's Law as an observation rather than a physical law, and said techniques like 3D stacking and advanced packaging are used to maintain progress. Bharathi emphasized that Marvell's investments span standards work, foundry technology, EDA tools, and applying AI in chip development, and that partnerships with customers, vendors, and foundries are fundamental to their strategy. In a July 2021 talk on chiplets and data infrastructure, Bharathi argued that the explosion in data generation has made hardware and silicon "hot" again, and that chiplets are needed due to the prohibitive cost of advanced geometries and the slowing of technology scaling. He advocated for standardized interfaces and modular architectures to achieve cost effectiveness, while noting that modularity involves trade-offs, such as latency considerations. Bharathi stated that not every product requires a chiplet approach, but that it is becoming imperative for most compute and networking segments. He also discussed the role of thermal management, the move from copper to optics at higher data rates, and the need for collaborative industry approaches to solve challenges like power and packaging.
Transcript (49 segments)
H
Host0:11
Of central engineering system on chip group. In this role he oversees the development of SoCs for various end markets in advanced process nodes. He's a recognized expert in digital and mixed signal design, IP, and product development. Prior to joining Marvell in 2019, he was at Intel as Vice President of Engineering, where he led the FPGA product and technology development. Prior to that, he held various senior engineering leadership roles at both Silicon and Advanced Micro Devices. He's a long-term industry veteran, a proven innovator, and has brought multiple CPUs, GPUs, FPGAs, SoCs, and custom designs to market, which together have generated billions of dollars in cumulative revenue.
S
Sandeep Bharathi1:10
To present. Yeah, good morning. Thanks, Ravi, for enrolling me into this one, and thanks to all my fellow speakers as well. It's a great honor and my pleasure to be here. So let's get right into it. I think there will be a lot of commonality in the first few slides, although John and I never talked about coordinating our slides, but it'll be good. So one of the things that my talk will focus on is what is the role of chiplets and how do we look at the opportunity that it presents.
You have to transmit it from off chip. Compute and actually store has made hardware and silicon hot again, and maybe air quotes in more ways than one. This has also coincided with another inflection point of slowdown in PPA (performance, power, area/cost) with advanced process geometries. As a result of these diverging phenomena, there's an urgent need to look at not only end markets but also different architectures, and that's where chiplets come in. How do we implement these to scale up to the market needs of designing hardware?
Certainly in advanced applications in AI and such, we look at why monolithic-only solutions may not suffice. And Jan eloquently stated this: we are at a very important starting point. Our chiplets are here to stay, and we will look at the opportunities that they present. But certainly, what I think will require a collaborative approach to solve the challenges across the industry, especially in looking at standardized interfaces instead of proprietary, for example. Why not only monolithic? From a simplistic viewpoint, it is...
They may not all be ready with the schedule. Hence, you also need to take a look at the cost structure of a particular product and all the functions that you need to put in a particular product. The reticle is starting to get too small. So what does that mean? I think Jan touched upon this, but here are some specifics of what yield indicates in terms of die size. There have been many publications on this. When you introduce a product in an advanced geometry...
In order to have a sustainable path to cost effectiveness, modularity of architectures allows you to go to the right side of the spectrum where, in the picture, you would see that for smaller die sizes, you can get better yield. That is an obvious thing, but it's very easy to say and hard to do when you have to look at how architectures need to scale to a certain die size. So modularity also leads to better validation. Validation is a big cost driver too for complex chips through standard interfaces. Of course, that also means we have to solve the...
What point in a process lifetime would you introduce a die, and what size it should be, and how do you really map your architecture to be modular? Now we'll take an example of a Marvell switch or a switch for any particular segment. If you look at the center, you have a lot of the I/O high-speed serial I/O SerDes, but in the middle you actually have two or maybe three slices, and I would call that chiplets, because the functions that you need...
The center slices. Not only that, what happens is with that you can have the cost curve: smaller chips, better yield, lower cost. And when you look at either a 7 nanometer, 5 nanometer, or 3 nanometer and going forward, the tape-out costs are also increasing tremendously. With a combination of chiplets, it's going to become a package design play, and you can introduce multiple skews of the products at a lower tape-out cost on top of a validated chiplet strategy.
That modularity is not free. We do need interfaces. Standardized interfaces are better. For example, what happens is if you have a chip which is too big, then you want to slice it such that architecturally they are possible to be on two smaller die. We will look at how that fits in with the cost equation, but then you need to have a chip-to-chip interface that requires duplicating the interface. So we need to pay attention to the power consumption, but also what kind of chip-to-chip interface you need, whether you need a high bandwidth, high number of...
Chip-to-chip interface for various end markets is going to be critical to choose. So let's take a look at chiplet area versus cost consideration. If there's a large die reticle size of 750 square millimeters and you need to put the same function and split it, then you can do that with two smaller die at much lower cost per square millimeter. This is an example of a product that we have. You can afford to add the interface now. The question always comes down to how do we manage power, and there is some cost that...
Now not only are the die costs, but the package costs considerations are also hugely important. This is just a different view of what is the OSAT cost versus the number of I/Os and the build-up layers that you may have. If you need to take a look at the escape of the bandwidth of the I/Os that is necessary, you will really need to increase the size of the package. Somebody asked this question: how big are the packages? The packages are getting...
For build-up layers of 8 to 8 or 7 to 7, it's getting prohibitive. And now you have a new vector that has been introduced in the last one and a half years as a supply chain situation. I'm sure we will solve that in a few quarters or a couple of years, but then do you go 8 to 8 to even more? How does that scale? And then there's the optics piece of it. The explosion of data to get in and out of the chip has also increased, and we have to focus on channel loss. Materials become a big deal.
So then let's take an example of what the next generation network switch looks like. If you look at this, in order to get more bandwidth out of it, your SerDes is also tapping out. You have obviously 112 Gbps. You can get the average chip bandwidth higher. So if you want to go from 25 terabits to 51 terabits and more, you can go from 56 Gbps to 112 Gbps and 224 Gbps, but they have their own challenges there. It drives greater than 90 millimeters package size, but the demand for data explosion and the...
Really the end markets are also a huge inflection point, and that's why we are scratching the surface today. There is still a lot of innovation that needs to come, and we have work to do. So let me give a test case example of why we need to look at interfaces as well. If you want to do 50 terabits on one chip, you have the practical problems of shoreline or the chip edge. If you have a long reach SerDes which needs to be at 30 dB insertion loss and beyond, and most people require different configurations as well. For example, do you need to have a PLL per...
Can't beat it, right? So if you are doing double row of SerDes, then yeah, sure, you can look at 35 terabits as you can see in the curves here. The green and the blue curves you can get to with double stacking. Then there is the escape that you need to work through that increases your package. So obviously now you're going up in the build-up layers. So how do you solve this? We may have to solve it with a different standard. So let's say we don't do long reach, we do XSR (extra short reach), that's also a standard. But then you need to take a look at if you want to do a 50 terabit...
It's by 8 or by 16. But at the end of the day, we do need to take a look at multiple other things. So where I'm leading with this is there is the cost situation, but there are other solutions that we need to keep in mind as well. Hence, let's take a look at why we do this. We've gone through that chiplets are needed for the cost curve, there's the package complexity, and also in order to do this you have to overcome some of the readiness. Technology scaling, as we all know, is not going at the same rate and pace as it used to go 10 years ago, and that's obvious.
So one of the things that people do, and this has been well published as well, is can you do your mixed signal hard-to-design IP which is not scaling in an N minus one node? That could be in 10nm or 7nm, whatever it is. And then you do the core, whether it is your CPU, graphics, FPGA, switch in a core that is faster, to get to the market. And use a standard, maybe it's an AIB, maybe it's an XSR, maybe it's whatever your proprietary interface is. That allows you the total cost of product development lower because you...
Either so then you look at what are the solutions for integrated optics. Then you have silicon photonics. The future of scalable systems is really getting high bandwidth die-to-die interconnect. You can do chipless base switching, but where it is coming to roost is really enabling your next generation optical interconnect. Silicon photonics is also another area that is going to be here to stay. You need obviously proven optical and DSP technology, but there is enough research going on out there in terms of products as well in compact...
The problem is to get data movement from one place to another as fast as possible, as cheap as possible. That's where solutions for integrated optics come into play.
H
Host18:24
Would you mind if I break in and ask a question?
S
Sandeep Bharathi18:27
Absolutely.
H
Host18:28
What is your opinion of thermal management, and how do you think thermal management will play, and what is the power anticipated here?
S
Sandeep Bharathi18:35
Yeah, so the thermal management is absolutely important to solve. However, it also depends upon huge data centers. The eco footprint is more important for power and thermal management. So you can go from 100 watts to 200 watts depending on...
And there are specific power envelopes and thermal budgets that we have to hit, otherwise we have no product. So that's absolutely an important part. As you look at architectures, what innovations need to happen? And now going from copper to optics allows some of these managements to be done differently, but has other problems to solve as well. Alignment of lasers, modulation schemes, and the process technologies that are necessary for embedding lasers. All these come into...
Higher data rates and distance drive the move from copper to optics. That has its own set of innovations that we need to do. This is just a slide that we have from Cisco's where data rates are driving the move to optics and the distances that need to be covered. So in essence, the future carrier and data center switching is an important market. You will have to, and you ask about thermal, then there is doing air cooling or liquid cooling. What's the cost of that as you do integrated co-package optics?
Independently, and that's why I am a big proponent of standardized interfaces and standardized modules. That is really a place where I think there are many forums where we can discuss this. So in summary, chiplets and modular designs are for multiple reasons. Again, having said that, not every product needs to have a chiplet. There are people who ask me that question constantly, and it's not really necessary because not every solution has to be the same peanut butter across the board. However, given the cost of development in...
Advanced packages and chiplets provide that. Co-package optics is the next frontier to cross. With that, I thank you for the time and am happy to take any questions.
H
Host22:24
Sandeep, thank you. This was an excellent talk. There are a few questions in the chat window. Let me walk through them. The first is: you showed a Marvell switch. Was that implemented on two different technology nodes?
S
Sandeep Bharathi22:37
Yeah, so in that picture that I showed, there are in the center actually two or three of the slices that are on the same technology, but you can see the periphery was on a different process technology. That allows you to mix and match and...
That's the way I view the question. But you can harvest the periphery for a different product.
H
Host23:16
Yeah, no, that's absolutely right. That's another way of harvesting. Then there were two other questions. One is: how far, if it will ever come, do you think we will integrate optical waveguides and laminate substrates?
S
Sandeep Bharathi23:32
I think there is a lot of research to be done still on that one. It'll take some time.
H
Host23:43
Nice. And another question was: is there a specific team at Marvell who's working on chiplets, or how is the work distributed in Marvell?
S
Sandeep Bharathi23:50
Yeah, so it's really product specific. I think we have more of a...
Together to figure out the core of the product. So it is not a single monolithic team; it is a distributed team, just like chiplets.
H
Host24:26
What is the major standard you are interested in? Open HBI, AIB, or UCIe?
S
Sandeep Bharathi24:33
Yeah, so we have looked at a bunch of interfaces. We also, due to our Quantia acquisition, they had an AQ link, and we looked at that. We also support a standard interface like XSR.
You need to take a look at it, but today we are looking at standard interfaces.
H
Host25:18
Got it. One more question: can you provide some examples of how chip-to-chip interfaces affect package technology considerations?
S
Sandeep Bharathi25:27
Certainly. For example, if you look at an AIB or one of the high number of connectivity where there was more silicon interposer, a lot of connections. So if that is the chip-to-chip interface, then you have to figure out a different package technology, build-up layers, etc.
I think what happens is your choice of the chip-to-chip interface needs to be mapped to the cost of an equivalent package technology. Because at the end of the day, where I look at it is: are we delivering the right performance and power for the right package cost? That becomes a bigger and bigger deal. And now we have the more important and present problem of: if I choose the package technology today, do I have the substrate supply in two years' time?
H
Host26:48
Yeah, absolutely. Another question we have is: what kind of die testability features have you needed to implement on silicon to guarantee...
S
Sandeep Bharathi27:10
Decide on functional testing and what is functional loopback testing versus ATPG on the digital side? All I'm saying is there are specific test structures that we build and test features in both the digital as well as the mixed signal interface. Because if you look at it, that is one of the bigger reasons for chip re-spins these days: not getting the interface right and not getting the actual yield testing. So without going into much of the details on what techniques we've used, I think we have a big focus on that piece.
H
Host27:52
Another question we have is: when do you see CPO...
S
Sandeep Bharathi28:11
Our acquisition of Inphi, we already are doing CPOs. In that, we have the lasers and TIAs and drivers, but they're not all on the same piece of silicon obviously. They are in package in different MCMs and modules. It's at module level. We are going to make further modifications in our next generations of chips, and CPO will be there. It will be starting off with uni-directional but lead to bi-directional.
H
Host28:52
Got it. I'll squeeze in one more question, and there are many here. What is the latency impact of...
S
Sandeep Bharathi29:11
Techniques where you can hide the latency, not all the time. So we have to really look at where the latency is acceptable and where it is not. In some CPU applications and depending upon the interfaces, in memory maybe the latency is not acceptable, but in serial interfaces it's okay for a networking application. But it's really, there is no free lunch. It's all about trade-offs.
H
Host29:48
Yep, no, that's fair. There are other questions, but we are at times. Thank you very much. This was a really, really educational talk. Maybe we can figure out a way to answer questions offline. There's...
Possible. Adventist rated the best ATE company 2021 in the VLSI customer satisfaction survey of 2021. Amkor, as we heard an excellent overview from Dave as to their work, but as the leading OSAT with differentiators in the area of technology, quality, and service. Synopsys with their vision for supplying solutions from silicon to software. With that, I will thank everybody for joining us today and once again thank you for your attention and your participation. We look forward to seeing you again.