Please welcome Jim Keller.
Clicker. We have a clicker. I got everything but the clicker. A... that's a sad story. You know when you wake up and you're really excited and you're worried you forgot to put on your pants? I forgot my clicker. It's a little different. Same topic. So, welcome everybody. I really appreciate you guys coming out. As you know, we're starting to release performance benchmarks, and our investors said, 'You really need to get on these pages.' And we had the best superhero landing on a page ever. Go check out Artificial Analysis. Our video stuff is unbelievable. And we have a pipeline to keep winning benchmarks by a lot. So it's really great. But I want to tell you a little bit about that. We have a lot of material today.
Did it click? Ah. So, first, this is our Galaxy box. We're now shipping. JJ, are you here? I just saw you. JJ, does anybody know where JJ went from? Quanta? Over here. JJ, thank you very much. So, JJ is our partner. He's our partner at Quanta, helped build this server which is now in production, shipping, and volume. It's unique because it includes SRAM, DRAM, compute, networking that lets us scale this up, and we are now running what we call Supercluster 36. It's 36 of these Galaxy boxes tied together in one seamless computer. So, thank you very much. GD, are you here? GD, could you stand up? So, the chip inside here is Black Hole. GD was one of our partners to help build that chip. We are shipping a zero silicon. Thank you very much, my friend.
By the way, that doesn't mean there was no bugs. We have software. We've programmed around this beautiful thing. All right. This is our quiet box. So, I want to build AI that's low-cost, scalable, right? So, we build a server. We can scale up to the fastest computer ever built, actually, technically. But we also built a small box, and this is quiet because when we sent our first small boxes to a couple developers, they said their partners complained they were too loud and they had to turn them off after dinner. So we made it water-cooled. And then that one was too big. So then we made it smaller. So we have a new one for sale now. And this is a great developer box. 128 GB of memory, good AI performance, runs any model. You can plug it in your living room and it's fine. There's demos upstairs.
Click. All right. So, this is just two years. I was looking at this and I first thought, wow, there's been a lot of change in the last couple. It's just two years. Does anybody even remember Llama 7B or 7B? Does anybody care? Well, technically we care. We run everything, right? And now, you know, the big fad is disaggregation, special-purpose hardware, SRAM. Do you know how many people are going to be talking about that in two years? None. Okay. But why is that? So performance really matters. Change is happening. AI is accelerating, right? So if you're building AI computers, you have to think about this really hard. And Tenstorrent has always had a different mission than some people. We are not trying to be a narrow provider. We're not trying to be a point solution. We want to solve a lot of different problems.
So, how do we do that? All right. We always thought about scale. We put 16 100-gigabit Ethernet ports on our Wormhole chip, 12 400-gigabit Ethernet ports on our Black Hole chip, and we made sure our software stack goes from a single tensor processor to a chip full of them to a box full of them to a whole bunch of servers. And that's starting to really pay off, right? General purpose. This means you don't just run part of an LLM, right? That's not a good business plan long run, right? You could be one-shotted by one model. So we have always worked on inference and training big models and small ones, image, video, LLMs, the full stack, and then we build a compiler on top of that, and Stan's going to tell you today about how we run any model.
And finally, people really care about cost, right? So what does low cost mean? Well, first of all, you have to think about low cost when you build your product. That means you don't put four dies in a big expensive package with a silicon interposer. That will never be cheap, right? Our Black Hole chip is 6nm. We use GDDR6. We use standard Ethernet. It's air-cooled. Our next one, we have customers requesting water-cooled. That'll be an option, but we design for lower cost. We think that's important.
All right. Now, in terms of fundamentals, does everybody know that AI hardware design is easy? You raise your hands. No, come on. It's true, right? Well, I mean, if you do it wrong, it's not that easy. It can be hard. All right. So, this is our tensor processor. So, we put five little RISC-V processors in there. That makes it very programmable and very adaptable. It has a very competent matrix multiply unit, vector unit, and local SRAM, right? And because of that, we can do all kinds of programming in one mode. One processor gets data, one formats it, one does compute, one reformats it, one puts data, and it's attached to a very high-bandwidth NoC. So our LL layer and TT-Metal is where we program the fundamental tensor processor. And you'll notice there's no GPU in there. There's no SIMDs. There's no register file. There's no history of computing. This is just a modern tensor processor, and technically it's fairly simple.
And then we put an array of them in a chip. So at TT-Metal you program the tensor processor, at TTNN you program groups of processors. Yes, Mina will walk you through that in a little bit of detail today. And then when we go from one chip to two chips, we take the same exact data movement semantics, transport it on Ethernet, and the actual code doesn't care very much if it's on the same chip or a different chip. It's just further away. Technically, there's a different cost function, but it just works.
We want to run anything, right? So starting at the top, what is anything? PyTorch, TensorFlow, CUDA, ONNX, Triton, we just added PDL, right? We also have a CUDA path. So what does it mean to run a model from PDL? Well, in the world of agent coding, if you have a proper AI model description, you run it through that and either make a PyTorch program, which our compiler will take care of, or you write to TT-NN lang, which is a proper DSL. It's very flexible, right? We actually converted an AI paper which had a proper description of a model into a model that ran. There's no code involved. So Stan's going to tell you about how our compiler basically runs anything. And then we do layers of software. Metal is for hardcore people who program low-level tensor processors. TTNN is where we build big op libraries. That's where you'd see the thousands of operations that PyTorch uses. TT-lang is new for us, but it's working out great. It's a proper DSL and lets you describe at a higher-level language. It's both machine-readable and writable, but also human-readable and writable. So, we have a proper software stack, simple tests for processor arrays that lets us scale.
Here's results. We're starting to post these. This is super fun. Did anybody watch GTC where you got educated that running big LLMs today can be complicated? There's a prefill and a decode part. The decode part, with some solutions, you know, you need lots of DRAM for KV cache, accelerated computing. That means you need networking. So since our chip has DRAM, SRAM, and networking on it, we run the whole stack on the same chips. The performance comes from scaling. So that 350 number is on 16 Galaxies. It's very stable, runs very fast. We can disaggregate if we want. 4 seconds first time to first token is pretty good. And remember, this is 671 billion, large context, batch size 32. This is not a small model. This is not a narrow benchmark.
And then on video, this is amazing. I've been thinking about real-time video. I have this project I call Book to Movie where I want to take my book and put it in and make a movie out of it. So you need real-time video, continuous prompting, and we're going to show you some demos today about how that works. So simple, full-stack generality, lower cost, and now we're delivering the performance thanks to our partners. So, click. And I love this wall of pictures. AI is a beautiful thing. All right. So today you're going to hear from Yasmina and Stan, and then Ammer runs our BizDev and sales guys. He's got some of our partners and customers, and we're going to walk you through that, and I'll come back at the end. Thank you.
Please welcome Yasmina Vasilovich.
All right, I got the clicker. We're ready. So today I want to tell you a little bit about what we've been cooking here at Tenstorrent the last few months. Some new things that you haven't seen from us before. Some pretty cool demos, models, and performance. They are built on some old things, actually, some existing things. So, our architecture, as Jim was describing, our software layers that we've sort of fine-tuned and crafted over the last few years. So, we got something old, something new. I'm pretty sure there's something blue in there. But in case you don't know me, I'm Yasmina. I run the AI software team. I always start the talk with the software stack. It's like a professional deformation, I think. But Jim walked through it, and you guys have followed us for a while, so you're familiar with the layers. We have Metal at the bottom where we address and solve the data movement problem for AI. Turns out that AI is all about data movement, which is pretty exciting. So we're pretty happy to lean into that. And then TTNN that allows us to code in Python and launch ops on a supercomputer, TT-lang, and Forge compilers and Python kernel language.
Now this is our Black Hole device. It's essentially a two-dimensional torus of Tensix cores. So they execute compute. LLK APIs allow you to write kernels. Kernels run on RISC-V processors. With the LLK APIs, you can do any kind of math imaginable. So, it's a completely general, fully general AI machine. Now the data movement part is a lot more interesting because there's just so much of it in AI, and to unlock the best models to run on our hardware, we have to make sure to solve the data movement very thoroughly. There are two key things. So the cores in Black Hole, they are independent. Okay. So this is not a giant matrix multiply machine. We didn't just throw FLOPs at the problem. It's not a brute-force solution. Every core propagates and progresses at its own pace. The kernels that run on it move data freely. And they're very, very flexible. So every core has SRAM that's local to the core itself. So the data movement kernels can move data between the cores themselves, between two SRAM cores, two Tensix cores. They can move data between DRAM cores and Tensix cores on chip, and then of course they can push them into Ethernet where they get propagated off-chip and to other devices. Addressing the data movement problem is critical. Like when we see models now, we get really excited. DeepSeek V4 is a really good fit for our architecture. Anything fancy that people want to do with ML and data movement is a perfect fit for our hardware.
Putting a bunch of Black Holes together gives us Black Hole Galaxy. So, Black Hole has a tremendous amount of Ethernet bandwidth on it. Basically, it's this device that really just wants to collaborate with everybody and talk a lot, which gives us our scale. So, we put 32 of these guys together, 4 by 8. The Black Hole has Ethernet connections that allow devices to connect directly to each other, and then we can cable them up. So the rightmost column connects to the leftmost column, and the topmost row connects to the bottommost row with cables, and that's how we achieve a 2D torus within the Galaxy as well. Now the red links, they go out of the Galaxy. So they allow us to connect multiple Galaxies together. So as we continue to scale, the sophistication of the data movement levels up as well. So one single node there is essentially four Galaxies put together in a quad like this. And then we cable these guys up in a torus. So you have a single quad that has a torus, and then we put toruses together to create a supercluster. And in the supercluster we use those red links that leave the Galaxy to do an all-to-all connection, so that you have essentially nine quads that are connected in all-to-all topology. Now the key thing here is that there are no Ethernet switches in the picture. All the data is moved through the Galaxies themselves and through simple Ethernet cables, which is how we achieve our effective cost as well as scale. And it's really, really critical for performance because that's what gives us our low latency and high throughput. So the real question is how do we run applications on this? So here is a simple example. We can put decode there on the left in blue across 16 Galaxies. So a single instance of decode, what you can see in our console in our cloud today, it runs across 16 Galaxies like that. There's two instances of prefill up there. So each prefill that runs at 4 seconds, it runs on a single quad, and then we can put video generation. So you can mix and match. And the really cool thing is that they don't have to be beside each other because everything is beside each other. It's connected all-to-all.
And one more thing, the scale doesn't stop there. So Galaxy has a tremendous amount of Ethernet ports, and if we take out a single quad, as we show here in the middle in blue, if we take it out of the compute configuration, we can use it as a switch, and it becomes a gateway to other superclusters in its neighborhood. So again, no Ethernet switches, all the traffic goes through Galaxies and is programmed by our fabric. So in TT-Metal you write kernels and CCLs that move data between the clusters, between the Black Holes. They run on the Ethernet cores themselves. And then in TTNN you can invoke operations with a single line of Python code that execute across the entire supercomputer. All right. So the real thing that I want to show you is our benchmarks and our performance. So, let's cue up a quick video, and after that we'll take a look at how they perform.
All right. Everything you saw runs on our cloud, and it all runs on Galaxy. And I'm going to show you how you can log into console and try it today, actually, from your phone, from your seat. The demos are pretty cool, but the real question is, you know, what's the cost and can it be deployed? So, you know, we have this saying, and I think you guys know it, like it goes something like: in theory, theory and practice are the same, and in practice they're different. Right? Now we know in practice, if we look at top providers from Artificial Analysis there on the bottom, serving TPUs and GPUs, those are the speeds at which they're serving DeepSeek right now. We know that NVIDIA has a 300 token-per-second version of DeepSeek, but why is nobody serving it? There's a missing provider there. At Tenstorrent, what you can try in our cloud, in our console, is our 350 token-per-second-per-user DeepSeek, and we will be serving that. As a matter of fact, we'll be serving decode and prefill at 4 seconds for 100k sequence length.
The real question is why, and to answer that we have to take a quick look at token economics. So classic graph with TSU on the bottom and cost per million tokens on the y-axis. Now this is the NVIDIA curve, the NVL72. In this region here is where mainstream service operates. Why? Because the costs are low, right? So if you're a provider, you want to run DeepSeek, if you have a bunch of GPUs, that's how much it costs you to run them. That's your cost. Right? So lower is better. Right? Now, this part of the curve, this is poor economics, right? What happens here is that if you want to get to 300 TSU with a GPU, you actually need to lower the number of simultaneous users that you're pushing through your decode. So, you're lowering concurrency, you're lowering the utilization of the GPU itself, which is where the poor economics come. That's simply how GPU architecture works. Tenstorrent operates on this curve. So we nicely extend the low cost. We serve 200 today on our cloud. What you saw in the demo you can try ran at 308. We have the 350 version that will be released in two weeks. And then we march on to 500 TSU, which will still be at a cost of $6 per million tokens. And that's total TCO.
So what's next in terms of LLMs? Our current hustle is in the big cluster of three there. So DeepSeek V4, GLM, and Kimi. Those are immediate. You'll be seeing those come out from us in May. And then a quick proliferation through the rest of the models. We are committed to crushing everybody at everything. Simply put, we have some of the smaller models that run in the thousands of TSU. So that's the mission. We're pretty excited about it and can't wait to show you. All right, let's switch gears a little bit. So, we're going to talk quickly about Video Gen. I want to introduce one of our really special partners, two co-founders of Prodia, Mikuel and Monty. Please welcome them to the stage.
Thank you, Yasmina. Thank you. Hello, everyone. We're Prodia. We're the world's fastest AI image and video generation API. You know, back in March, early March, we hit the world record for Wan 2.2 video. Everyone else, all the other providers were around a minute, minute and a half. We got to 22 seconds. Then we were very lucky to partner with Tenstorrent to take it to the next level and improve performance 10x, verifiable by the benchmark. We are now the only media inference provider that has gone multi-silicon, and we've done it with Tenstorrent.
Taking Wan 2.2 from 22 seconds to faster than real time was not just a hardware swap. At Prodia, we've spent years optimizing the full inference stack from routing and serving to techniques like quantization and caching, and now model distillation. Tenstorrent has proven to be the perfect target optimized for inference, not just scaling through brute-force GPU. Porting the model was also extremely easy, and we look forward to integrating more with Tenstorrent. We're not just beating them, we're crushing them. Thank you, everyone. Thank you. Thank you.
'We're crushing them' is like official Tenstorrent lingo. All right. Thanks, guys. We love working with Prodia. All right. One more thing. So, you can go to console.tenstorrent.com. You can do it on your phone. You can log in with GitHub or your Gmail, and you can try these demos that we showed you today. You can try DeepSeek Conference. You can run video that generates in 2.4 seconds on a Galaxy Quad, and it all runs on Tenstorrent hardware. So, in the age of AI, we set out to build an AI computer, and an AI computer needs to run everything, and this one does, and you can try it on the cloud. So I'd like to say confidently that Tenstorrent understood the assignment. There it is. Thanks.
And now our first panel: Deploy in Depth.
Can you tell me about the Equinix Distributed AI Hub?
The Equinix Distributed AI Hub lets enterprises run secure agentic AI where their data and workloads live.
And who are the partners involved?
Equinix is the data center layer. Tenstorrent builds AI accelerators. Orion VM is the cloud orchestration layer. And BetterBrain is the agentic AI application layer. Together they form a full-stack sovereign AI platform.
Thanks. And what sort of ROI is enabled by this stack?
This full-stack approach enables 100x efficiency gains in enterprise processes and workflows.
Great. Well, thank you guys. Let me introduce myself. Omar Alashmei. I'm the Vice President of Strategy and Business Development here at Tenstorrent. I'm excited and thankful to have some of our partners with us today. Why don't each of you introduce yourselves?
Yeah. Hi, everyone. My name is Shang. I'm the CEO and co-founder of Orion VM. We're a cloud infrastructure provider that's worked together to bring this solution together. Mhm.
Hi, I'm Abishek Brigava. I'm the CEO of a company called BetterBrain. We're an AI application layer company.
Love it. Hey guys, I'm Justin Aion. I'm Director of Partnerships and Ecosystem Development for AI Strategy at Equinix.
Right. Great. Great. Happy to have you guys. Great. So, why don't we start. Justin, tell us what is the Distributed AI Hub that Equinix has just launched, and how we're working together, all of us.
Yeah, absolutely. It's a pleasure. And just for folks that aren't familiar, right, Equinix is a global data center company. We operate 280 facilities across 75 metros and in 35-plus countries. And so our Distributed AI Hub framework is really a thought leadership model of bringing inference closer to users, data, and models, right? And optimizing for price, for performance, for guardrails and governance. And that's why I'm really excited about how Tenstorrent, Orion VM, and BetterBrain kind of snap into that function.
Yeah. Great. Great. Great. And Shang, want to talk a little bit about how the orchestration works and how it's all integrated as part of this whole AI Hub?
Yeah, sure thing. And I just wanted to start by thanking everybody here for inviting us to be part of this. It's been an amazing day so far. So, thanks everyone for coming as well. So, Orion VM, we're a cloud infrastructure orchestration and platform company. We've built a really innovative private and sovereign AI stack. And what we do in this partnership is we take all the amazing building blocks that each party has made. For example, really highly secure and performant data centers with the best ASICs we can find with Tenstorrent, and bring it all together into something that's consumable by enterprise. Enterprises today really are looking forward to be able to consume something turnkey, and with all three of us, we're actually bringing together a solution that providers like BetterBrain can take and build solutions on top of.
Great. Great. Abishek, why don't you talk a little bit about agentic AI and how it solves the problems that customers are looking for?
Yeah, absolutely. So, BetterBrain is the top of the stack. We're the AI application layer. So, we work day in, day out with companies to figure out where AI can drive ROI in their businesses. And so, there's a consulting component to what we do where we identify the highest-impact opportunities across their data, workflows, and operations. And then once we align on the strategy, we go implement. And that means building agentic workflows, enterprise search automations, and other solutions that are custom to how that company operates. And so at the end of the day, what that means for companies is they don't just get AI infrastructure, they get a full-stack AI platform that actually does work inside their business.
Yeah. Perfect. Yeah. So what that means is we've got the infrastructure layer with Equinix, we have got the orchestration layer with Orion VM, the compute layer with Tenstorrent, and then now finally to solve people's problems is the agentic layer, which is the application layer that BetterBrain brings there. So that allows us to take this to market to customers quickly and easily in a cost-effective way to adopt AI, and so that's important. So Justin, why don't you talk a little bit about your customers and what they can benefit from this?
Absolutely. Increasingly when I'm talking to either boardroom members, C-suite, other executives, right, they kind of distill their core requirements into, I think, four buckets. I like to call it the four Ps, right? We talk about price, performance, privacy, and proximity. With price, you heard from Yasmina, right, talk about token economics, right? Cost per token, it's a new economic metric that we have to take hold of. With the performance layer, you've got Tenstorrent optimized for prefill and decode, but there's the token throughput, and it's one of the reasons why Jensen actually involved the LPU into their stack, right? And Tenstorrent is actually optimized on their purpose-built chip. You also have deterministic latency, which is what you know Equinix is steeped in building interconnection and ecosystem around our software-defined network, and so I think that's what brings a lot of value. And the last two points to bring home really is privacy: are you secure, are you performant in a secure measure, and are you sovereign when you need to be? And proximity, that's all about bringing data and inference engines closer, as well as users and models closer. And this is really an out-of-box solution that gives instant value, increases time to value for customers, and increases time to revenue, right, for folks that are selling it.
Yeah. Perfect. Yeah. So, you know, AI is not a compute problem. It's moving the data, storing that data, and being able to process that. So, that's really important, and Equinix brings us that value. Shang, why don't you talk a little bit about, you know, your ability to help us with sovereign AI and, you know, the orchestration and drilling that, right?
So, when we work with our customers, we really understand that they have a few different challenges. Some of them are essentially getting access to the high-performance inference infrastructure, but realistically it's a holistic problem. They need to be able to address how they run high-performance databases, load balancers, all the networking around the cloud that they need to deliver. Enterprises are used to consuming infrastructure over the last decade as infrastructure as a service. And so what we do with Equinix and Tenstorrent here and BetterBrain is bring together a solution that a customer can have private, sovereign, turnkey within their environment, and essentially cross-connect to their existing cabinets over Equinix Fabric, things like that. So the idea is really to be able to break through and deliver a private, secure, and mission-critical enterprise AI environment.
Yeah. Yeah. Yeah. So sovereignty is important, right? Because it's not just sovereignty of an enterprise or a government. It's actually country sovereignty. So sovereignty is the big next jump for us. And I think with our partnership, we're well positioned there for that.
I'd like to jump in and just say that, you know, when we look at power being the main limiter, right, with AI, talk about some of the metrics that Tenstorrent's bringing to the table here. I see AI factories being built within our organization across the globe, and you have, on average, a two- to three-x lower power draw with Tenstorrent deployments, right, which allows customers to benefit from cost efficiencies, but also allows the Equinix partnership, right, to kind of deploy that in a more repeatable manner and at scale. And that's what we're looking for. We're looking forward to bringing AI implementations from the garage-type...
projects to actually bring it to a scale, and I think BetterBrain actually brings value into that.
Yeah. So, Abishek, talk about how when we go to customers, it's not just about the hardware, it's not about the data center, it's all about solving a problem and how you guys bring that problem to fruition and solve it for them, and also connecting to their legacy systems.
Yeah, of course. So, what you saw today in the demo was pre-built AI infrastructure that BetterBrain has. And this goes to one of the problems that we see enterprises face a lot today. The landscape is basically you have product companies and you have services companies, right? And with product companies, it's really hard for them to integrate directly into a customer's existing workflows and processes. And so you see poor adoption. And on the flip side, you have services businesses that typically are a little more expensive and slower, right? And so we see a lot of value in this hybrid approach of having this pre-built AI infrastructure that you can then customize to deliver the exact solution that a client is looking for. And this is one of the reasons why we love working with Tenstorrent because their chips enable such a huge diversity of models. We can tackle any use case on top of Tenstorrent. But not just that, it's also the feature set, right? Some use cases demand high throughput, some use cases demand low latency, and Tenstorrent has the ability to optimize their chips to support use cases across that spectrum.
Yeah. Great. Great. So this stuff is up and running, right? Where's it located?
Up and running in Ashburn in DC16 with some other deployments coming here soon. So keep a lookout for that.
I think there's a lot to be had and a lot of scale to be done both this year and years to come.
Yeah. Yeah. So when we go to customers now, we give them a full AI stack, fully integrated with the infrastructure, the compute, the orchestration layer and the application. They don't have to worry about dealing with things, plugging things together. It's all bundled and we're pretty excited about what the customers are going to benefit from this. And so we're very thankful for the partnership, the deployment that we've done, and we're looking forward to grow the business together as partners. Thank you guys. Really appreciate it.
