It's really not about building the most powerful AI company. I have plenty of friends who are doing this. So they can work that out between them. I'm trying to figure out how to make all the gifts that we're getting from the labs maximally valuable to people. I really care about this attitude of helping the world fulfill its desires and its dreams. I think this is exactly the attitude we would want in the company that ends up becoming most powerful.
Welcome to the Derby Mill series. Intrepid pioneers of the next economy featuring discussions with entrepreneurs at the forefront of deploying machine intelligence and brainstorming sessions about where the technology may go at the limit.
Welcome to the Derby Mill series, Intrepid Pioneers of the Next Economy. I'm AJ Agarwal, co-founder and partner at Intrepid Growth Partners and today we focus on the intersection of machine intelligence and online commerce. Joining me for this conversation is the most successful entrepreneur in Canadian history and that's Toby Luck, CEO of Shopify. I'm here with my four collaborators. Rich Sutton of the University of Alberta who pioneered reinforcement learning. Sendel Mullenan who uses machine learning to better understand human decision-making. Neve Gavin, an applied AI scientist who works on foundation models and Suzanne Gildert, the founder CEO of Nirvanic who is an expert in the area of machine intelligence, quantum computing and robotics. Rich, Senel, and Suzanne are all intrepid advisers. So today's episode we focus on the intersection between AI and e-commerce. And I think what's so compelling about this session is the scale of the business that Shopify is operating, about close to six million storefronts and that number is growing every day. Close to a billion customers that are purchasing about $30 billion a month in gross merchandise value transacting on the platform. And so all of that creates scale which generates data which is the essence for how machine intelligence learns. And so deploying AIS into a workflow that's operating at that scale is very significant and gives us a lot to talk about. So let's just begin with for listeners who might not be as familiar with just Shopify as a business. I'm just going to ask you to describe the scale of the business because in my view what you're going to hear about today, you know this year in Time magazine they had these sort of architects of AI as the cover of and I suspect that down the road there will be a front cover of Time will be the most significant commercial applications of machine intelligence and the one that the ones that you're going to hear about today in my view will be a contender for that and part of that's because of the scale of what Toby is going to tell us about. And the other thing just to set the table here, last week I spoke with a woman who Sendel and I know named Susan Ay who used to be the chief economist at the department of justice and she's been very focused on the impact of AI and power and the idea here being that to what extent is machine intelligence going to further amplify the power of very large companies and there's a very good argument here for scale economies and with one of the interesting things about Shopify is that this is a counter example of where they're using machine intelligence to in fact give a significant boost to smaller companies and so as you listen to the description of the size of Shopify on the one hand think about like the size of the company itself but on the other hand the number of merchants that are empowered by this platform. So Toby, welcome and if you could just give an overview of the size of the business.
Yeah, thanks for having me. I look, we're a proudly Canadian company. We started here 20 years ago. It's now powering millions of merchants. We sort of stopped giving the play-by-play updates because they tended to become permanent artifacts of the internet. If you give specific numbers and then I get numbers from very long ago quoted all the time and it just all mumbles together. So ever since we've gone past 2 million, we just have been saying millions. Every one of these small merchants is a small business. It's usually the websites you buy from when you shop online. If you purchase something on the web and it wasn't from Amazon and you like the experience, it was very likely a Shopify store. Recognizably often through these purple buy buttons and the shop logo. So what can I say? It's gotten gratifyingly big. If you put millions of small things together, you end up with a very, very large thing. The original internet scaling laws and we are excited about the company because it's a little bit of an implementation of the vision that I think all of us here likely had for the internet in its emergence during the 90s and early 2000s, in as much as it's almost an implementation of this idea that in the future everyone would have their own website and would self-actualize and share. And there were blogs for a moment and then one by one these things sort of got picked apart into different places and are much more amazing than anything we could have imagined. But we kind of nailed it with online stores because they are these little outposts of sovereignty for customers. It's one of the very few things you can own on the internet. It's your website, your design, your brand. So altogether the size of this makes up to be about 10% of all commerce in the United States. It's very international, we have customers in 175 countries and during our peak moments of Black Friday Cyber Monday, which is an event we celebrate on behalf of all these entrepreneurs, it peaks up to 20, 30, it's hard to get the exact numbers but it's a significant chunk of the internet and a significant chunk of internet's traffic goes to Shopify stores.
Okay. So the format that we're going to follow here is they last week announced this Winter Edition which was effectively just an orgy of new AIs in on everything and there's so many it's overwhelming. So what we're doing today is we're just picking three things that I think are very specific applications that are a good reflection of the power of the technology in their setting and so we'll do one at a time, just clarification of how it works and then we will do a macro where does this go at the limit. So the first one is called Sim Gym.
Think about playing a sport. A gym is where you go to work out so you can do well at that sport when you go on the playing field. That is what this is. And when we say simulator, it's because we know a lot of folks, we just talked about launching with confidence in the one above about AB testing and rollouts and launching with confidence using rollouts, but Simjim allows you to launch with confidence when you don't quite have potentially the traffic or you don't really want to test with real humans, real consumers. We said, 'Okay, what are we going to do for those people?' Because we are the entrepreneurship company. We help everybody grow. And a lot of those people don't have traffic yet. We took all of our knowledge across the decades of building online stores and we built AI shoppers that represent what we see real humans do. And the craziest thing about it is that we are creating simulated versions of the people that are like ones that are usually browsing on your particular store. Basically with this system, we can take your rollouts and simulate them to the best knowledge of AI based on the personas of people who are visiting you, actually clicking around on the store on the design changes and looking at whatever metrics you're going for and predict the future. And this is delivering a thing that is just kind of blowing our minds because we found it to be extremely accurate for what it is. And again, an AB test is something that is super good when you're Fashion Nova, you can run AB tests in a couple of minutes given the traffic you have. If you are starting, it's going to take sometimes for real statistical relevance on a small detail change months. Right now, you can just go and simulate it and that is a game changer. I'm not aware of anyone having done this or having used AI in this way. It's an incredible breakthrough from our teams and in our machine learning labs, the term is sufficiently nerdy: gyms are the environments that reinforcement learning happens in.
Yeah, that's the nerdy version.
And the basic idea here, it's AIS that are predicting the behavior of customers, the response of customers to things. And what's fascinating is it's not just the average customer but it's reflecting the response of the archetype of that merchant's customers.
So we'll do clarification questions here and Sendel, if you want to kick off this one and we'll go Sendel and then Neve and then Suzanne and Rich.
So I guess my clarification question is very straightforward. How do you, what is the data backbone that allows you to get such personalized prediction for this particular business?
Yeah. So the data backbone starts with the log files. We have a transactional history. We see the browsing behavior. We can match to certain personas that we've established through fairly standard clustering. And then also due to deliveries, there are demographics areas, so we pick from tens of thousands of personas, we sort of match for that and then we go from there.
I guess part of it, I think I get that workstream. How do you think about your example with Christmas coming up and 20% is fantastic? How do you think about baking in a macro component because there's the store, there's the regions they buy from, do you think about including that forecasting component or are you thinking of it idiosyncratically?
Yeah. So I think what we do, this is the next phase: let's get it out and figure out what is actually the how to. This is a completely new thing. What it allows people to do in the end of the day is to get to conviction faster with their testing. The core here is that AB testing and these kind of rollouts that we are launching, if you are small, which is always the thing that Shopify obsesses about, how can we help people to scale faster, running an AB test and getting it to real statistical significance just takes a very long time. If you're Nike, you can basically run anything on your website and get a result within the day. If you are not Nike, if you're very small and starting, it'll take so long that it's probably not worth doing, but it's extremely valuable. So we have found that we can get to a statistically relevant prediction quite quickly and then we are seeing the correlation is quite high with the actual counterfactual, which is exciting. And as I'm sure is being discussed especially on these panels, all of this gets back into an RL loop which hopefully will help us hill climb the prediction quality further and further. What is fascinating however, I should say, the simulation actually does web browsing, it's really at that level. So you can test how many transactions do you lose to cookie banners, which is a question that I think a lot of people should ask themselves and then afterwards have a conversation with their regulators if they happen to be in those kind of areas that require cookie banners. Putting some numbers for your particular business with your particular implementation is an excellent use case. And I think there's utility value beyond just is it perfect and will it be predictive. There's enough forecasting technology that will be able to work with regressors like Christmas shopping and so on, and we could parameterize it, we have not so far though.
And Sendel, maybe you could just share a couple of sentences about what you're doing at MIT in the bike lab in terms of thinking about decision-making. In my opening comment, I remarked that what Shopify is doing is the counterweight to this scale advantage that the large companies have. By doing the simulation, otherwise AB testing is so powerful but only large companies have enough data to exploit it, and they're giving this power to small companies.
Yeah. I mean I love what you do. My parents ran a small business, not a web business, the web wasn't around then. But what's striking, and there's so many studies about this around the world as well, is how if you look at decision-making, small business people have to make tons of decisions that are just insanely consequential and which they are just not equipped with the data to make at some level. And it seems like you've got an amazing capacity. That's what I was sort of asking: the ability to deploy scale economies to artisanal producers. So you sort of get the benefits of scale and the benefits of small because you can bring to bear all of the scale, whether it be data. I think that's why I was kind of asking about forecasting. In some level, it can be my data, it can be the data in the world, you can be that intermediary and you are starting to do that and have done it on other dimensions to sort of say, 'What are all the things I struggle to decide? Great, here are all the places we can help.' So I'm very passionate about improving decision-making and this just seems like an amazing vector for doing that.
Yeah, so Shopify, this is not one of the three things we want to go through, but we have integrated a co-pilot thing called Sidekick into the software. Sidekick is an enormously successful company. Think about it: the people who are coming to us are showing a lot of courage to put themselves out there. It's a very public thing to build a business, but they are very often quite scared of failure for very good reasons. You're defensive when you're putting yourself out there. It might be that starting a business was plan B for them and plan A has somehow come to an end. They may have lost their job or some reason. Therefore, they probably also don't have a lot of money to invest. So they have no background in doing this. Shopify is a company built for a customer base that is courageous, reaching for independence, reaching for glory, but has no money, is unsophisticated, and probably has very little time. Shopify as a product is forged in these fires. I think what Sidekick can already do is incredible. The amount of conversations that people are having, it's an amazing, massively retentive, massively integrated system. It can do agentic tasks for you. You can give it a CSV and it builds you a little app that turns that CSV into an importer. It has all these features. But it's cool to talk about features and symptoms, but what people actually are doing is like, 'Hey, can you tell me anything that I can tell my friends tonight when we meet about that's going well?' And it'll find something. It's able to access benchmarking data and forecasting data. We've actually been amazed because people often upvote these things and then we get to read the conversations as part of looking at if it's golden data set material. It just finds these little factors saying, 'Hey, you are more successful than 90% of other shops that started in the same week,' which is a wonderful thing to tell an entrepreneur because it's free to sign up, there's lots of trials, lots of people playing around or just clicking around. So it's not a hugely consequential statement from a financial perspective, but it is something that's worth telling your friends on a Friday over drinks. There's a humanness to the AI world that's quite different. Software is very autistic. It never ends up talking in the language that humans actually like. The way I talk to my friends about computers and what they can do and the empowerment I'm getting from them has never been their experience. In fact, it's quite the opposite. From their perspective, they often think, 'Hey, this AI stuff is amazing because you guys fixed the computer. They finally do the stuff that you always talked about for the last 20 years and I couldn't see.' We see these effects hit, and this is the thing I am personally just incredibly excited about. I see Shopify as a company that's kind of a bridge. On one side is the country of incredible research and all of you publishing papers and improving state-of-the-art, and on the other side we are making these sorties in the world of papers and the technology world and the global network of commerce, finding these morsels of incredible value and trying to bring them back to entrepreneurship land where people are trying to self-actualize. This is an incredibly interesting time. Editions is really us saying, 'Hey, here's like Shopify releases two editions a year.' We are mapping this to the fashion world which has new collections twice a year. Right now it's just AI time because that's where all the value is and it's incredibly valuable. Sorry for the unscheduled proviso. I'm just extremely passionate about it and I really want to also have people who are doing the thankless and complex postdoc work in the labs hear that it is just so valued on the other side in industry as millions of people having conversations. I'll say one more thing about Sidekick. We had a problem for a couple of hours last week with a completely unrelated login system and our Sidekick system was hitting it too hard while it had problems and we had to take it out. For about an hour we didn't have Sidekick on Shopify and we have never gotten so many emails about something. People basically sent us messages saying, 'Hey, my favorite employee of the month didn't show up to work.' Someone wrote an obituary for Sidekick. People told us we murdered or held hostage their favorite colleague. So it's quite lovely on this level.
You know you have a good product when people complain when it's taken away and it's like how much will they pay to have it back. So kudos there. Going back to Simjin, I think the difference here versus the current flood of eval gyms, RL environments, time series, etc. is it's based on millions and billions of real world usage, not synthetic data because you're super fortunate to have this full end-to-end real RL environment which most of the even frontier labs lack. So whilst creating these synthetic customers was hot earlier in the year, few actually made it reality and especially on an NF1 like merchant and user level. So what's amazing is you've enabled Simjim to not only do AB testing, but the thing that's even more mind-blowing is preferences via these optimized focus groups. In terms of clarifying question that's super expensive both on a storage and simulation basis. So is your aim for now just to store everything, lean into it and rely on the cost rapidly decreasing over time or have you found a hack that others aren't aware of?
No, it is quite expensive to run. You're quite right there. All the usual optimizations apply. Distillation is playing a big role here to make it all tractable. It's not the cheapest, so we are in a rollout and everything's compute limited. I feel like I'm getting a front row seat to what everyone else's last three years looked like in terms of conversations with various inference capacity. So it's all good. We don't charge anything for Sidekick or so, it's all built in because again, if people use it to build bigger businesses, our business model is fully aligned with our customers. We take a tiny little percentage that we're getting for payments and these kind of things and the monthly fees. So we figured it all kind of works out. That's another nice thing: we don't have to think about the monetization and the margins to the same degree that maybe others do. So far, the token costs have also been coming down so enormously that we feel like even connecting the curves here, it'll probably work out to bring this to a full rollout. But if we do contributions to the world of research, it's going to be by validating some invention or something like that. That's probably not the thing to look at for us.
Excellent. Thank you for that. And just to make sure that listeners fully got that point because I don't want it to get lost: so many AI companies are struggling with pricing. In other words, they generate some prediction for a thing and then they sell their prediction and nobody knows what to pay for a prediction. So imagine you're building an AI, let's say in drug discovery that predicts binding efficacy, which molecules will most effectively bind to which proteins, and then they try and sell that to pharma. Pharma says that's valuable but they have no idea what to pay for that prediction. There's no sort of market for prediction of that kind. Shopify's giving this as a service to their merchants and the idea is that they will benefit if it increases the thing that merchants care about which is their sales. So it's very aligned and the pricing or the value of the AI is just being absorbed in the existing business model because of such tight alignment. Suzanne, over to you.
So, I want to take the conversation in a slightly different direction and maybe ask a bit more of a sci-fi question about the future of all this. So what we've seen on social media over the last decade or so is that where we were originally is the situation you guys are in now where it's this beautiful ecosystem. You've got all these lovely boutique kind of artisan small businesses that are now getting out there, getting their products to market and it's in this beautiful phase. But that was the way things like social media started where everyone got a voice and it was really nice and what happened over time was with the invention of AI tools we started to see more and more auto-generated content which was engagement farming and all this stuff started happening and then it's just kind of taken over the whole system. I guess my question to you is: are you worried about or have you even seen already any signs of automated store production where there are just fully automated pipelines now where stores are being set up and created in large numbers by AIS? Maybe they're drop shipping type systems and this is now starting to push out these kind of small business owners that are actual people. Maybe you might call it shop slop. Are you seeing any of this now or is this just total sci-fi that's not happening?
I feel like our world is a little bit different here because you have two governors in the system. One of them is that at some points atoms have to be assembled into a final product and to do this someone has to part with money which is a rivalrous resource. You don't have the zero marginal copying of generation of tokens that causes some of these effects in the same way. In fact, a purchasing transaction is actually a vote for everything that is behind it. Every time you purchase something, you're voting for the product existing, the business behind it existing, the supply chain that created it, and so on. In that way, this has always been a tension. Shein and places like this are fundamentally going around and looking what works in the entrepreneurial space and then just kind of creating shop slop. I like that, I'm going to run with that. And that is okay. It wouldn't be my choice of what to spend my time on all day to enable it, but it also plays a role. Usually the people who purchase this would potentially never have purchased some of these kind of things. Honestly, this all kind of exists. I think the highly automated cases are starting to see them. It'll be a factor because some areas like swag creation really lends itself, also just event based quick stores. We'll see it and all of this is entrepreneurial and all of this is if someone purchases then it was valuable. It has its own evaluation of merit. That's the interesting part of our little niche here.
Suzanne, we will put a trademark on shop slop and it originated right here on the Derby Mill podcast. Rich, over to you.
Oh, this all is really cool, but I'm just amazed that it can work because, let's say you want to do an AB test on two different website layouts that you're proposing because they both seem plausible to you and then you can send those out and get feedback from simulated customers. How does that work? And how does it actually do better than a shop owner's intuitive judgment of what might be appealing? How do you train them? What do the simulated customers see? Do they see the layout as a picture and then they click?
It's literally parameterized. Essentially agents arriving to the website doing the vision model browser use loop. They are scrolling, they are looking at pictures, and they are not going to navigate to the new product that you would like to test in your market if that doesn't happen. You're essentially running a Monte Carlo simulation that's parameterized to correspond to the audience that usually browses your store to the best of our ability. It is delightful that it actually produces results worth having, but it does and that's the remarkable thing. I have to say I was very skeptical of this effort. Now I'm looking very clever here because I was shipping an excellently valuable thing. I initially was extremely skeptical that we would be able to actually get better than coin toss results on this. I suppose it just turns out when you're training on the average output of the internet and average human output, for the purposes of essentially a focus group you have a lot of significant generalizations in the base models. Then by us being able to turn it into structural data based on true ground truth on logging and demographics of the transactions, it just combines into something quite worth doing.
Rich, in some sense, in many of the companies that we've spoken with on this podcast, they're building AIs that they then sell to companies that are an input to some process that company is doing. Like the Facto's episode where they were making predictions on some inputs into a factory operation and then the factory had to operate that thing and then it would produce the thing and then it would ship the thing. When we were talking about an end-to-end RL system, we always had to think about what's the goal and what's the end goal and then if we work backwards what are intermediate goals, proxies. Don't you think in this setting that one of the things that's so beautiful is they have the data for the end goal which is the sale and so they've got the ground truth. In so many examples we've looked at, there's quite a distance between the thing that AI company's doing and the ultimate goal. What's your thinking here? In my view, it just seems like there's so much goal data.
Yeah, you're right. This is special about the shopping application and I guess Shopify has access to all the data of what's sold by all the merchants. So for once we really have the ground truth in the goal department. That's really important.
The actual transactions are end to end, deterministic essentially because it's the full data: the storefront transaction, the payment gateway, and then the merchants' fulfillment plus the tracking data all along the way. So it helps a lot to have a data set assembled first body.
And you can track on a long-term basis so that if customers are dissatisfied, you can see that in their subsequent behavior.
We see exactly the reverse logistics, returns, and chargebacks against the cards. So it's really the full life cycle of a transaction, which is fairly unique. In most other cases where companies have end-to-end data, they run one marketplace, like Amazon comes to mind, and therefore the demographics using it is the demographics using Amazon, which is probably a fair proxy to all demographics. But we have millions of individual URL stores that are quite separate and they own worlds with their own language, their own aesthetics, and changes to the aesthetics over times. So for all these reasons, I think we could put something together that no one could quite line up.
So Rich, I'm going to address this question to you and then we'll shift into our 'where does this go at the limit.' I've been watching Shopify's application of AI for a few years and just thinking about it and I've come to the view that, and I'm not saying this because Toby's here, I'm saying it because of how I think about the future of this technology, that at some level 24 months from now, Shopify may be the most powerful AI company in the world. Here's why. If you believe that the future of machine intelligence is RL, and in order to really flex RL's muscle you need goals, and if a key goal in commerce is to sell something, you need data and specifically you need goal data. Furthermore, you need as many of the antecedents to the goal as possible. Then you might say, 'Well, isn't that Amazon?' But we also know that RL the way to flex an RL muscle is through quite a bit of experimentation. Arguably, if you take the degree of experimentation on Shopify relative to Amazon, it feels like there's so much more experimentation on Shopify than there is on Amazon. So if they have a vast amount of data, more experiments than anyone else with that data, and the end goal that for example Anthropic does not have end goal data, then doesn't it seem like if you were to try and pick what's the company that has the most key ingredients to exercise the power of RL, can you think of something better than Shopify?
I'm thinking you might have something there, AJ. The key thing is the experiments. All the other companies could do experiments in their logistics chains or how they supply their customers, but with so many independent merchants, they'll effectively be doing vastly large numbers of experiments. That seems unique. You may have something there, AJ.
All right. Well, I'll let you keep chewing on it and have the machine running in the background and we'll go around. So I'm just going to mention the two other products. We won't go as deep into them like we do with Sim Jim. So one is Shopify Product Network.
This is major. I think it's a great example of how merchants can leverage Shopify scale to achieve more as a merchant community on Shopify. Merchants will definitely know this pain point really well. We all work so hard to get traffic to our stores, to our sites, and sometimes a customer actually bounces because they couldn't find a complimentary product to add to their cart. Enter Product Network. What do you tell us about this? How does Product Network help merchants with conversion? What is this all about?
Yeah, Product Network is a stealthfully amazing feature for conversion mainly because for large retailers they have always known if you have more products the chances of a customer checking out is higher. If I'm looking for a chair, I'm also maybe looking for a table because I'm shopping for my home office. But how do you get a really full-fledged merchandising catalog and keep it up to date? That has been something that takes a lot of time and effort. Product Network solves that for you. With just a click, you access the millions of merchants on Shopify and then you can basically tap in and say, 'Hey, I'm a home wares store. I make chairs. I don't really make or want to source tables. Shopify, help me fill that category.' I'm in control. I can say I don't really love this brand or I want specifically this category. Then automatically my online store will start to fill in the points that I want, whether it's empty search results or collections, it starts to fill with tables in my scenario. The chances now of my customer not just finding my chairs, but also finding these tables and making one nice big purchase for their home office increases.
