It is an attempt to turn imprecise English language into not machine-runnable code, but something that is more precise than just words. I think one way to think about the protocol versus direct integration one-to-one client server is this idea of the cathedral and the bazaar. What's so interesting about protocol creation is the people who create protocols have to understand the problem better than anyone else.
Hello, everyone. This is another Context episode. And this time I'm here with Ilya Grigorik. And we are going to talk about why protocols are such an important thing that engineers talk about. And in fact, I think if you do our job right, you are going to, I mean, have a much stronger appreciation for the protocols that run the modern world that underpin the world of engineering and sort of what makes a protocol, what makes them good, and why it's a thing that when you probably find yourself at some place, even if you're not in engineering and you buy drinks for engineers and get them onto their most philosophical self and into the topics of internet history, you will at some point hear about protocols. And then next time you'll have hopefully something interesting to contribute to those conversations.
Ilya, you have worked on a lot of important protocols of the internet in your career. Obviously, the largest one of those is the W3C, the internet body that specifies what web browsers are allowed to do. HTML is one of those protocols, I suppose, I guess. HTTP is definitely the thing that makes all web browsers work, which is something you have worked on. And most recently on UCP, which we talk a lot about because it's enabling the agentic world. And so, welcome.
Let's quickly start with HTTP. Tell us a bit about that protocol, the origin of it.
Well, that's a fascinating story. I think the key insight from HTTP is Tim Berners-Lee made, I don't know if he deliberately made these decisions, but he walked into a design that evolved that we can learn from. So if you look at the very first iteration of HTTP, it was literally one line that you would type in. And it says get with the name of the resource. It is the simplest thing that you could imagine.
And it worked. It's not machine language.
No, it was literally three letters, get followed by the name of the resource. It's as simple as it can get. That is HTTP 0.9. It is kind of protocol. You can only do one thing at a time. It does not understand different resources. But it worked. And everyone has seen this a million times. Like you've looked at the address bar in your browser. Like it says HTTP, usually HTTPS now because he had to make it secure at some point. But he sort of skipped over that back in the day.
But that came later, right? But then you forward evolved that. And what happened was there's this Cambrian explosion of people building stuff because they said, hey, this is cool. There's something here. So then somebody comes along and says, you know, it'd be cool if you could fetch an image. But how do we differentiate that? And so headers come to existence. And headers, one of the key decisions at the time was it's an open protocol. So there was no central governing body. Tim did not sit there and approve every proposal for every header that said, this is how you do it. Instead, it was community of server builders. And then there's a growing community of client builders who just said, look, we're just going to try stuff. We're going to throw some stuff at the wall. Hey, Apache server, if you add this header, and I will add this listener on the other side, then the two of us can negotiate, and now we can speak images. And look, ta-da, right? And this is the genesis of the internet as we know it. There's this Cambrian explosion of experimentation, and it's lawless. People are just doing stuff, and they're experimenting. And then after the fact, smart people come together and say, look, we've tried five different variants of how to negotiate an image. Let's just agree on a convention. And that becomes the IETF lore and you have these like RFCs that get put out. But note that it is not by consent or approval of somebody. Tim was never in that way.
A request for comments. Even that's kind of like prosaic in a way, right?
Exactly. It is, yeah, you put it, yeah, it becomes standardized, but it is still a request for comments, right? So everyone can put an RFC together. It's almost like its own sort of, it almost looks a little bit like engineer-tinged legalese. Like there's a particular, like a should and must and often uppercase, at least in some areas. So it's where there's like, there's an aesthetic to how to put up an RFC. And you get a number assigned or maybe you pick it up.
It is our best attempt. Well, maybe not best. That it is an attempt to turn imprecise English language into not machine-runnable code, but something that is more precise than just words, such that somebody on the other side could implement it in a more respectful way. Which, by the way, is also what the legal language is.
Exactly. It usually starts with a variable definition at the top of terms, and then it goes into a subset of English language, which is more or less executable, and then it all kind of is committed to the source code by case law, right?
Right. Into the mono repo, so to speak, at least in the common law area of the world. And so it's like you make a request for comments because you want to make a change or you want to make an enhancement or you want to specify something that already works, which is actually also an interesting, like very often an RFC is happening for something that's almost backwards facing documenting about something that someone has already tried.
That is key, right? So an RFC is just another step in kind of a market negotiation mechanism. Because the way the internet still operates is, we can set up a server right now and invent a whole bunch of new headers. And so that is the thing that was like that HTTP, I mean, the day before HTTP, basically, it was like a network of a traditional sense. You would make a piece of client software and a piece of server software, and they could sort of talk to each other over a socket. And the genius of the protocol in the middle meant that now it wasn't the same person who had to make the server and the client. It was like one client. And whatever that client supported, it could always send a prophylactic request to the HTTP server you put into the address bar. And if it got a response back, it would then show this. Therefore, what the protocol allowed to do is an end-to-end collaboration. Suddenly every server could talk to every other server or every other client by just sort of trying it and seeing if something came back. And the important part for the protocol itself was to establish that negotiation by accessing. So for now we wanted to add support for compression because that turns out to be useful. You just have to negotiate. We have to specify how to negotiate that capability.
You sent your request saying you would accept compressed version of a content.
Exactly. Exactly. And literally, it's an Accept-Encoding header. And then if a server decided, like, I have no idea what you're talking about, it just sends a plain text, and that's also okay, right? Because you advertise that you support fancy zip. I don't know fancy zip. But if you come back to me with a different request, maybe I will respect it.
So this is fascinating. And then, of course, HTTP evolved a lot, and it's gotten a lot better since with HTTP 2 and 3 and so on. You worked on 2, I think, and 3. Foundations are free, which is very, very good nerd credit. Very nice work. I think what's so interesting about it, though, is like some people listen to this and they're like, well, that sounds not exactly like engineering. This sounds very ad hoc, right? And very like engineering as opposed to, you know, you have blueprints. You build a bridge in a world where gravity doesn't change. You figure out what the tolerances are and then you build on tolerances. And if you, within the margins of the tolerances, you can decide to optimize for cost or optimize for beauty or, you know, just like, you know, the Golden Gate Bridge and the random bridge over the local river all like work on the same principle, but like different decisions were made based on the envelope of a project. So there are some variants in there, but no fundamental, almost market-like price-finding. And this really is messy. For instance, I think HTTP famously misspelled the word "referrer" in it. And that's just in the spec. Did we ever actually fix this?
Yeah, so I'm pretty sure in a thousand years or a hundred years, the official spelling of referrer will at some point just flip. I think every dictionary already accepts it the way it's written in the HTTP standard as not wrong, at least, because it's so commonly used. But it's just like someone misspelled it, I suppose, at some point.
I think it's a different type of engineering. So protocol design is just like, it's reasoning about abstractions. You can reason about data abstractions, data model abstractions, API abstractions. An open protocol like HTTP is an abstraction for you to negotiate amongst many different participants in the ecosystem of what you're willing to accept, what you're willing to negotiate, and how are you going to communicate with each other. So it is done right, it gets out of the way and allows you to do things that you wouldn't be able to otherwise.
Yeah, and so the consequence, like again, really, really nerdy topic here, but then the consequences of this are basically the world we all live in. Everyone here at Shopify has the job which I think hopefully we love and appreciate because of these messy protocol creation exercises early in the 70s, 80s, 90s. Because at some point in 2004, I could just take a like a random computer connected to the internet and I could respond to port 80 and hopefully to 443 as well to just launch a piece of software that I wrote but nothing else than just like listen to these two ports on the internet and if a request arrived that asked for the snow devil or any of the files and paths there it sent back e-commerce website and it had in it like a checkout which...
A soup of XML tags that happened to have checkout flavors.
Exactly. And it just like, you know, like to computer there is no such thing as Snowdevil or a checkout, but like that doesn't matter. Like no one had to pre-plan the fact that e-commerce might happen in HTTP. In fact, they tried and then we didn't sort of with Listen, I suppose, with 402 status code. But anyway, that's a different topic. So this is sort of, and then protocols are really all around us. Like, obviously, computers talk to each other over DNS, which is a protocol, and then obviously they package everything into container-like packets in something called TCP IP, which is the reason why anything on the internet even works. In a very real way, although it might be just on the margins of a protocol and you can pick on which side it's lying. But like Liquid feels like a protocol because it's a way for designers to state the online store they would like to be responded with when someone comes asking for it. And a couple of rules related to what products to show and so on. And those are provided to Shopify, the software, and then Shopify uses its current best way of turning this into a real online store and sending it back. And that's basically what Shopify has been doing for those 20 years that's existed. There's a couple of protocols that are really important and why it's fun to talk about them is, you know, HTTP is a little bit special because it's somehow finagled its way to the browser bar and therefore we actually are aware of it. But really, when a protocol truly works, it's invisible. The only protocols that people know about are the ones that are really wrong in important ways, and therefore everyone's complaining about them, or they will never have heard of them because they didn't get adopted. And so one of the aspects of protocols is that they allow a lot of different people to openly collaborate. I think that's one of the just absolute... I think that needs to be true for a protocol to be successful.
A protocol is... It's a language and a set of rules by which you achieve an outcome. So that can be liquid because you've defined a templating system and you learn that language and you achieve your outcome. Ruby is exceptional at DSLs. Those are protocols. Every time you invent some DSL for configuring, for defining your config or something else, you're defining yet another protocol on a set of conventions. And they succeed and die by their merit and their ability to help you do the things that you need to do.
That's right. People query their own data in Shopify with ShopifyQL, which is a language protocol. The amazing thing is this has existed for a very long time. We've had ShopifyQL for, I think, over 10 years, maybe actually much longer, I think, 15 maybe. And nothing about how ShopifyQL is executed is the same. Right. Like the same query works. You can still ask for your orders based on the weekday, but we can take the same packet that you stated and now execute them in a completely, like everything about the way shop is deployed, it's on different data centers, it's scaled to the entire world. It's like all these kinds of things. So I think this is what protocols also enable, that there can be progress. Protocols don't calcify progress in the same way that a lot of, you know, just like monolithic software does. Let's talk about the, like, let's fast forward to today. I think one way to think about the protocol versus direct integration one-to-one client server is this idea of like the cathedral and the bazaar, and which seems always super appropriate thing to talk about for us in commerce.
Yep. So we talked about open protocols already and the need for kind of market making or, sorry, market making is actually the opposite. You want to facilitate the market to negotiate what they want to talk about, right? So we talked about HTTP and how servers and clients can negotiate themselves without having IETF ratify and specify every compression algorithm, right? We can invent ourselves. I think same thing applies for commerce. You can model commerce as a cathedral. You could say here is the one true flavor of checkout.
It can be very, I mean, cathedrals can be very fancy. They can have lots of crannies. By definition, I would say they are more fancy in many regards because you can have architects that define precisely what they want with the level of flourish that you can't in a kind of open market participant, right? They are obviously impressive. They are obviously, they feel structured. They feel trustworthy. You know, like it's like in a very visceral sense, you walk up to a cathedral and you like, you know, you're like in a place of incredible deliberation. You just look up and you stare in awe, right?
Yes, exactly. And then right beside it is the bazaar, and the bazaar has a very different, you look far and wide and you go, oh my God, there's so much variety here. There's so much-
Exactly. Someone selling rotisserie chickens in the middle of people selling bears of cloth. And it's just like none of it makes any sense.
Right. Some of it you can barter, some of it you need stable coins for. Some of it you need fiat currency. Some of the world delivery. The list goes on and on and on.
That's right. And both need to coexist. And this is the difference between kind of a closed protocol where you can control everything end to end cathedral because you have the blueprint for the cathedral. And you've told everyone to build to this exact spec. But there's no blueprint for the bazaar.
Yeah. Right? It is just kind of controlled chaos. It's not even the same bazaar even like three hours later. Right? Like it's like it's completely changes based on...
Because the participants chose to swap out the rotisserie chicken stand for something else because it wasn't selling.
Exactly. So it's, you know, it's absolutely, it's chaotic. It doesn't look impressive. You look up and see nothing. You open my ear. In fact, it's intimidating probably the first time you walk up to it. It's like, how do you even make sense? No one knows. Like you ask someone, hey, where can I find these goods? They have like no idea because again, tomorrow the whole thing is going to be different depending on when someone shows up where. So now the fun challenge. So it's not easy, but kind of obvious how you build a protocol for the cathedral. Because, well, by definition, it's a blueprint, right? How do you build a protocol for the bazaar? That is the hard problem. And this is why I brought an expert onto the show.
Yes, I specialize in bazaars. How do you write a protocol for bazaar? Well, we just talked about it. HTTP is a good example, right? You do not establish the blueprint for every type of negotiation, what type of goods can be sold in what form with what payment methods. Instead, you define a protocol by which the two sides can negotiate. And then you get out of the way and you facilitate that open bazaar of innovation.
