AWS Summit 2015- Low-Latency 멀티플레이어 게임 글로벌 배포하기(Nate Wiger, 아마존 웹서비스 글로벌 수석 게이밍 솔루션스 아키텍트)

Good to be here. This is my first time in Seoul. I'm really excited, actually. It's a beautiful city. My name is Nate Wiger. I run the global gaming solutions team. Previously, before joining AWS, I actually spent many years at PlayStation building games and launching them on AWS. So the stuff I'm going to share with you today is not only from gaming customers worldwide but also some of my own first-hand experience, kind of trial by fire.

Throughout this talk, I'm going to illustrate some different patterns that game companies have found very successful worldwide in order to deploy very popular games. To start with, I wanted to highlight some particularly relevant use cases to the type of games that you guys are developing here in Korea. Some of these on the slide you guys recognize probably most of them. Supercell with Clash of Clans, one of the most successful games in the world, actually runs entirely on AWS and they don't own any data centers anywhere. They run their game entirely on AWS. Ubisoft used AWS to launch 10 mobile games in 18 months. They also said that there was no way they could have done that in a data center because they were moving too fast. Naughty Dog on the console side actually runs all of their backends for their console-based multiplayer games such as Uncharted and The Last of Us also runs on AWS. Nintendo, who Shinpei from the Japan team is going to talk about later, runs their Miiverse infrastructure on AWS as well. Both Nintendo and Naughty Dog interestingly have said that they saved over 90% in server costs when they moved from physical data centers to AWS. So not only did they gain agility and the ability to develop very quickly and deploy quickly, but they also saved a considerable amount of money as well.

A few others I want to highlight before moving on into the technical details. On the MMO side, because MMOs obviously are huge in Southeast Asia and Korea, Red 5 Studios, which is actually a small studio that was started by a bunch of Blizzard veterans, runs their MMO Firefall entirely on AWS. This is a game that has global users, very low latency requirements as a first-person shooter style MMO. If you haven't played it, it's a really fun game. They actually automated their entire deployment so that they could take their servers that they had deployed in the US and deploy them to Europe using our CloudFormation template language. They were actually able to deploy to Europe in less than one day using CloudFormation. This afternoon, Yoon Jin from the Korea team is going to give a talk about DevOps using AWS. I very much encourage you to see that talk because he's going to walk through a lot of the specifics about using CloudFormation and templating. Last but certainly not least, the Korea companies such as Nexon and Devsisters also use AWS. Devsisters has a great use case they're going to talk about later where they were actually able to absorb a huge spike due to an app store promotion, very similar to what Mike highlighted when he mentioned PewDiePie. The same thing actually happened at Devsisters' game Cookie Run. The app store ran a promotion and didn't let them know, and overnight they suddenly saw this huge spike. Because they were using AWS and using AWS auto-scaling, they were actually able to scale up and meet that demand. They woke up the next day and saw that their servers scaled up and everything worked out. So they're going to talk about that use case this afternoon, another great use case as well.

In order to talk about the details of how do you actually make this happen, how do you actually deploy a game successfully as these companies have done, how do you deploy it on a global scale, what I did is I created an imaginary game. I just made it up and it's called Knights vs Samurai. The idea is this is an online game and you just go online and you find your friends and you hack them into little pieces. That's the whole gameplay. You go on and you hack your friends to bits. There's a global leaderboard and a friends leaderboard, and you just try to see who can one-up the other person. The leaderboard has stuff like total kills, decapitations, maybe you can set people on fire, shoot an arrow, so how many people did you set on fire, all that kind of stuff. Good clean fun as we call it. So we want to deploy this game globally because we want the opportunity for anyone around the world to be able to hack their friends apart. So how are we going to be able to do that?

The first thing top of mind is latency. With a game like Knights vs Samurai, it becomes super important for us to have that low latency experience because any kind of delay is going to result in a bad player experience. You don't want to be trying to hack your friend's head off and then have jitter in the game. This is a map of global latency in different countries around the world. The exact numbers actually aren't super important, but what is important to notice is that as long as you're staying within a given country, for example within Europe or within North America, your latency is very good. 10, 20 milliseconds, 40 milliseconds in the US. Those are all perfect latency figures in order to have a very good gameplay experience as long as we can keep it under about 50 milliseconds. Our gameplay is going to be just fine. But on the flip side, any time we have to make a call across to a different continent, for example crossing the ocean to talk to Europe or crossing from North America over to Tokyo, then our latency goes way up. All of a sudden we're talking about 150 or 200 milliseconds latency. That will never work for any of the low latency requirements that we want for our gameplay. But we still want a global experience. So how are we going to resolve these because they're inherently in conflict? We want a fast low latency experience but we also want a global experience. One of the big keys here is we need to start thinking about our game a little bit differently.

The first thing we might do is when we're thinking about how are we going to deploy our game, we would say okay, what we're going to do is we're going to take game servers and we're going to deploy them around the world in regions that are closest to end users. So we might deploy them in Oregon or in Frankfurt, Tokyo, South America. That would give us good coverage. We have game servers near users so they get that low latency experience. But what are we going to do about their data? What do we do about their actual user profile and their preferences? How do we give it so that people can see each other's scores? Well, the first thing we might try is take all of our data that we have in all of these different areas and try to replicate it around and build this big database replication layer. But that introduces a whole bunch of problems. All of the low latency that we just tried to accomplish by having the servers near the users, now we've lost all of that. In order for us to have the data be consistent, we would have to wait for all of that replication to go all the way around every time I make a change. Let's think of a real world example. Say I'm running around and I hack my friend, or I go and I pick up an item, or I spend 100 gold. You're going to want to store that in a database so that it's accurately reflected. If I get a new sword and then I go and I play another game, I still need that sword even if I'm connecting somewhere else. I don't want to lose my sword because the data hasn't replicated. So I have to wait for that whole cycle to complete. Now rather than me having a response time that is 40 milliseconds, all of a sudden my response time has gone back up to three or four hundred milliseconds on those critical different applications. So we're not going to do this. It's a terrible idea. Please don't do this. If there's one thing I can leave you with, do not globally replicate data. Please. There's much better ways to approach this problem, and we're going to talk about them.

Here's the best way to think about this. You want to take that concept of a game server, and I want you to think about if you've been doing games for a while, you probably have this idea that you build this game server like an old Quake server. It's a server and it runs on a server somewhere, it's a piece of code. Now I want you to take that idea and I want you to just visualize just throwing it on the ground and just breaking it into a million pieces. As you're picking up these different pieces, I want you to separate apart the ones that do need low latency from the ones that don't. What I mean by that is there's some parts of your game where you're going to have to have that very good response time. All of the multiplayer running around, you need that 20, 30 millisecond response time. But a lot of the game functionality you don't. If you think about what you usually do when you go through a typical game, you go through the app and you'll login, maybe you'll get a list of your friends, you'll get a list of the online games you can join. All of those can take two or three hundred milliseconds to respond. That's no problem because you're clicking a button. By the time you get a response back, it populates a list. It still looks very responsive. So you want to split apart these two different pieces of your game. What we're going to do then is we're going to put our game servers near our players, but we're just going to have the game servers just do the multiplayer traffic. They're not going to be responsible for any of the state information. Everything else we're going to build into a set of HTTP APIs. This in a nutshell is service-oriented architecture for gaming. A lot of the gaming companies I highlighted at the start are using exactly this approach to be successful on a global scale. They're building functionality like login, friends lists, leaderboards. They're building those into web services, and then they're still keeping all of the multiplayer via UDP low latency local game servers, and they're using these two in a hybrid mode. That's what I'm going to spend the rest of this talk really getting into the details of what that looks like. As I mentioned, data replication bad, don't do it. But having the local cache is okay. While I'm actually playing the game, caching my data locally on the game server and then when it's done syncing it back to the main database, that's okay. We'll look at that pattern. The last thing I want to highlight is that nobody really plays everywhere. A lot of times when you're designing a game, you say, hey, we want people to be able to play everywhere. It sounds like a great feature, but think about it. If I'm sitting at my home in California where I'm from and I log in to play Knights vs Samurai, do I want it to automatically connect me to a game server in Brazil? No, I don't want that. The latency will be terrible, and plus I don't even know how to swear in Portuguese, so how am I going to understand what the person is yelling at me at the other end? I want to spend the vast majority of my time in my local region with my local friends playing in my local timezone. Then there's times when I want to make the choice to go over and play against a friend in another region. I might say, okay, I would like to play against Shinpei, he's in Tokyo, I want to go and connect and play with him right now. That's a very different flow that opens up a lot of possibilities for your backend design. Now what you can do is you can hand the person off and take the player's data, cache it in Tokyo while I'm playing against Shinpei, and then when I'm done, sync it back to my home region. A very successful pattern.

Let's look at some nuts and bolts. When we follow this pattern, this is what we wind up with. We have our HTTP APIs, and for many games you can actually just have one stack of HTTP running in a central region. Many games do this. They'll just choose one central location like Virginia or Oregon, for example, and have all of their users make the web request to that region. Then when it comes time to actually do gameplay, your gameplay goes to a local server. That hybrid approach gives you the best of both worlds. You have a simple central database that you can use to store your top leaderboards, your friends lists, have that global experience, but then you can also have game servers distributed around the world that give players the best low latency experience. The key here is when you're designing your game server, when you're splitting it apart, you need to pull apart all of those different services and figure out which ones actually need to be near users. It's only a little bit. It's just the gameplay. Everything else you build in that HTTP layer. Let's dive into this in a little bit more detail. This is that HTTP layer, and we'll walk through the architecture for what this looks like. This is a common pattern that many game companies use, works very well. At the top, we've chosen a region. A region is an area of the world like Oregon or Frankfurt. Within that region, we're using two availability zones. Each availability zone is designed to be isolated from the other for high availability. The easiest way to achieve high availability with AWS is to put half your servers in availability zone A, half your servers in availability zone B, and put a load balancer in front of them. This is a very common AWS pattern. If you've looked at the AWS website or done anything with AWS, I'm sure you've seen this pattern. Very similar to high availability web patterns. Again, this is just for the HTTP layer. When we get to the game servers, it's going to look quite different. But this is just for our HTTP layer. We've got the front-end web servers that are talking to the actual client through the load balancer. The next thing you see there is a caching layer for performance. You're going to want some kind of caching layer so you're not hitting the database with every single request. Very common with games. Redis is very popular. We have our AWS ElastiCache service. What ElastiCache does is manage the caching layer. When you deploy it, you just choose your choice of either Memcached or Redis, and then we take care of managing the rest. So you get a managed caching layer there. For game companies, we usually see Redis being very successful. It's what I used at PlayStation as well. The reason why is Redis actually has more advanced data operations that can do in-memory sorting. It makes it really easy to build a leaderboard or top activity list. You can build it all into memory. Finally, at the bottom there is our database layer. For this I've chosen DynamoDB. You can run MySQL or SQL Server on AWS as well. We support those through our RDS product. Shinpei from the Japan team, when he speaks this afternoon, is going to give some examples of that. But we find that DynamoDB is very popular and very successful with gaming companies for a few reasons. First of all, the fact that it can keep up with the high level of write throughput that games generate. If you think of a normal website, most websites have about ten to one reads to writes. Most of the time they're reading data. You have a few people that publish articles to a blog, and then most people are consuming it. That's what relational databases were built for. Relational databases were built for read-heavy workloads. But games are the opposite, exact opposite actually. In most games, you're writing three to five times more often than you're reading. So you're actually doing three to five times more writes than reads. Many game companies that scale have run into huge problems with MySQL, for example. This past year at our re:Invent conference, EA talked about how they replaced 120 MySQL instances that were sharded out, a huge cost for them, with a set of DynamoDB tables. They dropped their storage costs by 90% for their database and they solved their scalability concerns. The reason why DynamoDB is such a great fit for games is the fact that it's designed to handle all the little tiny updates that you're throwing at it from a game. If you think about a game, I hack my friend, I want to update that. I pick up an item, I want to update that. I spend gold, I want to update that. I die, I want to update it. It's a constant barrage of updates that you're doing to the database over and over again. DynamoDB is specially designed for that. It has a little less query flexibility but much better scalability, and that's the trade-off there. So it's a great fit for games.

A few more items here on the HTTP stack, and then we're going to dive into some more details on the game server side. Any kind of static content, downloadable content, maps, user-generated content, maybe photos of your avatar, you want to put that in S3, which is our Simple Storage Service. That's at the bottom right, the red bucket. You can link that up with CloudFront, our CDN. We have over 50 points of presence around the world. What that does is it gives you much better performance when users around the world are trying to download maybe the latest game patch. You upload the file to S3 and then CloudFront distributes it to all of those users around the world. We have lots of people using CloudFront plus S3 for their game patches, downloadable content. There's a lot of flexibility it gains you as opposed to pushing all of those through the app store. If you think about a use case like I want to do a Halloween-themed item or a level for Christmas or the holidays or New Year, you could push that through the app store but it's a long process and you have to wait for approvals. Or you can just put it in through the CDN and your app can just download it directly. Then you have full control over when you release it and when you withdraw it as well. So there's a lot of flexibility. A lot of people using S3 and CloudFront for time-limited content or custom items, custom maps, etc. Two more things. Anything that doesn't need to happen right now for your app, if you think about an example of I upload a photo, let's say I upload a custom photo to my character, I don't need right now to generate all of the different thumbnails. That can be done in the background. All the app needs to know is okay, I received your photo successfully. So you can use SQS, which is our queuing service. You can put an item into SQS and say, okay, when you have a chance, I want you to generate a thumbnail for this image. Then you can have a separate set of EC2 instances doing that. That's a great way. When you hear us talk about loosely coupled architectures, putting queues between layers is a great way to achieve that. What you're doing then is you're creating a buffer. You can put a whole bunch of items into a queue and those can be processed later, and you don't have to worry about holding up your main application. If a whole bunch of people all of a sudden upload photos, then you can just put them in the queue and it doesn't affect the scalability of that main application stack that you have in the center. Two other items I'll mention briefly and then move on. Cognito, we recently released and it's becoming very popular for mobile developers. It's basically login and identity as a service. It integrates with Facebook, Google, and Amazon logins. You can also use your own custom login. But the really cool thing it does is if you integrate it, and it's just a few lines in your mobile SDK, in our AWS mobile SDK it's just a few lines of code, then it will actually take care of syncing your preferences for a given user across their devices. Say I play, I'm on the metro and I'm playing Knights vs Samurai, and then I get home and I want to switch to a tablet to get a bigger experience. When I then login to my tablet, if you're using Cognito, it can actually sync your state between those two devices automatically. It does it all out of the box, it's built into the SDK, and you can integrate it with all the other AWS services. So it's one less thing you have to build. Rather than having to build your own authentication service, you can use Cognito for authentication and then just focus on the unique parts of your game, like the gameplay and hacking people to bits. Last point I'll mention here is the concept of a background queue. Say I login and I beat my friends and I one-up on the leaderboard. As Mike was mentioning in the first session, you need to build that monetization, that retention cycle. A great way to do that is through mobile push. SMS mobile push is a very cost-effective way to do it. All you need to do then in that case is you would have your EC2 instance say, Nate beat his friend's score. You could have your EC2 instance just trigger an SMS notification that was sent to my friends. It says, hey Nate just beat you, you better come back in. You could click that message and jump back in the game. It's a great way to increase the retention cycle for the actual game.

This is our completed HTTP API with all the features we have. The thing I'll mention about this is you have a couple of choices here. You can either build just one of these, or if you want to split it out by services, you could split it apart by major services. It's really a trade-off depending on how you want to do your development, what works best for your team. You can have just one stack, or you could have one of these that was for authentication, another one that just handled your leaderboards, another one that just handled friends lists. The advantage to that is that then you have the flexibility to distribute these around the world. The disadvantage is that then you have to keep the versions in sync between these different stacks. Both ways work, but when you're deciding on your game, just keep it in mind. If you follow this pattern, it's very easy to template using our CloudFormation language and be able to create multiple ones of these, maybe just with different application code, so you have that flexibility.

Let's look at how the actual game flow works when I come into the game. The first thing I do is I'm going to login. As mentioned, I'm logging into my HTTP stack. I'm going to get any kind of asset downloads. Maybe there's a holiday-themed item or a new custom map. Then I'm going to do matchmaking. Now we're talking about game servers. I want to go in, I want to find a friend, and I want to go hack them apart, hack and slash, move up the leaderboard. What the HTTP layer then says is, okay, you're in luck, I have a game server in your local region. It hands back to the client the IP address and the port. Then the client actually goes and just directly connects to that game server. This is usually UDP, maybe you have TCP for control channel, but typically your gameplay data is then UDP. So as the game player, I'm running around, I'm sending all this positional data to the local game servers. Maybe a friend joins, we're in there playing together. Then when the game finishes, I sync my data back to the main HTTP-based API. The key part about this is I have these two different channels I'm using at the same time as a game. I've got this HTTP channel where I'm saying login, give me my friends list, give me a list of games to play. Then I've got this other UDP channel that's handling all my position information. This is a real big advantage if you think about it because these can go totally different places. I could login to a central authentication server say in Virginia, and then I could find my leaderboard. Maybe there's a separate leaderboard server that's over in Tokyo. They're all just HTTP requests, so they can go anywhere. Then my gameplay data is all going to go locally. Maybe the gameplay data is going to Tokyo, or maybe I'm a player down in Brazil, so my gameplay data is going into Brazil. That's the big advantage to having these two separate channels. It gives you a lot of flexibility in how you distribute your game globally for the best player experience.

We've talked about the HTTP layer. We're going to go into deep detail now on what the actual game servers look like. These look a bit different than our HTTP application. We're still using multiple availability zones, a subnet in each availability zone for best high availability and failover. But the big difference is this is now a public-facing subnet. We don't have a load balancer in front of this. We're going to have our game clients directly contact the game servers because we want as low latency as possible. Behind that game server layer, we're going to have a private subnet. That's going to have our caching layer. As I mentioned at the outset, caching is good. Having local cache data works to your advantage. So we're going to have those local caches that are near our game servers, and we can have our game servers then contact those caches. The last part about this that's very important is a lot of times you want your game servers to be able to communicate with one another, especially if you're building a big MMO world or something that has a lot of state going on in the world. Historically, game servers implemented this via multicast. Back when I was at PlayStation, our previous versions of servers all used multicast. It's a very common pattern. You connect them all with a private network and then they send their positional updates between the different servers. It's nice because it's easy to implement, but it's not very secure and it's also very constrained. You have to have all of your servers located in one area. It doesn't give us the flexibility where we want to distribute these around the world. Instead, what a lot of game companies are doing, a good pattern, is to use a low latency message bus and have all of the game servers then join that message bus. In this example, this is using RabbitMQ, which is one of the most popular. What RabbitMQ does then is when you have a game server transmit a message to RabbitMQ, RabbitMQ will propagate that to all the other game servers. So it's a great way to do basically broadcasting between game servers so that they can all understand the current state of the world, especially important for MMO style games. Deployment of RabbitMQ on AWS is actually pretty easy. You just have a pair of EC2 instances. You can use our Elastic Load Balancer in front of these as well. It works very well. You just run it in TCP mode rather than HTTP mode. Then you can have your game servers just connect to the ELB and it will distribute the load across your different RabbitMQ servers. I'm going to pause for a minute. Feel free to take a picture of this. This is a great pattern in terms of distributing these game servers around the world. You're going to have these independent little pods, as I call them. You can think of them like pea pods, or sometimes the way I think of them is like spaceship escape pods. If you think about Star Wars and they're escaping from the Empire and they're shooting out in these escape pods, the idea behind these is that they can communicate with one another but they're not dependent on one another. We're going to be able to distribute these around the world in a way so that they can communicate but they don't have any interdependencies. If something happens to one of our regions, say we have game servers crash in Singapore for example, it doesn't impact the rest of our game. That's a big advantage to having these loosely coupled architectures. If we were doing that big database replication that I mentioned earlier, you could have a situation where a game server outage in one region could affect all the different regions. That's obviously not what we want. By building these as independent pods, we can distribute them and any failures can be localized just to those servers. Maybe a few people get kicked offline, but the vast majority of the community is going to keep playing.

I'm going to mention a few details. Feel free to just take pictures of these slides because I'm just going to go through them quickly, but they're good references. Highly recommend our C3 instance family. That means it has more CPU versus memory. Game servers are typically CPU limited because there's a lot of simulation going on in them. The C3 has very good network performance due to its enhanced networking capabilities. These are some links for how to enable enhanced networking and make sure you're using the latest version of our C3 instance to get the best performance. The difference it makes is without enhanced networking, all of the network actually has to be handled by the hypervisor. With enhanced networking, it actually gets handled by the device firmware. That means it gives you much better performance because it's not having to be handled in software at the hypervisor layer. Again, feel free to take pictures of these. I'm just going to go through, but I wanted you guys to have details in terms of setting up RabbitMQ with Elastic Load Balancing. These are some details on the ports you would need to open. You just set up a pair of nodes and make sure to open those ports. There's one set of ports for them to replicate and then another set of ports that you will actually connect the ELB to. Here are some helpful links. Feel free to take a picture of this that just show you specifically which ports to open up. The last option I'll mention here is if you're already using Redis for other things, as I mentioned, ElastiCache with Redis is a good choice because you can do leaderboards and build other pieces of functionality into Redis. Redis also has a feature actually called pub/sub. That's what you see at the bottom here. It's the same architecture but we're now using Redis pub/sub. The advantage there is it's one less separate layer you need to manage. It's just included as part of the ElastiCache service. So if you're going to use ElastiCache with Redis, highly recommend you look at using pub/sub in Redis. Same piece of functionality as RabbitMQ. It allows you to connect your game servers together so they can all communicate.

In terms of automation, I have one slide on this that I just want to mention. Yoon Jin this afternoon is going to spend a lot of time on this. In terms of automating this, you're going to use a combination of CloudFormation, which is to spin up the AWS pieces, and then an automation framework such as Chef to actually configure any of the application software. This is one very successful approach game companies use. As I mentioned, Red 5 Studios used this exact combination. They used CloudFormation to boot up all of the EC2 pieces and all the databases. Then as those EC2 servers booted up, they used Chef in order to configure the application level software.

I have a few more details that I want to share with you in terms of things to think about when you're really deploying this game globally. One of them is how to deal with your log data. You're going to have all of these game servers around the world and you're going to have to figure out a way to get your game log data back, not only for debugging in the case of errors but also to see where your different users are playing and how the activity is globally. We have built into CloudWatch what's called Amazon CloudWatch Logs. It's a little agent you can actually install on your EC2 instance and it just shuttles your game server logs to S3. A great way to just, if you want a simple solution, just use CloudWatch Logs and then all of your logs will just be put into an S3 bucket. You can then attach that S3 bucket to a piece of post-processing software. Maybe you can write a script that pulls those logs down and processes them. So it's a great way to get your log data off. Another more advanced pattern that some game companies use when they have thousands and thousands of servers, so you probably wouldn't need to do this to start but something to keep in mind, is using our Kinesis service, which is our managed transport. What Kinesis does, the advantage there is you can actually send a given stream of data to multiple places. It's very popular for stuff like game analytics. A lot of people will connect multiple different game clients and game servers to Kinesis, collect all of their game analytics like who died where, etc., and then they'll send the Kinesis data stream maybe to our Redshift data warehouse or to S3 for longer term. You can do the same thing with your game server logs. We have an adapter for Fluentd, which is a very popular Linux daemon, and you can send that stream of data to S3 or Redshift again. This is a little more advanced. You probably wouldn't do this necessarily when you're first launching the game, but as your game continues to grow, you can always add these features on. That's one thing I want to emphasize that I think is most important about AWS. You don't have to design everything up front. You can start small. You can just build that HTTP core and then you can keep adding these features as your game grows. So don't feel you have to design everything and think of everything up front. That's a great part about AWS. You can just use these building blocks and keep adding them on as you go along.

Two final things to think about and then I'll wrap up. The first one is how to actually do server location around the world. If you've been thinking and following along, you might say to yourself, okay, this pattern makes sense, but how does that main HTTP layer actually know where all my game servers are? If my game servers are coming and going, if my game servers are booting up in different regions around the world, how do I actually know where to send a user? The answer here is what's known as server registration. There's a lot of details here, so I don't have time to go in super detail, but I did want to share it with you guys. I'll go through it quickly and feel free to take pictures and grab me after the talk as well. The basic idea is when your servers boot up, perhaps as part of that Chef step for example, they boot up and they then register with your main HTTP service. Remember, it's a web API you're building, so there's no reason you can't have other game servers contact it the same way that you would have game clients contact it. It's the same idea, it's just HTTPS. So when your game server comes up, it makes that HTTPS call saying, okay, I'm a new game server, I'm available and I'm in the Tokyo region. Then you replicate the same pattern. Say, okay, I'm a new game server, I'm available in the Oregon region. Then you have your matchmaking server, that's your HTTP layer, and it has that map of servers. So when users come in, you can then direct them to the appropriate server to communicate with. To really make this super slick, advanced if you want to call it, you can have a background job that's just monitoring and doing any kind of cleanup. Maybe you have servers that went away that didn't properly unregister. You could just have that running in the background and looking at that same DynamoDB table in this example that has your server list, and then it just makes a call to the EC2 API and can just clean up those instances. I'll pause here. Feel free to take a picture of this. This is kind of the completed diagram. Feel free to take this. These are some details that I would recommend if you're doing server registration. You're going to want to include details like the server's public IP address, what port the server is listening on for the game server process, and then maybe if you have different game modes. Maybe you have a game that is running Knights vs Samurai, maybe we have different maps of different parts of the world. That would be information you would include back as well from the game servers when you're registering back to the main matchmaking service.

Last thought is China. Obviously China is a huge market, very hot, everybody wants to get in there for obvious reasons. It's a massive exploding market. There are a few challenges with China obviously because you do have that network firewall. It's really hard to build services that traverse that firewall in a reliable manner. But luckily, and hopefully this drives home the advantages of the service-oriented architecture, luckily this is able to actually overcome this by following these patterns. The best way right now is to deploy these same services that we deployed through the rest of the world, just build another set of them in China. Have a separate set of HTTP services that Chinese players can use to login to, and then China-based game servers. You can use our AWS China region here for example. It's the same set of APIs. That's a great thing about our AWS regions worldwide. They're always the same set of APIs. All you do is you just change the region. So once you build it one place, you can deploy it another place just by changing the region. The best way to do it right now is to deploy it into China. Then when you come to think and you say, well how do I still give that global experience? How do I still let players feel like they're part of a global community? Well, you can just have a global leaderboard service. You have your separate services that you've built out that are all independent HTTP-based services. You can just put your leaderboard service in some place globally, maybe you put that in Oregon for example. Then all of your different game clients can communicate to that. Again, that's the big advantage of the decoupled architecture. It enables you to scale on a global basis and give the best experience for your players. With that, I'd like to wrap up. Thank you very much for your time. It's great to be here. If anyone has questions, please do grab me afterwards.