Keep it simple for your first game. For quite some time I’ve been pondering a new multiplayer server technology and netcode architecture. After spending 15+ years in the gaming industry I know this is no small task - there’s a graveyard of game studios and aspirational game devs alike who have tried and failed to write good netcode. In fact, I was at a startup that was plagued with network issues that they spent loads of money and manpower (myself included) to figure it out.

But I wouldn’t be writing this devblog if I hadn’t 1) had a plan and 2) already got it working. (:

I know what you’re thinking. It’s ambitious to come up with a new networking/server topology for my first indie game, but it’s definitely the only complicated or big idea I’m throwing into my first game, I swear. Except for the custom proceedural generation… oh and inventing a new art style without being an artist… and a few other things we won’t mention here. Yet.

But here’s hoping my career experience will give me a leg up here. Oh right… given this is my first blog post, I should probably introduce myself.

whoami

My name is Brandon - alias: Porschiey. I wrote my first website by hand in HTML w/ the help of my dad in 1998. Since then the most important thing I’ve learned is that the only constant is change, and that I must keep a student mindset, or I risk being left behind. Perhaps in another post I can detail my career in more detail, but the short version is that I spent time freelancing, then at Xbox, Azure, a large game studio (343 Industries), a small game studio (Starform), and now I’m back at Xbox yet again working on emerging technologies. And on the side, I am working on a new indie game called “Parchment” - a 4X persistent world fantasy game.

where were we?

Oh yeah that’s right, a new game server / multiplayer / netcode design. Well, if I’m honest, I’m not quite sure if it’s new - I’ve not been to every single game studio and asked them about their (probably secret) proprietary tech. It’s likely more accurately described as a collection of patterns, practices and ideas from different systems I’ve seen in the past 15+ years. I will be attempting to explain how it all works generally, without getting too technical, in the hopes of keeping a wider reading audience but also to keep some secret sauce for myself.

simulations that aren’t transient

When you play a traditional RTS or 4X game, the world exists only while you’re in it. You log out, the simulation stops, and the universe blinks out of existence until you return. Parchment isn’t like that. Every realm keeps ticking — even when you’re offline — evolving under the stewardship of both human and AI civilizations.

Designing that kind of persistence meant building a server model closer to an MMO, but one tuned for asynchronous strategy, not 100% twitch combat.

introducing your personal “bee”

When you join a realm, your own personal “bee” spawns in the hive of bees. This bee takes in all of your requests, validates them, does most of the heavy compute work, and ensures all your input is handled sequentially or in order. It is not bothered by what the other bees are doing, and it will always go to a honeycomb that is least busy. (For the technical folks out there, this is the Orleans Grains/Silo technology, aka Actor Model).

When the bee has finished a unit of work, it delivers it to the queen bee, who in turn distributes information about that unit of work to any others that care. In this way, the queen is in charge of the realm. This “actor model” concept isn’t a new thing in the software industry, but something that’s rather special about how I’m doing it is how seemlessly and quickly it scales.

As you can see, for any given realm that’s running, the work is spread over multiple servers. This gives us several wins:

  1. A “hot” realm is never on one server alone. “Cold” realms automatically make space for the hot ones.
  2. A simulation error or issue doesn’t have to crash the realm, just your worker bee. Most game servers have one ferociously pumping update loop that any one single exception can bring the entire thing crashing down.
  3. The primary “update loop” is automatically multi-threaded across several servers and doesn’t have to be concerned with large compute or physics tasks like pathfinding or terrain generation.

multiplayer games are classicly very “chatty”

The standard issue multiplayer dedicated server is noisy and busy. Every single frame of the simulation the server is sending updates about the game (typically game state or partial state updates) to every connected player. At a 20FPS simulation with 10 connected clients sending input, the server is sending/receiving hundreds (often thousands) of events over the network per second. Granted, this is a mediocre design scenario, many modern designs are better than this - but even the best ones are transmitting hundreds of updates per second for only ten players.

Rather than flooding the network with constant position updates like a shooter, Parchment communicates in orders, not transforms. For example, the client sends a “move to” command; the server validates, computes a path, and sends back waypoints. The client then interpolates locally and reconciles occasionally.

Bandwidth is tight and tidy. A single move command takes a fraction of what it would in a typical multiplayer persistent world server. World of Warcraft, for example, clocks in at 21KB/s per player average according to this academic source (and that’s fairly low). Parchment will hopefully sit between 0.5KB/s or 1.5KB/s.

This opens up so many exciting possibilities:

  1. Parchment as a mobile game isn’t going to frustrate players for consuming mobile data
  2. More realms can fit onto a single server, drastically reducing costs to maintain realms. The current ambitious goal is to have a single server have the capability to serve 10,000 concurrent players (100x realms per server, 100 players per realm). Web servers can do this easily, but game servers… I rarely see a single game server (VM) that host more than 1000 players. (In fairness, some notable callouts that can include Minecraft, PlanetSide).

I haven’t even covered some of the best parts

That’s all for now… this blog is now getting a little long, so I’ll save some of the more interesting parts for next time. (:

Next up:

  • Keeping time in a persistent world
  • Fast-fowarding and parking
  • Snapshots & per-realm upgrades