Tribal Wars is a browser-based game from German developer InnoGames. Mixing classic strategy with ease of play, the game was first released in 2003. Since then, it has been in active development as web browsers, coding technology and user habits have evolved at a frightening pace. Jon Dawson has been with the team for the last ten years and he talks us through some of the changes and problems maintaining a browser-based game presents to a 14 year old game.
"When the game was first made available to the public in 2003, the world of web development was a very different place," he explains. "Internet Explorer had a market share of 95 per cent, HTML5 had not yet been imagined and interactive media online was limited at best.
"From an infrastructure perspective, the game began its life on a server sitting in the room of one of the founders. Shortly after it was moved to a small number of servers rented from a local company, each having one Pentium 4 processor and 2GB of memory. At this point, the game was developed by only one developer, an artist and a designer.
One of the first servers to run Tribal Wars. It’s now used as a foot stool by one of the system administrators.
We recently had one Polish world with so many active players that during peak times we had to handle over 120 player attacks arriving every second
Jon Dawson, InnoGames
GROWING THE GAME
"By 2009, the team behind Tribal Wars had expanded to our peak size of 6 developers," says Dawson. "At this point, the game had reached millions of monthly active players and performance struggled. This lead to quite an unbalanced setup because players would flock to the latest released worlds leaving very few on older worlds. Our latest worlds would struggle to cope with the high database load from so many concurrent users."
By 2011, the market had changed yet again. The browser game market began to evolve more towards highly graphical games and Tribal Wars reached its peak number of monthly active players.
"For several years, we had offered a slimmed down version of the game for mobile web browsers and in 2011, decided that it was time that we offered our own native apps for iOS and Android, says Dawson. "While the game was certainly playable in your browser, it became clear that mobile users expected a more streamlined experience within an app. Push notifications about events on your account like a new incoming attack are very important in a real time game where missing vital information like this for a few hours can be the difference between a successful victory or defeat.
"The first version of our iOS app consisted only of native registration/authentication, a nice menu, a webview, push notifications and of course, mobile payments"
The Tribal Wars is now smaller than at its peak with eight members across community, design, system administration and art. But the game continues to evolve despite the challenges that come from working with old code.
"We can deploy a new version of the game to every server in under 5 minutes, and try to deploy all finished content at least one a week," says Dawson. "We often do multiple deployments in a day. We have fewer players now than we did 5 years ago, but have a much more complicated infrastructure and so must still be careful about performance and balancing. Our largest scaling challenge remains: most of our players play on our most recently launched worlds. 90 per cent of players are on our 10 largest worlds. We still sometimes hit the vertical limits of our scaling strategy.
"We recently had one Polish world with so many active players that during peak times we had to handle over 120 player attacks arriving every second. The logic for calculating the results of one attack can be quite complicated, so scaling to handle this throughput was very challenging for us.
"In the world of Tribal Wars it’s extremely important that every action is processed in the correct order which means it’s not as easy as handling every task in parallel. For example, if 20 players in a tribe have all sent 50 attacks to one player’s village then we must make sure the result of each attack is consistent with the output of the previous attack, down to the very millisecond. We’ve noticed that the players of our Polish worlds are usually the most active, and so we have to adjust server capacity to accommodate that.
"Due to our small team size, we’ve found in many cases it’s better not to touch something if it isn’t broken and we don’t need to make larger changes to the surrounding systems. In other cases, we’ve ripped out whole features and created them again from scratch."
You only need to look at the last five years to see how much has changed in web development. Flash has gone from its almost omnipresent height to being replaced by HTML5. Preferred browsers have come and gone and macros and plug-ins constantly dominate ad revenue meetings. Even the game has moved on with a sequel, Tribal Wars 2. But some things never change and the simplest solution is often the best.
"Having personally worked on Tribal Wars for nearly ten years now as a cross functional developer on backend, frontend, mobile and basic server ops," says Dawson. "One of my key learnings has been that you don’t always need to refactor old code to be able to add improvements to a legacy game. There have been many defining moments when we could have stopped what we were doing to focus only on improving code quality and potentially spending months on something that wouldn’t have brought any true value to our players.
"I also think that working on a legacy codebase doesn’t mean that you should be afraid to try new technologies. Don’t be afraid to try smaller experiments like implementing browser notifications. Try larger projects like creating our nodejs backend. Embrace new technologies like HTTP2 as soon as possible.
"It’s better to have something that works than to have something perfect. You can release fast, release early and get feedback for future improvements instead of working forever on something that may have a perfect backend implementation but is hated by your customers. Be proud of what you work on and don’t use its age as an excuse to take shortcuts or settle for lower quality."