Finding the fun: How Midwinter Entertainment improved Scavengers gameplay with rigorous testing

It’s 2am on launch night and you’re in the war room. Players all over the world are experiencing your game for the first time – tomorrow, the first reviews will be all over the web. But the game is struggling to scale. You turn to the runbook, but it’s clear almost immediately it’s out of date. What do you do?

[This content was created in association with Improbable]

It’s a game developer’s nightmare, all too often turned-reality. For smaller studios that don’t have access to the infrastructure of larger companies, rigorous testing ahead of launch is a constant challenge. How do you get sufficient hardware to test your game? How do you effectively scale-test your systems? Where do you get the resource to manage and build your deployment process?

How Midwinter Entertainment pulled off the Scavengers Early Access launch with minimal backend issues

Thanks to a tactical collaboration, the Scavengers Early Access launch avoided “a lot of the classic challenges,” according to Midwinter Entertainment’s Vice President of Technology, Eric Matteson.

“[Working] with IMS (Improbable Multiplayer Services), we brought up and down multiple regions on the day of launch and split regions in two,” he explains. “We were able to respond to customer demand almost instantaneously – bringing up regions where we had a higher population than we thought we were going to have, so that we could place game servers and lobby servers in the location where players could have the best latency experience.”

Eager to know more, we caught up with Eric and Jared Dixey-Hefty – Principal Engineer for Scavengers Live Operations – to find out how their teams worked together to improve Scavengers’ gameplay and pull off a successful launch.

Parallel testing got Scavengers to market quicker

Parallel testing meant Midwinter Entertainment could deploy builds in clusters without impacting other builds. By maximising the number of iterations in development, they were able to test multiple scenarios simultaneously and fix things faster. Not only that, it was also more cost-efficient because the same machine was used for various tests.

“In what we call the development environment, it’s kind of the Wild West. Developers can actively make changes – they’re all running alongside each other in an isolated fashion, solving the problems they need to worry about, the things they need to iterate on quickly… That day-to-day experience gives them a nice way to validate,” says Jared.

“Games are fuelled by your ability to understand, diagnose and respond to real player issues and real player updates”

Eric Matteson – Vice President of Technology, Improbable

Daily playtesting allowed them to see through players’ eyes

Because the internal infrastructure available via Improbable Multiplayer Services (IMS) matches the infrastructure deployed for live production, Midwinter Entertainment were able to constantly playtest internally from early on in the development process – and get realistic results. They also used ‘sim players’ to load-test the system from end to end, giving them the inside track on what real players would see.

When they came across sticky issues, cloud-based infrastructure and global support teams in opposing time zones meant engineers could work in their sleep – or, rather, work got done while engineers slept.

“Games are fuelled by your ability to understand, diagnose and respond to real player issues and real player updates,” explains Eric. “The only way to evaluate a complex, large-scale multiplayer game like Scavengers,” he says, “is to touch it, to feel it, to play it – and to play it the way that players are going to play it.”

Red team and blue team QA testing kept runbooks up-to-date

When development is moving fast, it’s all too easy to assume things are working smoothly – simply because you haven’t been alerted otherwise. That’s why they took a two-pronged approach to quality assurance, tasking a ‘red team’ to act as saboteur. “The saboteur’s job is basically to go through common failure points, or dustier areas of the platform that we haven’t really tested as recently, and make sure that stuff is still up to scratch,” explains Jared. Meanwhile, the ‘blue team’ is tasked with fixing the mess they (intentionally) make.

Working on Scavengers, the red and blue teams identified out-of-date runbooks, important alerts that didn’t fire (or fired too often, which any engineer will tell you is almost as frustrating) and systems that didn’t scale as expected. All were caught in time to be fixed ahead of launch – thus avoiding potentially expensive mistakes.

“Players will always do something that you never expected. They will find another way to play the game that’s very different than what you thought.”

Eric Matteson – Vice President of Technology, Improbable

Community playtesting generated invaluable gameplay feedback

Live closed beta tests with the community meant tens-of-thousands of people played Scavengers prior to launch, rather than a hundred or so in-house. This allowed the team to stress-test more factors, as well as identifying areas for gameplay improvement.

“Players will always do something that you never expected,” says Eric. “They will find another way to play the game that’s very different than what you thought. And without that interaction with customers, with early playtesters, experiments with gamers, you don’t have that opportunity to get that information that allows you to craft a great game.”

Extra support enabled fast updates after Early Access launch

“The world in which you release a game and that game is the finished product – that’s gone,” says Eric. “Now, really, your game development process starts when you start active interaction and constant interaction with players.”

Midwinter Entertainment were able to update Scavengers in four days after Early Access launch – including fixing crashing issues, making quality of life improvements, and implementing a significant balance change. “Games are fuelled by your ability to understand, diagnose and respond to real player issues and real player updates,” says Eric. “Being able to do that in four days is a tremendous advantage.”

Turning your Achilles’ heel into your superpower

Sometimes it’s lack of experience in running online multiplayer games at scale that’s a game studio’s Achilles’ heel at launch. But often it’s just that they’re “focusing all their effort and energy into the game itself,” explains Jared. A studio’s prerogative, surely?

Far from being a vulnerability, obsession with gameplay should be a superpower in such a competitive space. Scavengers is proof that, with a bit of tactical collaboration, it can be.

Find out more at improbable.io

About Seth Barton

Seth Barton is the editor of MCV – which covers every aspect of the industry: development, publishing, marketing and much more. Before that Seth toiled in games retail at Electronics Boutique, studied film at university, published console and PC games for the BBC, and spent many years working in tech journalism. Living in South East London, he divides his little free time between board games, video games, beer and family. You can find him tweeting @sethbarton1.

Check Also

Submissions are now open for Develop:Brighton 2021 Indie Showcase

Developers can submit their games until Friday, September 3rd.