Keeping up quality once your game is live

As we’ve seen in the previous article, it is becoming more and more difficult to develop large games and release them bug-free. Another challenge developers are facing, particularly on mobile and now on other platforms, is releasing constant updates to a live game.

This can go on for months or years, and throughout this time each change and addition needs to be tested to ensure it’s not broken and also doesn’t break anything already established in the game.

These updates aren’t just on a monthly basis; they can be weekly, too. So keeping organised and planning ahead is paramount to ensure a smooth release. The world’s most popular games, such as League of Legends, DOTA 2 and Clash of Clans, have regular updates and are played by millions, and small issues could quickly become big problems.

Plan ahead

Bigpoint’s director of live games Nina Müller says for its titles, such as DarkOrbit Reloaded, the weekly release cycle means the QA teams have to keep agile in terms of planning and execution, and they need to be sizing and preparing for other upcoming releases.

“In many occurrences there are multiple updates that are being developed in parallel,” she says.

“This can add a lot of strain to the QA team and we do a lot to minimise this strain by creating good relationships between the code, production, design and QA teams to make sure all the stages of the development are seamless and nothing unexpected occurs. We are also implementing QA at much earlier stages of development to get a forward-look at any potential issues that may arise.”

VMC’s manager of live game operations Ryan Cloutier echoes Müller’s sentiments that releasing new content in a live environment can increase the testing requirement exponentially, as each release needs to be verified against existing content. It’s important to discover bugs quickly before players find them and exploit them for an unfair advantage.

“Quickly identifying and addressing user feedback is why the hybrid QA/live game operations support model is increasingly relevant,” says Cloutier. “QA and live game ops have a cyclical nature – as need for one rises, need for the other eases – so the hybrid model allows resources to focus on the issues that will keep people engaged with the game. The hybrid model makes these challenges more manageable.”

Eternal QA

The adoption of ‘games-as-a-service’ is completely changing how games are made and their longevity. Rather than throwing them over the wall like traditional, top triple-A releases, developers now work on games for years, steadily building their communities.

This means keeping up the game’s quality and providing a consistent, smooth experience is of paramount importance. Testronic’s head of QA Erik Hittenhausen says there are significant risks associated with breaking a live environment, including the negative social media that can blight a game – even if its problems are only temporary.

“QA needs to be vigilant for ever-changing platform holder requirements and the hardware landscape; as long as your game is live you need to continuously monitor these things,” he explains.

“You may have been compliant and compatible when the game first launched but that can quickly change. If a user has a problem running your game on the latest and greatest smartphone, in their mind it probably reflects badly on your game rather than their cutting edge device.”

He adds that continuously testing throughout ongoing development is critical, and offers some advice.

“They key is to look for efficiencies in our QA approach; what aspects of the game could be affected by the changes in this update?” he states.

“While the core features and functionality of the game should be tested and re-tested during every release cycle, it may become impossible or simply not economical to exhaustively test all content in the game for every release cycle. Understanding what content is new, changing or could impact existing content and designing your test approach accordingly is very important.”

In many occurrences there are multiple updates that are being developed in parallel. This can add a lot of strain to the QA team.

Nina Müller, Bigpoint

Despite all manner of testing, sometimes bugs can still make it through even the most stringent of QA, and it’s important to react quickly. Hittenhausen says it’s important to take the quality of your game as seriously as you would take your paying customers.

However, Cloutier says that while it’s better to address problems as soon as possible, it can be problematic to constantly issue hotfixes instead of patches. He also stresses the importance, particularly for competitive and eSports games, to address player concerns.

“If you want to be taken seriously in eSports or as a competitive game, you have to have the consistency to maintain a level playing field,” he says. “Customer expectations are higher than ever.”

Assuring quality

QA firms have had to adapt to these industry changes, ensuring they have enough staff to test games before and after release. Enzyme’s senior project manager Steve Paquin says one of the main pillars for live game testing is the customer support for which testers with necessary knowledge of the game will provide post-release support to end users.

“Users will enter tickets in a database and the customer support testers will usually have access to the back-end and will be allowed to fix most of the cases that don’t require coding support of anything regarding currency,” he says.

“We also have focus groups with teams composed of very specific profiles that will help and provide guidance to game developers and publishing about aspects of their games to hit their targets. Aspects could be their business model, the in-game monetisation, user retention, game balancing review, etcetera.”

Cloutier says that, in addition to QA, VMC offers in-game support, community management and global beta testing. Its global beta test network is designed to test a game in real-world conditions.

To be taken seriously in eSports or as a competitive game, you have to have the consistency to maintain a level playing field.

Ryan Cloutier, VMC

Testronic meanwhile has launched a new Smart Support service, designed to provide constant customer support and feedback to the publisher on core in-game issues.

Even studios like Bigpoint are adding their own new internal measures for QA. Last year the firm implemented a Community Bugging Platform that is available to board admins and moderators to enable quicker reaction within QA from community found issues.

Crowd-testing

But while all these QA firms have been expanding their services, why not just use player feedback to identify and fix issues? For example, Steam Early Access has proven a great method for developers to build a game that players will want to play, at the same time getting player opinion on the key issues affecting them.

Paquin admits that no amount of testing will ever beat tens of thousands of users, but says QA professionals will be able to make sense of the fire hose of information.

“As the industry evolves, developers are starting to use more complex tools to generate metrics and logs that will provide them with important details that will help fixing the issues,” he says.

“First parties have very specific and severe standards that need to be respected and tested using development kits that are not available to the public. While the players can provide feedback and also report issues, especially during open beta testing, the QA support and expertise will always be sought by development and production team as they are reliable sources with defined methodology, technical test approaches.”

Pole to Win marketing manager Regina Walton says there is a key difference between players and those that service providers hire.

“Feedback is part of quality assurance and should be shared proactively,” she says.

“The best testers will have to detect as many defects as possible and be able to describe quickly and clearly. Achieving this is not a simple task and sometimes takes years to master.”

—

For all the news straight to your inbox, sign up to the Develop Daily.