Standardised testing: Why QA/Loc firms are calling for best practices

Go to any conference or event about QA and localisation and you’ll often hear mutterings about the need for standards.

With so many high-profile, triple-A titles suffering from a myriad of bugs and server troubles at launch in recent years, much of the quality assurance sector is calling for just that: quality.

“We as an industry need to be accounted for with the way we are releasing unfinished products,” says Testology CEO Andy Robson (pictured). “We wouldn’t be happy if we watched a film and the last 30 mins were missing. So why do we think it is acceptable to release games that don’t meet the quality level consumers expect? 

“We should have a standard where no Class A bugs are released in a product along with Class B bugs, whether functional or LOC issues. Class C bugs are always going to be in games, but don’t affect the experience, so we could be more lenient.”

Loreto Sanz Fueyo, director at Universally Speaking, agrees: “QA and localisation need to come together to offer standard guidance to help make the process smoother. Establishing these standards and integrating them deep within development would save time and costs, ensuring a higher quality product and happier players.”

Pole To Win’s senior QA director Katsuri Rangan observes that platform holders already have compliance standards, although even these can vary – particularly on mobile.

Why do we think it is acceptable to release games that don’t meet the quality level consumers expect? 

“The game-changer lies in the enforcement of a standard across platforms,” he says. “The Android submissions team could be more strict about enforcing their standards. Apple seems to be a tougher platform to release on, as they tend to be stricter on the specs and quality of apps.”

Many QA and localisation firms hold themselves to their own, self-imposed standards, which should help devs improve the quality of their games.

“An experienced partner will be able to walk you through each step of what sometimes appears to be a daunting process,” says Sanz Fueyo. “However, by the time that guidance is requested, it is often too late to make a significant difference. Speak to your partners early on and often.

“When we are working on test plans there are a set of common elements we look out for. These form part of our internal checklists, which range from ‘simple’ debug options to design elements and their impact in testing time. We repeatedly see similar issues we’re used to resolving ourselves or recommending the best course of action to studios before we can begin.”

We asked experts about the standards they believe the industry should hold themselves to. But Robson says that there needs to be consequences, not just compliance.

“We should have a set amount of time – I’d say 144 hours – to test the final build of any game,” he says. “If major issues or gameplay flaws are found then it should be failed. If this delays the game by three months or so, then so be it. Games should not be released because marketing says so or the teams get bonuses for meeting deadlines. We all know this goes on and I’d love someone to tell me differently as they’d be bullshitting me.

“It’s time for us as an industry to come together and form a process all games have to adhere to. Then we’ll release quality games that our consumers are happy with. Isn’t that why we all make games?”

QA/Loc Industry Standards: What should be included?

QUALITY ASSURANCE

Live game release candidate update testing
Putting out consistent high-quality updates and patches is integral to a live game’s success. Issues reported by QA should be considered against both the monetary impact of the issue going live and the need for a fix, while considering the impact a poor update has on brand integrity.
Kirstin Whittle, VMC

Standard test management tools
Usage of systems such as TestRail and DevTest in games QA is not as widespread as it should be, with many teams still relying on clunky, static Excel-based checklists for executing test cases and tracking coverage. Wider adoption of test management will bring increased agility in test planning, efficiency gains in test execution, and enhanced reporting and metrics.
Marc Kent, Testronic Poland

‘Transparent-box’ testing
Having some basic development abilities within the team – such as downloading repositories, compiling the build and launching/debugging as devs would do – would enable a high-quality approach for testing, allowing ‘to the line’ bug-reporting in cases such as null pointers, asserts or exceptions. You could even have remote sharing to let the developer take control of the debugger to handle the situation themselves.
Emilio Cazorla, Lollipop Robot

Closed beta testing
Conducting closed beta tests on production environments provides publishers and developers the data measurements required to validate whether their game is ready for a successful day-one multiplayer release, and subsequent iterations when applicable. 
Kirstin Whittle, VMC

A platform-independent ‘best practices’ guide for finding usual bug suspects
It would be nice to have a list of common items to evaluate, like “Has your game’s GUI has been tested in every possible resolution?” and so on.
Emilio Cazorla, Lollipop Robot

Maintaining up-to-date design documentation
With options for collaborative documentation from Google Docs, free or cheap wiki creation tools, and licensed software such as Confluence, it’s easier for a large team to keep track of all changes in a single, up-to-date format. This provides a fast, reliable way to bring new or replacement staff, remote workers and vendors who join the project partway through up to speed.
Edd Buffery, Testronic London

LOCALISATION

String files to contain additional information and context
Information such as where the text appears, who is speaking to whom, and the name, age and gender of the character makes it easier to provide a better contextual bridge to localisation.
Orad Elkayam, MoGi

A standardised file format for in-game text
This is would lower costs relating to file parsing and negate the requirement of supporting tons of different formats in each and every tool along the localisation chain.
Katsuri Rangan, Pole To Win

Adopting MQM (Multidimensional Quality Metrics) and DQF (Dynamic Quality Framework) 
The widespread adoption of these modern frameworks would allow the games industry to create a common understanding of the issues and the way of measuring quality, all together bringing a positive effect on everyday operations.
Fabio Minazzi, Keywords Studios

Ensuring source file/language is clean and consistent
Fixing errors, confirming the meanings of sentences and having a source file that is messy and with errors will cost more time and might transfer problems into the target language as well.
Orad Elkayam, MoGi

Moving away from strings housed in offline files
Translation shouldn’t be performed in isolation and all translators – and key stakeholders – should be able to collaborate with the translation effort. Offline files do not allow translators to easily view other languages and share queries on strings. Secondly, if changes are made to source text whilst translations are being worked on then these have to be manually tracked which creates a greater potential for error.
Katsuri Rangan, Pole To Win

ALTERNATIVELY…

Nothing
Every firm has a different way of working, planning, executing and so on. Homogeneity is not only boring, it’s dangerous. Companies should actually try to think outside the box before standardising, before simply doing what their neighbours are doing. The more standards are put in place, the less room there is for innovation, learning and doing things differently.
Mathieu Lachance, Keywords Studios