Rewriting the rules of QA in games

Those who are not involved in QA might think they understand it: it’s the process in which games are tested for bugs and functionality so developers can fix any issues before or even after release.

Those who are involved know better. QA is not a necessary evil as some claim, but a vital process that can make or break a game and inform the development process about a lot more than just glitches.

During a roundtable discussion hosted by Develop and Testronic, experts in the field revealed the full potential of in-depth QA that some studios might not have embraced.

“It’s not just about reporting hundreds of bugs, it’s about digging into the metrics,” says Stuart Pratt, delivery manager at online poker firm PKR.

“You’ve got to see how many bugs you’re finding on a daily basis and why you found so many in a certain piece of code – you can even determine how many builds you’re going to need before release. This is all information QA knows and should be reporting back to executive producers.”

However, Testronic’s operations director Chris Rowley says that those above the executive producers aren’t always willing to listen to these statistics: “QA’s opinion will often be ignored because development teams tell corporate entities what they want to hear, which is that a game will hit its date. But if QA can show by its metrics that, no matter what velocity they’re fixing bugs at, it’s still going to be three weeks late, that’s irrefutable.”

One member of the roundtable reminisced about a time when the QA and development teams placed bets on how many bugs would be found in a game. Some scoffed at QA’s estimate, double the dev team’s and based on historical data from past entries in the series, but it ended up out by less than half a per cent.

I wouldn’t rule out using QA as a stepping stone. There are good careers in QA but there aren’t enough to sustain everyone.

Jim Woods, Sega West

The panel also said that QA is often overlooked when it comes to additional investment, either in training or facilities – something that Pratt says needs to change.

“Every other industry sees QA as a highly regarded profession and it’s something that we’re catching up on slowly,” he says. “But I’m not seeing it as highly regarded as it is in other industries.”

Time for change

Arran Oakes, head of production at events firm Gaming IQ and part of the team behind the Game QA & Localisation Forum, says attitudes also need to change at the bottom as well as the top: “QA has typically been perceived is as little more than a stepping stone into other aspects of the games industry. One way to change that perception is by having better qualified people coming in, by raising the bar of entry to the field.”

Sega West’s director of development services Jim Woods agrees, but adds: “I wouldn’t rule out using QA as that stepping stone. There are good careers in QA to have, but there aren’t enough to sustain all of the people you bring in, so they have to move on to do something else, and they’re acquiring skills that can be used elsewhere.”

Part of the frustration around these QA preconceptions is that they aren’t changing as fast as the market is. New platforms and business models have seen the process, and indeed the purpose of testing, evolve beyond its traditional definition.

“We’re moving away from testing a piece of software that goes in a box, and more towards software-as-a-service,” explains Rowley. “For firms like PKR, it’s not just testing a game, but testing an infrastructure as well. That requires far more technically minded people than what we traditionally had five years ago.”

Oakes adds: “The mobile industry and free-to-play model that came with it has played a very big role in the way that QA is evolving across the entire industry, irrespective of platform. The demographic of gamers has shifted and tolerance levels to quality have dropped significantly.

“I used to save up all my money for one Mega Drive game and, regardless of the quality, I would be invested in that. I put a surprising number of hours into Shaq Fu, because I’d saved up all year to get it. But nowadays, if there’s something wrong in the first five minutes of a free mobile game, I uninstalled it and download something else.

“That’s the age we now live in, and I think that means while QA may have been historically seen as a necessary evil, today it’s much more of an immediate essential – particularly in the mobile arena where it’s do or die.”

PKR embeds testers and tools engineers into its development teams for its online poker games

The rapid growth of the mobile market is causing other problems for QA teams, particularly for smaller firms like PKR.

“There’s been 139 new Android devices released this year alone, and there’s no way we can buy every single device to test against,” says Pratt. “And we can’t just concentrate on the big companies like Samsung – now there are tablets from supermarkets, too. The Hudl from Tesco sold out over Christmas, so we couldn’t get one and didn’t know if our game worked or not until later. But what are we meant to do?”

Rowley agrees: “The fragmentation around Android is an absolute nightmare, and it’s actually accelerating. We’ve always tried to focus on the top 20 or so of the most popular platforms, but that isn’t consistent across different territories. It’s worse than PC in some ways. And there’s no interest from manufacturers to standardise across devices.”

Testing times

Back in the traditional gaming space, some developers believe they have found a solution to the growing need for comprehensive QA in the form of public betas like Steam’s Early Access. Many indies use this to gather feedback from early adopters, a process that was instrumental in the success of Minecraft, which was originally released as an alpha.

But Woods warns that few developers truly understand how to use this model effectively, pointing to two major issues.

“The first is that, even if you’re putting it out as a beta product, there’s still an expectation that it’s going to work,” he says. “The second is, even if you get it to an acceptable beta level and release it, how does that feedback loop work? How do you let the consumer tell you that there’s an issue?

“You need that ongoing support to make sure all that feedback actually then gets put into a bug database that your developers can do something with. I think the whole alpha and beta release labelling is nonsense.”

Oakes observes that some developers, like Minecraft creator Markus Persson, are able to release games in alpha because the audience understands that it’s a project by just one man, and are therefore more forgiving.

Woods argues: “It depends on what the definition of an alpha is. A complete version of the game is not traditionally what an alpha was. With Sega, we’ve got very clear definitions of what an alpha, beta and release candidate should be based on historical packaged products. But when it comes to games-as-a-service, you kind of rip up the rules and throw them away.

“I think what we’ve actually got to do is write new rules for what happens with games-as-a-service. That’s what we’re in the process of doing at the moment. And you’ve then got to get devs to buy into those new rules, because without that you lack structure, and without structure, you can’t put proper processes around things and you end up releasing things that aren’t ready.”

Rowley agrees, adding that, by definition, beta testing serves two purposes: proving whether what you’ve built works correctly and consumers play it the way you expect them to, and stress testing your server base.

Oakes adds that Swedish games firm Paradox Interactive effectively uses a community of select players as beta testers, but with a crucial difference: “They were very open about that, and they had process: standardised forms and bug reports to be submitted by players. It was a process that, while not flawless, worked.

“Doing alpha and beta tests is not a chunk of your QA time. You should do your QA time and then put those out there to catch those unique things that, no matter how many hours of testing you throw at it, you just don’t catch until real people get their hands on it.”

Automatons

Another way in which the QA process is evolving is the advent of automated testing. PKR, in particular, uses it to ensure that its websites, registration, payment and withdrawal processes are working, quickly identifying any issues that will prevent players from using its service. It has been an expensive investment, but a worthwhile one.

“We can get tools and scripts to cover the work of 100 testers,” says Pratt. “We can leave them running during the day, they cover all our websites, and we can have return on investment within about four weeks.”

It’s not just online firms like PKR. Microsoft and EA already use various forms of automation; in fact, in the case of the former, processes used from other areas of its software business have been introduced into its games division.

“That’s the kind of thinking that’s creeping into the games industry; we’re not a completely unique environment,” says Oakes. “Software is software and there are tools applied in other software verticals that could be incredibly valuable but for some reason the games industry doesn’t know how to utilise them.”

Woods agrees: “We as an industry are definitely behind the curve in terms of looking at that. The trick now is trying not to repeat everybody else’s pitfalls.”

However, there are misconceptions about automation. For one thing, it’s not a direct replacement for traditional testing. Companies that invest in engineers might expect to see an immediate cut in black box testing given the cost of automation and engineers, but games still need to be tried out by human operators to guarantee players will be able to enjoy them.

No matter how many hours of testing you throw at it, you just don’t catch some things until real people get their hands on it.

Arran Oakes, Gaming IQ

Additionally, it requires a lot more communication between development and QA because the latter needs to prepare any automated processes for whatever devs can throw at it – something that becomes complex if QA is forced to update those processes.

“You have to have buy-in from your development teams and they have to have the maturity of their processes to make sure that what you’re writing for your test rigs are robust,” Rowley explains.

“If they either don’t buy in or their own processes are so airy fairy they’re changing all the time, all that happens is your test engineers are just updating their test suite and never actually doing anything else. You end up in a vicious circle.”

To QA or not to QA?

Some companies have found their own ways to have development, QA and automated testing work hand-in-hand. Pratt reveals PKR doesn’t have a separate QA team, instead embedding testers and tools engineers within the development team.

But Woods warns it could be taken too far.

“We had a very in-depth discussion with a developer/publisher and they were very proud to tell us that a particular game they had released,” he reminisces. “It was hugely popular on PC, and they said it had been developed with a very small team and no QA.

“When we inquired a bit more about how that worked, they told us the dev team just played it themselves and fixed the bugs when they found them. There were a number of Sega heads and I could see their eyes lighting up at the prospect of getting rid of QA.

“At that point, I felt that I had to point out that maybe this was the reason that said game had shipped about four years late and several million dollars over budget. Yes, you can develop games without QA but there is a big price to pay.”