MCV/DEVELOP News, events, research and jobs from the games industry
Last month we introduced the growing area of user research, and its key aim of improving game quality.
This month we discuss a seven step process which will guide you through running your own user research sessions.
So how do we do this user research? Unfortunately there is no standard process which we can just simply apply, it will mainly depend on firstly identifying which questions you want to answer about the game, then choosing the right methods which help to get you those answers.
As a ‘rough guide’ to conducting user research, let’s outline a process which could (and probably should) be used at each phase of game development.
1. What do you want to know?: Are you interested in testing the player’s reaction to a new feature or control technique? Perhaps you want to compare different versions of the build? It may even be the case that you don’t know; you just want to keep an open mind and see what a player’s first reaction is to your game.
2. Design the study: This stage is of critical importance. Choosing the methods you’re going to use will dictate the data you’ll have to analyse later on. If you haven’t captured the correct data, you may not be able to answer your original questions. Some of the most commonly used methods are:
Focus Groups: These are ideal at the start of projects, even before any code is written. They involve open discussion with groups of players, and are most useful for brainstorming and getting feedback to questions such as, ‘which features would players expect from a game of a specific genre?’ or ‘How would you expect to control a game which used this type of controller?’
* Advantage: They can help to inform a design direction and save costs by perhaps not creating unwanted features.
* Disadvantage: Can be difficult to include all people in the discussion equally.
Observation: This involves directly watching the player interact with the game. You could be in the same room, or observe them remotely with video cameras or through a one-way mirror.
* Advantage: A straightforward process to setup and run.
* Disadvantages: If dev team are in the room, players are probably not at ease. Also, unless you’re capturing the video data, if you don’t observe an event, you’ve missed it.
Group Playtest: Similar to observation, only with groups of players in the same room.
* Advantage: You can increase the number of players you test your game with.
* Disadvantages: With more players to observe, you may be missing more findings than you’re seeing. Players also may influence each other consciously or unconsciously.
Think Aloud: This involves asking the player to talk out loud while they play the game, the aim being to get inside their thinking process ‘in the moment’. You can talk to the player during or after the session as you wish.
* Advantage: It reveals insights into the game that are otherwise difficult to capture.
* Disadvantages: It may be unnatural for the player at first, talking about one’s thinking process is not normal behaviour.
Post-analysis: In contrast to players talking whilst playing the game, you could get them to talk after the game session. By recording the game video, you can take a note of moments of interest during the test, then replay those sections and ask the player what they were thinking.
* Advantage: This allows you to time gameplay sessions.
* Disadvantage: The player may not perfectly recall how they felt at the
exact moment.
Biometrics: Currently this represents the cutting-edge of understanding player’s behaviour. One of the key problems with all of the above methods is that players may tell you what they think you want to hear, or give in to peer pressure. Biometrics is the science of capturing and analysing signals directly from the players body, the most commonly ones used in game user research are GSR (skin response or sweat), ECG (heart rate), EMG (muscle movement) and EEG (the brain’s electrical activity).
* Advantage: Can help reveal a player’s physiological state and the player experience.
* Disadvantages: The sensors need to be attached to the player which may not feel natural at first. Also, a player’s body signals are unique, and so you need to baseline each player before the test begins.
There are many methods that can be used to deliver each of these approaches. As choosing the right methods is critical to the quality the results, user research is typically conducted by those with research degrees (Masters or PhD’s). If this stage isn’t done correctly, then the results may be in question.
Finally, once the experiment is designed, do a dry run. Catch any design errors early.
3. Recruit the game players carefully: If you test with the wrong people, you’re wasting everyone’s time. Some guidelines: do not use friends, family, and especially not colleagues, as they are more likely to tell you what you want to hear, or already know too much about the game. Also, pay them: users are more likely to provide quality information if they feel their feedback is valued.
Finally, do not re-use people. In most cases you’ll want to capture the users’ first thoughts and reaction, and this is lost if you re-use. However, you may want to bring players back for studies over longer periods of time or when comparing between builds.
Don’t always think that testing with more players is better. Focus on the what you want to know and choose the number of players accordingly. For example if you want rich qualitative data, then testing with eight to 12 people may be enough. As a rough guideline, once you start seeing the same findings starting to re-appear, you’ve probably tested with enough players.
4. Conduct the playtests: By this stage the player is in the test lab and you’ve briefed them on what they’re about to do and what’s expected from them. This is where the majority of data is captured. As a minimum you should take notes, preferably time-stamped, but you may also wish to take video of the game, the player’s face and body position, the buttons they are pushing, audio and biometric data from their body. It would be advisable for at least two people to run the study, one to interact with the player and another to focus on note taking.
5. Analyse the data: There are two main ways to analyse the findings – quantitative and qualitative – and it’s usually not an either/or approach; it’s both. Quantitative is probably the place to start: for example, if eight out of ten players had difficulty with a key feature of the game, then you should examine that in more detail. This is where qualitative analysis comes in, which will help say why those eight players had difficulty. Quantitative approaches may include time taken to complete a level, number of deaths/kills/restarts or number of times the player had to ask for help.
Qualitative analysis will typically mean meticulously going back through all recordings to explain why players where experiencing difficulty.
6. Write the report: This doesn’t have to be a formal report; you can just send out a brief e-mail after each day’s user testing and follow it up with more substantial detail. For quantitative-driven reports a spreadsheet may be the most useful way of reporting the results, but a more traditional report with annotated screenshots or links to videos may be more suitable.
Depending on the data you are trying to represent, other forms of data visualisation might be more effective, such as heatmaps as used by the Halo 3 and Half-Life 2 developers.
7. Communicate findings and take action: Make sure that the entire team is informed of the results and show evidence of them. If you have video data then use that. It’s difficult to argue on the design of a contentious feature when video evidence shows that no one can figure out how it works. Communication with the whole team is critical as this is the stage that enhances the game quality.
This process is only a guide: within each stage there is a minefield of further techniques which can be used to help deliver the best results. In terms of when to do user research, starting early and iterating often offers the best chance for improving quality and reducing development costs.
The economics of games are changing. With episodic gaming likely to become more popular and downloadable demos available, buyers can sample fragments of your game and if the experience is not what they expect, the majority of a game’s revenue potential may be lost.
Something to consider: how much would you pay to increase the quality of your game rating by one percent? How about ten per cent? With the average cost of a game going up to around $28 million, this puts the total cost of user research on a title at around 0.1 per cent of the budget.
When done correctly, user research should provide a morale boost to the team, strengthen the brand of the studio and publisher and perhaps most importantly, help to increase sales and deliver a high quality title.
——–
Dr Graham McAllister is the director of Vertical Slice, the first UK company to focus specifically on video game usability. He is a senior lecturer in human-computer interaction at the University of Sussex and is also an Apple Distinguished Educator.
www.verticalslice.co.uk
