1-1 playtesting sessions, where the moderator is sat in the room with an individual player (or small groups playing on one console), are a key part of games user research, and are essential for getting useful usability feedback throughout the development process.
During my time at PlayStation I’ve been lucky enough to work with some very good moderators, and picked up a lot of insight on the best ways of running these sessions to get useful and actionable feedback on games. This post aims to go beyond the basics, and share some insight into the more advanced techniques that I’ve found have greatly improved how we run these playtest sessions, and research games.
Always confirm what you’ve seen
When watching people play, it’s often tempting just to observe, and assume you completely understand what’s going on. In many cases, the problem seems obvious when a player’s actions imply that there is a usability issue, and you feel confident in reporting the finding.
However, there are times when this can be misleading. Because the moderator doesn’t know what’s going on in the player’s head, they cannot distinguish between issues such as “the player hasn’t seen the next objective” and “the player has seen the objective, but is looking for something else first”.
The fixes to both of these issues would be different, so it’s important to ask players what’s going on, to ensure that you are identifying the issue correctly. This also makes it a lot easier for your note-taker, and ensures that they have seen the same issue as you.
The difficulty is asking the players about what they are doing without changing their behaviour. As always, we want to ask non-leading questions, and the easiest way of doing this is to start very high-level and general. For example, ask “What is your goal currently” if you’re interesting in objectives, not “Where do you need to go next?”. The latter implies that they should be going somewhere next, and will change their response and subsequent behaviour.
And as always, don’t interrupt gameplay with these questions; instead wait until the action has died down.
Players have often surprised me with explanations that I couldn’t possibly have anticipated, and so I’d recommend asking players to describe their actions even if they seem obvious. By doing this throughout the session, it also becomes normalised and players won’t feel they are being judged when asked at the critical moments you are really interested in.
Repeat everything. Repeat everything.
This is a really simple technique, but has many benefits. When asking questions to players during gameplay or in the final, always repeat back to them the things they have told you. For example, when the player says “I think the controls are bad”, say back to them “So you didn’t like how the game controlled”? In order to make the conversation more natural, you will probably want to paraphrase what they say, rather than repeat it verbatim.
Repeating the things said by players back to them has many benefits. First of all, it makes it easier for your note-taker, as it ensures that they have heard and understood what is being talked about, and gives them more time to write it down. It also helps check that you’ve understood what they’re describing correctly, and that you are correctly identifying the issues.
Most importantly of all, repeating it back to the player encourages them to give more information. In the example about controls, after I said “So you didn’t like how the game controlled?” players will often follow up with further information, such as “That’s right. The jump button felt unresponsive, and it felt like I had to tap multiple times before the game noticed”. You are then clearer about what the issue is, and can follow up with more questions to identify why the issue occurred.
Rather than just agreeing with you when you repeat their opinions back to them, players will often give more information or clarification to qualify their opinion, which gives you more insight into the usability issue, and helps uncover actionable findings.
Imagine telling the development team
When speaking to participants about issues that have been observed or that they have raised, I always try to think about how the issue will be reported to the development team. It’s important that usability issues are actionable, and that changes can be made based on your report. How this works in practise is that the moderator shouldn’t stop asking for more detail until they reach something actionable.
For example, if a participant tells you the “controls are weird”, this would not be useful feedback. If you reported this to a dev. team, they’d ask more questions such as “in what way are they weird?” and “what shall we do?” As a researcher, you need to know the answers to these questions.
So, to avoid embarrassment, the moderator needs to keep probing deeper, with questions such as “What were your expectations with the controls?”, “how did using the controls differ to your expectations?” and “what did this cause?” When you’ve identified the cause of the issue, and the impact, you can feel a lot more comfortable with reporting this to the team and knowing that it is useful information.
Clarification is especially important for difficulty. As a statement “This was hard” is not useful feedback. Was it too hard? Was it bad that it was hard? What made it hard? The moderator’s job is to continue probing until they have something that can be used constructively by the development team to increase the quality of the game. Thinking about how you’d describe the issues to the team is a great way of ensuring that you are getting everything you need from playtest sessions.
Ensure players are describing their own experience
In playtest sessions, players are often tempted to give feedback based on what they think others will think. I’ve often heard people say “my brother would love this game” or “this would be too hard for kids”. This is because the participant doesn’t understand that we’re interested in their own experience and opinion, not in their expertise as a market analyst.
It’s always important for moderators to deflect these kinds of comments, and turn them into something useful. The easiest way of doing this is by putting emphasis on the fact that we are looking for their feedback. So when a player says “My kid would love this game” the moderator should say “and what do you feel about it?” You may find that they don’t actually enjoy it themselves, and they think kids would like it instead because it’s too easy or simple for them personally. This is much more useful, and truthful, than the opinion originally expressed, and would have been missed if you hadn’t challenged the original response.
A moderator should never use or report opinions projected onto others, as they are unreliable and unverified. If you’re interested in the participant’s brothers, or kids, opinions, you should be speaking to them directly!
Moderating is very easy to get wrong, through leading question or accidentally making the participant feel stupid or that they’re being judged, and this will compromise the validity of your data. It can also be difficult to balance ethical requirements, such as declaring who is watching the session, and describing the purpose of the study, with getting good data. However, I’ve found the techniques above have helped me be thorough in gathering useful data, while still making the session seem natural. Please feel free to add any other tips about moderating in the comments!
Leave a Reply