1-1 playtesting sessions, where the moderator is sat in the room with an individual player (or small groups playing on one console), are a key part of games user research, and are essential for getting useful usability feedback throughout the development process.<\/p>\n
During my time at PlayStation I\u2019ve been lucky enough to work with some very good moderators, and picked up a lot of insight on the best ways of running these sessions to get useful and actionable feedback on games. This post aims to go beyond the basics, and share some insight into the more advanced techniques that I\u2019ve found have greatly improved how we run these playtest sessions, and research games.<\/p>\n
Always confirm what you\u2019ve seen<\/strong><\/p>\n When watching people play, it\u2019s often tempting just to observe, and assume you completely understand what\u2019s going on. In many cases, the problem seems obvious when a player\u2019s actions imply that there is a usability issue, and you feel confident in reporting the finding.<\/p>\n However, there are times when this can be misleading. Because the moderator doesn\u2019t know what\u2019s going on in the player\u2019s head, they cannot distinguish between issues such as \u201cthe player hasn\u2019t seen the next objective\u201d and \u201cthe player has seen the objective, but is looking for something else first\u201d.<\/p>\n The fixes to both of these issues would be different, so it\u2019s important to ask players what\u2019s going on, to ensure that you are identifying the issue correctly. This also makes it a lot easier for your note-taker, and ensures that they have seen the same issue as you.<\/p>\n The difficulty is asking the players about what they are doing without changing their behaviour. As always, we want to ask non-leading questions, and the easiest way of doing this is to start very high-level and general. For example, ask \u201cWhat is your goal currently\u201d if you\u2019re interesting in objectives, not \u201cWhere do you need to go next?\u201d. The latter implies that they should be going somewhere next, and will change their response and subsequent behaviour.<\/p>\n And as always, don\u2019t interrupt gameplay with these questions; instead wait until the action has died down.<\/p>\n Players have often surprised me with explanations that I couldn\u2019t possibly have anticipated, and so I\u2019d recommend asking players to describe their actions even if they seem obvious. By doing this throughout the session, it also becomes normalised and players won\u2019t feel they are being judged when asked at the critical moments you are really interested in.<\/p>\n Repeat everything. Repeat everything.<\/strong><\/p>\n This is a really simple technique, but has many benefits. When asking questions to players during gameplay or in the final, always repeat back to them the things they have told you. For example, when the player says \u201cI think the controls are bad\u201d, say back to them \u201cSo you didn\u2019t like how the game controlled\u201d? In order to make the conversation more natural, you will probably want to paraphrase what they say, rather than repeat it verbatim.<\/p>\n Repeating the things said by players back to them has many benefits. First of all, it makes it easier for your note-taker, as it ensures that they have heard and understood what is being talked about, and gives them more time to write it down. It also helps check that you\u2019ve understood what they\u2019re describing correctly, and that you are correctly identifying the issues.<\/p>\n Most importantly of all, repeating it back to the player encourages them to give more information. In the example about controls, after I said \u201cSo you didn\u2019t like how the game controlled?\u201d players will often follow up with further information, such as \u201cThat\u2019s right. The jump button felt unresponsive, and it felt like I had to tap multiple times before the game noticed\u201d. You are then clearer about what the issue is, and can follow up with more questions to identify why the issue occurred.<\/p>\n Rather than just agreeing with you when you repeat their opinions back to them, players will often give more information or clarification to qualify their opinion, which gives you more insight into the usability issue, and helps uncover actionable findings.<\/p>\n Imagine telling the development team<\/strong><\/p>\n When speaking to participants about issues that have been observed or that they have raised, I always try to think about how the issue will be reported to the development team. It\u2019s important that usability issues are actionable, and that changes can be made based on your report. How this works in practise is that the moderator shouldn\u2019t stop asking for more detail until they reach something actionable.<\/p>\n For example, if a participant tells you the \u201ccontrols are weird\u201d, this would not be useful feedback. If you reported this to a dev. team, they\u2019d ask more questions such as \u201cin what way are they weird?\u201d and \u201cwhat shall we do?\u201d As a researcher, you need to know the answers to these questions.<\/p>\n So, to avoid embarrassment, the moderator needs to keep probing deeper, with questions such as \u201cWhat were your expectations with the controls?\u201d, \u201chow did using the controls differ to your expectations?\u201d and \u201cwhat did this cause?\u201d When you\u2019ve identified the cause of the issue, and the impact, you can feel a lot more comfortable with reporting this to the team and knowing that it is useful information.<\/p>\n Clarification is especially important for difficulty. As a statement \u201cThis was hard\u201d is not useful feedback. Was it too hard? Was it bad that it was hard? What made it hard? The moderator\u2019s job is to continue probing until they have something that can be used constructively by the development team to increase the quality of the game. Thinking about how you\u2019d describe the issues to the team is a great way of ensuring that you are getting everything you need from playtest sessions.<\/p>\n Ensure players are describing their own experience<\/strong><\/p>\n In playtest sessions, players are often tempted to give feedback based on what they think others will think. I\u2019ve often heard people say \u201cmy brother would love this game\u201d or \u201cthis would be too hard for kids\u201d. This is because the participant doesn\u2019t understand that we\u2019re interested in their own experience and opinion, not in their expertise as a market analyst.<\/p>\n It\u2019s always important for moderators to deflect these kinds of comments, and turn them into something useful. The easiest way of doing this is by putting emphasis on the fact that we are looking for their<\/em> feedback. So when a player says \u201cMy kid would love this game\u201d the moderator should say \u201cand what do you feel about it?\u201d You may find that they don\u2019t actually enjoy it themselves, and they think kids would like it instead because it\u2019s too easy or simple for them personally. This is much more useful, and truthful, than the opinion originally expressed, and would have been missed if you hadn\u2019t challenged the original response.<\/p>\n A moderator should never use or report opinions projected onto others, as they are unreliable and unverified. If you\u2019re interested in the participant\u2019s brothers, or kids, opinions, you should be speaking to them directly!<\/p>\n \u00a0<\/strong><\/p>\n Moderating is very easy to get wrong, through leading question or accidentally making the participant feel stupid or that they\u2019re being judged, and this will compromise the validity of your data. It can also be difficult to balance ethical requirements, such as declaring who is watching the session, and describing the purpose of the study, with getting good data. However, I\u2019ve found the techniques above have helped me be thorough in gathering useful data, while still making the session seem natural. Please feel free to add any other tips about moderating in the comments!<\/p>\n","protected":false},"excerpt":{"rendered":" 1-1 playtesting sessions, where the moderator is sat in the room with an individual player (or small groups playing on one console), are a key part of games user research, and are essential for getting useful usability feedback throughout the development process. During my time at PlayStation I\u2019ve been lucky enough to work with some […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[117],"tags":[62,92],"class_list":["post-1953","post","type-post","status-publish","format-standard","hentry","category-games-user-research","tag-best-practise","tag-user-research","grve-entry-item","grve-blog-item"],"yoast_head":"\n