This post is about why you shouldn\u2019t playtest your game. But it\u2019s not about why your game shouldn\u2019t be tested. It just shouldn\u2019t be you, the developer, doing it.<\/p>\n
Playtesting during development is invaluable, and it\u2019s great to see how enthusiastic many teams are about running their own tests. However there are a number of biases and pitfalls that we often see in tests run by the developers for their own game, which alter the feedback received and greatly reduce its value.<\/p>\n
In this post I cover some of these biases and look at ways in which they can be minimised, to make sure that the playtests are useful, and produce actionable results.<\/p>\n
<\/p>\n
Don\u2019t let players know you\u2019ve made it<\/strong><\/p>\n One of the first things we say to players when they come in to see a game during development is that we don\u2019t make the game. We tell users that they can be as rude as they like about the game and they won\u2019t offend us, and that we don\u2019t get rewarded if they love the game. We do this to distance ourselves from the game, and let players know this is a safe environment for giving criticism.<\/p>\n People naturally want to please others, and will tell you what you want to hear. This can often be unconcious, and players may not even recognise they\u2019re doing it. However their behaviour and feedback will be different if they think you are invested in the game, and you\u2019ll lose valuable data.<\/p>\n It\u2019s understandable to be proud of your game, but developers have to keep their relationship to the game secret otherwise the results can be biased. It\u2019s unethical to lie about your identity, so teams should wherever possible pick people to run the tests who are not heavily invested on its outcome. If this is not possible, you can just keep quiet about your role unless directly asked, and describe your role as a tester, rather than a developer, of the game.<\/p>\n Be careful with anticipating problems<\/strong><\/p>\n Another issue we often see with tests led by developers is that the team have an agenda they want to push. Research can be a great tool for answering questions the team may have, however particular care has to be taken during the test and during analysis, to ensure that results are represented fairly.<\/p>\n For example, if a team member hates the design of the menus, they could make the results express this by setting unrealistic menu tasks, asking leading questions, or over-emphasising the feedback from players who didn\u2019t like the menus.<\/p>\n The real danger from anticipating issues is that it can cause unanticipated issues to be missed, as observers will only be looking out for specific things. It introduces lots of biases into how the sessions are remembered, as only the behaviour and feedback from users who behaved as anticipated, or the most entertaining users, will be remembered.<\/p>\n It\u2019s important when running these sessions to act as an impartial observer, and to record everything that happens. Conclusions during the sessions are at a huge risk of bias, and it\u2019s only after all the results have been collated and analysed that reliable results can be found.<\/p>\n Recruit the right people<\/strong><\/p>\n Another challenge with internal playtests is finding the right people to play the game. Gamers range from kids playing on smartphones, through hardcore console gamers, to OAPs playing Brain Training on their DS. However your game will only target a subset of these players, and it\u2019s important that you find suitable players to playtest your game. This will ensure that the results you find are indicitve of the issues that your real players would encounter after the game comes out.<\/p>\n When teams run their own playtests, it\u2019s often easy to misrecruit. A common mistake is finding people from within the office \u2013 not only might they not be the right sort of players, but they will have background exposure to the game, even if they do not work directly on it. This will give them knowledge that real players would not have.<\/p>\n If a professional recruitment agency is too expensive, recruitment through facebook, twitter and word of mouth is possible. However I would definitely recommend setting up a questionnaire about their playing habits, and asking people to fill it out before they get booked in. This will allow you to screen participants and ensure that they are similar to the audience that will buy your game, and that the results will be valid. The extra time it takes to make and analyse the questionnaire will be nothing compared to the time wasted by running sessions for users who are unrepresentative of your audience.<\/p>\n Run tests in a realistic setting<\/strong><\/p>\n Trade shows are a great opportunity to market your game to consumers. However, they are not a great environment for getting realistic feedback from players.<\/p>\n Trade shows are nothing like how people play games at home. There is a high level of background noise in trade shows, which can obscure audio from the game. Players will also feel under time pressure, either from the queue behind them, or from their desire to see the next game. At trade shows players often have an audience watching them, which will alter player\u2019s behaviour. Also the players may be entirely the wrong sort of people \u2013 you haven\u2019t found out anything about them (other than they like trade shows), and so they may not be the sort of player your game should be made for.<\/p>\n To get realistic feedback, always ensure that the location matches a realistic play setting as close as possible \u2013 often this means a living room environment. As discussed earlier, you also want to distance players from development, so running these tests off-site can help with that. Hotel rooms are often a useful way of achieving this, as they can be easily rented, often have TVs already, and feel like a living room.<\/p>\n Do playtest your game<\/strong><\/p>\n Playtesting when making your game is fantastic. It gets invaluable feedback while there is still time to fix issues, and helps prioritise development. Some of the most common biases when teams are running their own playtests are easy to overcome \u2013 and an awareness of them will help produce useful and relevent results. Combining well run internal playtests with playtests run by external professionals will improve the quality of your game, while still keeping costs down.<\/p>\n <\/p>\n <\/p>\n","protected":false},"excerpt":{"rendered":" This post is about why you shouldn\u2019t playtest your game. But it\u2019s not about why your game shouldn\u2019t be tested. It just shouldn\u2019t be you, the developer, doing it. Playtesting during development is invaluable, and it\u2019s great to see how enthusiastic many teams are about running their own tests. However there are a number of […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[117],"tags":[89],"class_list":["post-1934","post","type-post","status-publish","format-standard","hentry","category-games-user-research","tag-bias","grve-entry-item","grve-blog-item"],"yoast_head":"\n