Being a good user researcher is not just about finding usability or user experience issues with games. Once the issues have been found, another part of the role is ensuring that the development team take the right action to resolve them. To achieve this, it’s important to create a strong trust relationship with the development teams you work with.
As usability researchers, our role is to represent the player as a stakeholder in the design process. In most game development environments there are many stakeholders and influences such as budget, development time, the marketing department and the publisher, all of whom want to give input on the game. In order to cut through the noise, it’s important to ensure that the team understand that your findings are the voice of the player, and not just another source of noise in the design process.
What should researchers do?
One of the most important ways in which the player’s voice can be recognised is to give the teams visibility of the tests. This means inviting developers to view the sessions (and following up to ensure that they actually do so), or streaming the session live to the development team’s office. The book Rocket Surgery Made Easy by Steve Krug gives some good advice on how to set up a viewing session, and encourage people to attend.
If live viewing isn’t possible, saving the videos from the session will help too, but this introduces the strong risk that the development team won’t find time to watch the videos, so the benefits of inviting them to sessions are lost.
There are three main advantages to inviting teams to view the session. The first is that it will help them get a deeper understanding of the issue, and an understanding of the context of the issue, than the researcher can convey when debriefing to the team. It will also help the researchers understanding of the team’s priorities and what parts of the test were particularly relevant for them. This will increase the relevance of the results to the team, and ensure that everyone has a shared understanding of what occurred during the session.
The second advantage of having teams view the session is that it helps build support during the debrief. There is a risk that issues (particularly ‘stupid’ issues) can be flippantly dismissed by development teams, in the belief that the players were wrong, or that the issues has been misrepresented. By having members of the team attend the session, you have created witnesses who can vouch for your findings during the debrief session. This helps ensure the issues are taken seriously, rather than being dismissed as the player’s fault.
Last of all, having the team present during testing also helps with the running of the test from a practical perspective. When weird bugs occur, when the game crashes half way through, or when the players discover the secret debug menu and can’t close it, having a member of the development team available can help resolve this issue, without the risk of losing progress or compromising the test by restarting. As always, keep the developer hidden, but by using instant messenger the moderator can receive ‘divine inspiration’ about fixing the issues.
Ultimately the researcher should become considered part of the team, to ensure that they are aware of current decisions and can advocate for the player’s experience. Inviting teams to view sessions can be the first step to achieving this, and building a successful working relationship.
What shouldn’t researchers do?
The top mistake that researchers can make, that will instantly destroy credibility, is misrepresenting the game, or what happened in the build tested. This can occur due to a lack of code screening (e.g. you don’t know the game well enough, or a last minute build replaced the one screened previously), or relying on assumptions about why issues occurred, and these can slip into a report as the potential cause for an issue. For example a poor researcher could observe the issue “Players failed to reload”, and assume it was because the game didn’t indicate when reloading was required. However if in the game does (such as through audio cues, or indications on the HUD), then writing “the game doesn’t indicate when reloading was required” as a cause will damage their credibility with the team. Players may not have noticed the cues for reloading, but that’s a different cause to the issue, and should be reported as such.
It’s therefore very important to make sure that any assertions you make about the game in a debrief are true. When an issue happens, and you are analysing the causes, make sure that you have a copy of the game available to check that the causes attributed to the game are true. When working in pairs, there is a temptation for the lead to delegate this task, or rely on un-verified observations provided by the assistant. This can be fine, but remember that it’s the person who is leading the test who will look stupid if it all goes wrong!
It’s important to note that sometimes development teams can challenge assertions made about the game, and it will be the developer who is wrong. Often changes are made between builds, or the build tested can be a few weeks old, and not everyone in the debrief will have a strong knowledge of the state of the game during the test. This can be mitigated by having members of the development team attend the test sessions, who can then verify what was in the build during the debrief, but these discussions go a lot easier if the researcher is confident they know what happened in the build tested.
Another thing to watch out for that can hurt your credibility with design teams is designing by stealth. This is a breach of trust with the team, and can be overstepping the role of a user researcher, and will damage the relationship with the team after.
Designing by stealth is when the researcher hides design suggestions within the cause of the issue. In the example about reloading, a mild case of designing by stealth would be “the issue was caused by the lack of a noise indicating when reloading was required”. A more extreme example would be “the issue was caused by the lack of a tutorial for reloading”. Each of these hide the researcher’s opinion about a potential fix in them, as indicated by the use of the phrase “the lack of”.
The risks with stealth designing are that it confuses the researcher’s role – they no longer become the objective presentation of the player’s experience, and start to become another person voicing design suggestions – one among many.
As researchers, it is true that you may have particular insight on potential fixes to the issues, based on your usability expertise, experience testing other games, or from what was observed during the user test. However it is really important to keep this separate from describing the issues that occurred – in a separate recommendations section, or in discussions after reporting, so that the issue remains a true representation of what occurred during the session, and doesn’t get disregarded or deprioritised.
Building a successful working relationship with your development teams is an ongoing process, and can take a long time. In the early stages of a relationship, keeping a strong divide between the player’s insight and your expert opinion can help ensure that the findings from user research sessions are given the priority they deserve.
Leave a Reply