Archive for the ‘UX Guides’ Category
The thirty minute facebook game usability test
Bit of a cheat this week, since this is an adoption of a recent email conversation I had, but I think it may be of interest to a wider audience. The idea is a proposed ‘simple’ study, suitable for a facebook or flash game, that will give an insight into major usability issues with a game. The focus is on getting the key insights quickly and cheaply, which will allow you to improve your game straight away with actionable results.
I’d be interested in hearing your thoughts on this – please use the comments to add critiques or alternative viewpoints, as I’m sure there will be many! Anyhow, onwards with the 30 minute facebook game usability test.
I’ve been thinking about the social games found on facebook, and I think the most important part is the first 15 minutes of a new player’s experience, e.g. what happens in those fifteen minutes, do they understand the game, and do they want to come back and play again.
This means the most important aspect of user testing is understanding and improving the ‘new user’ experience. For this you’d need some people who’ve never played the game before (and who are close to the target audience of players you want to attract), and simply get them to play the game from the start (without special instructions, just how they would if it was on facebook). Maybe a small incentive (like pizza!) would be enough to recruit people for these short sessions.
a small incentive
Explain to the player that you want them to just play the game as they would at home, and that you are testing the game – not them. Pre planning some notes on this introduction speech will make it easier. It’s probably a legal requirement to let them know if the session is being recorded.
You should have interested developers watch them play through a remote desktop tool (but they shouldn’t interrupt the player!), or record the session to review later. Free tools, such as team viewer, are available to do this. This will be invaluable for getting their buy-in for further user testing, and will prove the benefit of user testing to an often dubious audience.
It’s common to use a ‘think aloud’ methodology, where you ask the player to speak their thoughts aloud as they play. It’s not perfect, but it’ll give some insight into what they’re thinking. You could prompt them by asking non-leading questions such as “what are you doing now” or “what are you thinking” if they’re silent for too long!
what are you thinking?
Then after the fifteen minutes (or completing the tutorial), you can ask them questions to gauge how they understood the game – did they know what they were doing, were they confused by the game, did they know what to do next, would they like to continue playing, etc. Maybe you saw some interesting behaviours while they were playing that you’d like to ask about. Often people don’t remember what they did, and so you’d have to prompt them, or maybe the developers watching have some questions to ask.
Hopefully this quick methodology will show where the user’s are getting confused, or not understanding what to do next, or getting bored, or any other factors that turn player’s away from the game in that crucial first play.
Jakob Nielsen said that most major usability problems can be found by 3-5 users, so you wouldn’t need to run this test many times per iteration of the game. As to when this should be done, its best to get started as soon as there is something playable, as changes will be easier to make the earlier they are found, and then a similar test can be repeated with the next version of the game. Many social games go through an agile design process, with many iterations, and so this process will fit in well.
What do you think? Does this sound helpful? Or do you think that the ‘new user’ experience is not the most important part of a facebook game – maybe converting players to paying players is instead… let me know your thoughts in the comments here, and we can improve this 30 minute facebook game test.
The right environment for user testing
Most games testing, like the user experience testing done at Vertical Slice, is done in a quiet room with a comfy sofa and with access to free food and drink. This isn’t just a needless luxury; instead it’s an integral point of the user testing process, and gives superior results to those found in a typical lab environment, with rows of workstations and one way glass. Why is this, and what factors are at play here?
What is the environment?
With user testing, the environment comes down to the context in which the software or product is used. This includes where the user is, when they are using something, and who they are using it with. It’s important to not forget the factor that the environment plays when testing a product, otherwise your final results, conclusions and design decisions may end up way off the mark.
Considering the environment for testing is important for both usability and user experience. For usability testing, a realistic setting is needed to understand how a system will perform in that environment when out in the field. User experience focuses on how a system makes the user feel, and to truly understand this, you need to test users in an equivalent setting. This is why you’ll find comfy sofas wherever games are tested, since games are typically played at home in front of the TV.
pizza boxes are optional
Getting it wrong
Common mistakes with user testing involve inaccurately representing the environment in which a product is to be used (poor “ecological validity”).
This could include an unfair representation of the conditions in which the product would be used – for example testing a car radio without having the user drive would not give a true understanding of how the radio would function when the user’s attention is diverted.
Another common mistake is failing to test things in a time sensitive manner. For example, ticket machines are often used in high stress situations, minutes before the train is going to leave. Giving the user ample time in a comfortable lab setting does not recreate a typical interaction with the ticket machine, and would give an unrealistic impression of its usability.
User tests should also be separated, though concurrent, to the design process, and the development team should not be allowed to run the user sessions (though it’s a good idea to have them spectate from afar). Having the team in the same room as the player can intimidate them into praising the game or product, and reduce the validity of the results.
you liked it... got that?
Getting it right
So how can you be sure to get the environment right for testing? The following things must be kept in mind:
- The right place: The ideal place would be the actual environment in which the game or product would be used. However this isn’t always possible, and so it’s important to recreate the environment as closely as possible. It’s also a good tip with games to test somewhere away from where the games are made – a typical player would be excited to be at a game studio, and their opinions would be nicer, and more eager than a typical home experience.
- The right mood: The sofa, and snacks mentioned in the introduction are all part of making the environment comfortable for the user, so that they feel at ease when testing the product or game. Another advantage to the previously mentioned idea of running these tests off-site, away from the people who made the product, is that the user won’t feel intimidated into praising the product, and they’ll be free to give a true opinion.
- The right people: It’s important to recruit the realistically, and test with real users. I’ve written before about how you should test with real users, not your team, and it is one small step which can increase the the validity of your results massively.
- The right friends: Remember when testing multiplayer games that a new environment will initially make the users uncomfortable, and this is magnified if they are playing with strangers. If you want to see how your game is received when played among friends, or with families, there is no shortcut – you have to test with groups of friends to get a true representation.
Taking care when selecting the right environment is important whether you are making a game, a ticket machine, or a washing machine for your cat. A suitable environment for user tests will increase the validity of your results, and ultimately help make informed design decisions.
Test with real users – not your team
‘Free pizza and coke! Just play our game for an hour’. Sounds like a good deal right? And pretty easy to organise, just pulling kids of the street. It can even be done in the pub, for mobile devices. Even this ‘free pizza’ recruitment is better than testing your game (or website, or application) with people from within your office. But why?
Game development teams need a constant supply of fresh users to test the ‘new user’ experience with. I’ve seen teams keep their project secret from their colleagues, not for official reasons, but so their colleagues can be tested as ‘new users’. Other teams test their games with their HR and secretarial staff, since they are unlikely to have had much exposure to the game.
However it’s a good guideline to never test with your team (unless of course you are building something for them). It’s understandable why this situation arises – often budgets are too tight for intensive user testing, forcing teams to perform ad-hoc tests with their colleagues; however this often causes problems further down the line:
such as ...accidents
Dont test with your team
- Your team are not your users – Unless you are in a very specialist field, or are developing an internal project like an intranet, it’s unlikely that your team are the same people as your users. And they are very unlikely to act in the same way a typical user would.
- Your team know things users wouldn’t – It’s likely your team will have had prior exposure to your game or application that a new user wouldn’t, and will be bringing prior knowledge to the testing session. This also applies to people who do not work directly on your team. To get a true outside perspective, you need to seek outside users.
- Your team know you – Unlike a stranger, your team are already know you, and (hopefully) like you. Their answers, and interactions will be biased to please you, and tell you what you want to hear based on what they know about your job, the project your working on, or your beliefs (for example, attempting to validate your design choices).
Advantages of testing with real users.
How people act can often be surprising. If this wasn’t the case, there would be little point in user testing. That’s why it’s extremely important to gather real data, from the people who will actually be using your product. Only real users will approach your product from an authentic ‘new user’ angle, and give an insight into how your product will be perceived and used.
Getting real users involved with product development will get them engaged with the product. Asking their opinions, and being interested in their experience will make the user feel positive about you, and your product, and will mean they will be more likely to purchase it when it’s ready. In newsrooms, this has been widely known for years – hence the proliferation of lists of names in local papers.
Most importantly, involving users will get them talking about your product, generating true grass roots ‘word of mouth’ promotion (hopefully without breaking any NDA’s!). Giving customers an early exposure to your product can build excitement, and market your product for free!
saving millions on megaphones
Conclusion
Finding real users can be cheaper than you think. Not only is it possible to pull people off the street, using the methods suggested above, but new usability testing methods such as remote user testing allow you to find and test real users from the comfort of your office, for very low cost. These days there’s almost no excuse not to test with real users, and it can be just as easy as testing with your team, with much more rewarding returns.
The Problems with Surveys for User Experience Tests
In the run up to Margaret Thatchers’ election victory in 1979, a poll was taken to estimate who would vote for her. Only 1 in 100 said yes. However, as revealed by the final results, 1 in 3 actually voted for her. The poll was inaccurate, and inappropriate for the task.
Surveys are a common tool used to evaluate a participant’s opinions of the user experience, and usability of a system. I’ve written about how to make good questionnaires before, and have often seen them used as a tool when analysing a large group of participants. However, as a method of understanding users, they are imperfect, and not just because they are poorly designed – instead it’s a fundamental problem with surveys. Let’s look at why this is the case, and why people are tempted to use surveys despite this.
Where are surveys used?
When I’ve been involved with user tests for games, I’ve often seen surveys used as a way of recording the player’s experience. For example, after completing a level, or game mode, they would be asked to rate their experience on a Likert scale (1-10), on categories such as how difficult they found the level, how fun it was, how it compared to other levels. This is often complemented by text notes, where the participant can write in things they particularly liked or disliked.
Outside of gaming, surveys can often be found on the internet – such as website’s satisfaction surveys, or on professional survey sites, like Survey Monkey.
Survey Monkey in action
Why are surveys used?
It’s easy to understand why surveys are often used when testing user experience. Most obvious is that they are easy to quantify, since the scores are given as a numeric value, which can then be averaged, and given an overall ‘score’. This can then be stuck on a graph, to impress people too busy and important to be involved with the testing itself. Compared to moderated testing, simple analysis is easy, and ‘results’ can be gained with little effort – particularly if an online survey tool is used.
Similarly, with surveys it’s easy to get a large number of opinions quickly, and in a largely un-moderated setting. Hence, 10 (or 10,000) people can test a game at the same time, with only light moderation, and fill out a survey after to record their views. Surveys also don’t require a large degree of specialist equipment – just a printer, and a pen (or they can be done online). This makes them cheaper than many moderated settings, which require a lab decked out with recording equipment.
Problem with surveys
Surveys sound great, don’t they. Cheap, Easy, and give some hard numbers. However, there are a number of problems with surveys, and one key issue that prevent them being suitable for user experience analysis.
First of all, it’s easy for the data from surveys to be misrepresented (either unintentionally or to further a top secret agenda!). Without hard evidence, such as watching (and recording) an individual player of the game, the analysis becomes reduced to which level ‘scores better’, regardless of the intricacies of the play test. Minor issues become lost within the overarching ‘score’.
Much more importantly, the fundamental problem with attempting to understand user experience with a survey is that they log opinions, and not behaviour. People are (sometimes?) stupid, and don’t know what they think. So a player who has had a positive experience throughout a level, and got stuck near the end, will often be left thinking poorly of the entire level. And without an independent observer to monitor, their in-game opinions are lost, or forgotten. Just like I cannot tell how bad my singing is, a player is too close to the subject matter to gain a full understanding of it.
Its pretty bad...
Essentially, surveys introduce a layer of abstraction from the game that is difficult for a player to follow. It is difficult for them to recognise what parts of a game made it fun, and which parts frustrated them, and it often takes someone else to spot these patterns.
Pride, and psychology can also be a contributing factor – players who have needed 10 attempts to complete a section will still say it was “easy” after finally completing it – psychologically they will often believe it as well, since they have felt the satisfaction of completing the task. Other times they will be too proud to say the section was too difficult, and lie. Again, this rich data is lost through a survey.
What should be used instead?
To gain a truer understanding of the user experience (or player experience) of participants when testing a system, or a game, surveys are therefore inadequate. Instead, a moderated task based analysis session, which is recorded for later analysis, will give a truer understanding of how the participant found the system, and their true experience, unaltered by their own perceptions. I’ve written about recording these sessions before, and will discussed them further in the future.
As we have seen, surveys are cheap and easy, and hence should not be disregarded entirely. However they should not be used exclusively, as they can miss key user experience findings, and require users to know themselves, and their feelings, extensively.
Telling Tales – Stories for promoting user experience.
Stories have long been an important way of recording and imparting information, as evident through the survival of folklore, myths and songs from our past. As a tool for communicating, and retaining information, they are highly valuable, and the principles of this can be applied to the promotion of User Experience.
Why do teams need to understand UX?
When working within a team, whether it’s a small web development company, or a large multi-national company, it’s impossible for one person to oversee every aspect of development. That is why it’s important for everyone to understand the principles of user experience, due to the large variety, and span of tasks that it encompasses. A good user experience designer is not the one who works longer hours, or has more creative control over the final product, instead they are the one who makes sure everyone understands UX goals, and works towards them.
It is this reason that books such as Krug’s Rocket Surgery Made Easy heavily promote the idea of getting everyone, from the CEO to a junior developer, involved when testing with real user’s, as it is these people who will be able to implement user experience ‘fixes’. The solution Krug suggests involves inviting these people to view the user testing sessions, and bribing them with snacks.
Because encouraging successful user experiences requires everyone’s involvement, it’s important that everyone understands the goals of user experience, and can incorporate them into their work. The problem is making them understand what those goals are.
Pictured: not a goal
What are stories good for?
Homer’s Odyssey was composed in the 8th century BC, and yet a canonical written version wasn’t produced for another 300 years. Before this, the tale was remembered, and shared orally, through song. Considering that the Odyssey contains over 12,000 lines, this is no small feat. That’d be like remembering, and reciting on demand, every blog post I’ve done so far… So how was it possible?
The answer lies in the power of stories. Stories, like Homer’s Odyssey, are memorable, due to the plot and the events contained within, and hence easily form visual images within the mind, and aid memory. Plots make stories interesting, and hence easier to remember than non-fictional texts (such as remembering all of my blog posts).
Stories can also be used to impart useful information. These be obvious life lessons (The Three Little Pigs teaching the benefit of putting adequate effort into your tasks), or life saving, such as Ring a Ring o’ Roses imparting symptoms of the Black Death (although this may not be true)
or Cinderella teaching us the dangers of stepmothers
Using stories for UX
So, stories are memorable, can be used to impart useful information, and are easily transferred, as made evident by the 300 years that Homer’s Odyssey lasted without being written down. Hence they seem ideal for the promotion of UX within teams, as they can be used to quickly impart the benefits of UX, and what each individual needs to do to aid a user centred design process, in a memorable and transferable fashion. By bringing the whole team onboard with the principles and goals of UX, a greater degree of coherence can be achieved across the team. Therefore to get better results without manually verifying everything the team does, a User Experience Designer needs to champion user experience principles with all members of the team, and stories are an easy way to do this.
Some examples of UX stories
Many authors have understood how stories help to impart information in a memorable fashion.
- In The Design of Every Day Things, Don Norman uses stories of real world examples to emphasise the dangers of ignoring user centred design, with examples including nuclear accidents and plane crashes.
- Stories based on metaphors are also used by Alan Cooper in The Inmates Are Running the Asylum to emphasise the importance of designing for the end user, by comparing the software design process to film making.
Also of interest, Sam Nixon shared a short story by David Travis called “The Fable of User Centered Design”, which aims to bring clients and team members on board with the benefits of a user centred design approach. Definitely worth the read, and should be shared. The book is available to download from his website.
If anyone has other examples of stories that help define the importance of user experience, remember to leave a comment below!
