Posts Tagged ‘user experience’
Test with real users – not your team
‘Free pizza and coke! Just play our game for an hour’. Sounds like a good deal right? And pretty easy to organise, just pulling kids of the street. It can even be done in the pub, for mobile devices. Even this ‘free pizza’ recruitment is better than testing your game (or website, or application) with people from within your office. But why?
Game development teams need a constant supply of fresh users to test the ‘new user’ experience with. I’ve seen teams keep their project secret from their colleagues, not for official reasons, but so their colleagues can be tested as ‘new users’. Other teams test their games with their HR and secretarial staff, since they are unlikely to have had much exposure to the game.
However it’s a good guideline to never test with your team (unless of course you are building something for them). It’s understandable why this situation arises – often budgets are too tight for intensive user testing, forcing teams to perform ad-hoc tests with their colleagues; however this often causes problems further down the line:
such as ...accidents
Dont test with your team
- Your team are not your users – Unless you are in a very specialist field, or are developing an internal project like an intranet, it’s unlikely that your team are the same people as your users. And they are very unlikely to act in the same way a typical user would.
- Your team know things users wouldn’t – It’s likely your team will have had prior exposure to your game or application that a new user wouldn’t, and will be bringing prior knowledge to the testing session. This also applies to people who do not work directly on your team. To get a true outside perspective, you need to seek outside users.
- Your team know you – Unlike a stranger, your team are already know you, and (hopefully) like you. Their answers, and interactions will be biased to please you, and tell you what you want to hear based on what they know about your job, the project your working on, or your beliefs (for example, attempting to validate your design choices).
Advantages of testing with real users.
How people act can often be surprising. If this wasn’t the case, there would be little point in user testing. That’s why it’s extremely important to gather real data, from the people who will actually be using your product. Only real users will approach your product from an authentic ‘new user’ angle, and give an insight into how your product will be perceived and used.
Getting real users involved with product development will get them engaged with the product. Asking their opinions, and being interested in their experience will make the user feel positive about you, and your product, and will mean they will be more likely to purchase it when it’s ready. In newsrooms, this has been widely known for years – hence the proliferation of lists of names in local papers.
Most importantly, involving users will get them talking about your product, generating true grass roots ‘word of mouth’ promotion (hopefully without breaking any NDA’s!). Giving customers an early exposure to your product can build excitement, and market your product for free!
saving millions on megaphones
Conclusion
Finding real users can be cheaper than you think. Not only is it possible to pull people off the street, using the methods suggested above, but new usability testing methods such as remote user testing allow you to find and test real users from the comfort of your office, for very low cost. These days there’s almost no excuse not to test with real users, and it can be just as easy as testing with your team, with much more rewarding returns.
Evaluating User Experience in Games – Book Review
Regina Bernhaupt presents an academic guide to the application of user experience principles to games, as part of a series by Springer Books on HCI, and claims to represent the ‘coming of age’ of video games as a medium. The book is essentially a collection of academic papers, largely from recent conferences, and draws upon the talents of a wide range of authors, including Brighton based Graham McAllister and Gareth White, Katherine Isbister (Editor of Game Usability) and Emily Brown of SCEE. Although largely academic, the book does provide an insight into the cutting edge of this exciting field.
Early chapters in the book try to define what the field of user experience is, and what it means in relation to games. There is a degree of confliction here, with each chapter giving a slightly different interpretation, but they often settle on themes such as immersion, fun, presence, involvement, engagement, flow and playability.
Also wind, fire and heart!
The book gets more exciting when it presents a range of methods for evaluating user experience in games, with a variety of models appropriate for various stages of game development, from prototyping to post-production. This encompasses many custom models for different situations, such as a model for inexperienced gamers, or one for fitness games. The book also presents studies of the usability of game controllers, and the development of heuristics, which is particularly interesting in the last chapter which aims to collate and amalgamate previously created gaming heuristic.
All this content is interesting; however, a liberal spreading of maths means it often comes across as extremely academic (particularly in comparison to Isbister’s book). This can largely be accounted to the background of the various authors, largely coming from academic institutions, compared to the real world perspective of Game Usability.
Where the book excels is the divergence from this academic interpretation, notably in the papers by Vertical Slice and Emily Brown. Vertical Slice cover the current state of user experience evaluation across three Brighton game companies, and give an insight into the methods used through case studies, from the expert evaluation found in the earliest stages of production, through to the user testing close to the end of a project.
Brown gives a comprehensive overview of the range of tools currently in use, and shows optimism for the future, since she recognises only a “lack of knowledge” as a hindrance to the extension of user testing into gaming, rather than opposition to the methods. This conclusion is reinforced by the case studies by McCallister and White, who show game developers are looking to extend their application of user experience testing in the future.
Which will be the same as today, but with more robots
Unlike Remote Research or Game Usability, this book is not a practical how-to guide. Instead it presents the state of user experience in games, and where the cutting edge of research is. Personally I have found it very useful for developing my own research. However the book would be unlikely to be a ‘one stop shop’ for a developer looking to start user experience research at their company.
It will be useful to see how the wide range of interesting ideas found within this book can be integrated into practical solutions for companies to use when producing games. This move from the theoretical to the practical will greatly assist the field of user experience in games, and will truly see the ‘coming of age’ of video game usability.
Make work fun – examples of applying the UX of games
I write this blog out of personal interest. It never really seems like work, and so I’d be unlikely to blog more often if I received points for doing so (although I do like free things!). However this isn’t always the case – sometimes there are jobs you just have to do, regardless of how you feel about them..
Since my recent post on using mechanics from games to make dull tasks fun, a couple of upcoming applications have taken the step of applying some of the lessons from gaming, to make work fun. Epic Win is a ‘to-do’ list disguised as an RPG, and Dev Hub is a website creation tool with points. I’ll look at what these applications are, how they apply game mechanics, and how successful we can expect them to be at making dull tasks fun.
Epic Win
The new iPhone ‘game’ Epic Win is a productivity application (essentially a to-do list?) mixed with an RPG. After inputting your list of tasks (such as wash the dishes, or write a blog post), the game will reward you with XP for completing each task, allowing you to level up your character, as well as granting special items and other rewards.
Achievement Unlocked: Made Dog Useful
There is a trailer for Epic Win here, which may explain the theory better.
Essentially, through giving you points for achieving tasks in the real world, the game aims to incentivise the player to perform the real world tasks, by applying a common game mechanic.
DevHub
DevHub is a website creation tool, focusing on automating the process for creating blogs for niche topics (like this one?) and allowing authors to monetize their site (maybe I should be interested…)
Me, writing this blog
In the initial implementation, they found they had a problem. People were making only the simplest sites, using a small range of DevHub’s features. So to incentivise people to use a fuller range of features, they added game mechanics.
Now tasks like blogging, or linking your site to your facebook profile accrue points, which can then be spent on improvements to the site, such as new templates or widgets. This gives a gradual reveal of the site’s features, and gives the owner (player?) a sense of progression.
Thus game mechanics help DevHub’s users discover, and utilize a wider range of features than before.
Are they fun?
The game mechanics in both of these new applications seem to be simpler than those outlined as successful in the ‘Just Add Points’ presentation I covered recently. Both use points as a mechanic to incentivize players to repeat actions, or go further than they normally would. Points allow the clear construction of goals, and for progress to be measured. I imagine competition will become a key part of these two applications, as social media will allow players to compare and compete on points.
However many of the problems outlined in Sebastian Deterding’s presentation still apply.
- Epic Win doesn’t change the player’s goal (wash the dishes), it just monetises it, meaning the ‘fun’ derives from gaining points
- I cannot see how the validity of the points value can be enforced. Since the goals are self-assigned, and self-reviewed, players who want points will just set tasks such as ‘sit down’, and reward themselves (or just lie altogether).
- This makes points valueless, and hence points don’t help the game add ‘fun’ to achieving tasks.
- Also, this makes social comparison, a key factor in the success of points, flawed or impossible (so no high score tables)
DevHub may also run into trouble, since withholding features that can be found on other competing sites for no effort will only work if your site has a strong unique selling point. We’ll have to see what other monetizing blog hosts do.
It’ll be interesting to see how these applications do over the next few months. Deterding’s presentation implies that neither have a successful model for applying fun to dull tasks. I’m looking forward to seeing what the players think.
The Problems with Surveys for User Experience Tests
In the run up to Margaret Thatchers’ election victory in 1979, a poll was taken to estimate who would vote for her. Only 1 in 100 said yes. However, as revealed by the final results, 1 in 3 actually voted for her. The poll was inaccurate, and inappropriate for the task.
Surveys are a common tool used to evaluate a participant’s opinions of the user experience, and usability of a system. I’ve written about how to make good questionnaires before, and have often seen them used as a tool when analysing a large group of participants. However, as a method of understanding users, they are imperfect, and not just because they are poorly designed – instead it’s a fundamental problem with surveys. Let’s look at why this is the case, and why people are tempted to use surveys despite this.
Where are surveys used?
When I’ve been involved with user tests for games, I’ve often seen surveys used as a way of recording the player’s experience. For example, after completing a level, or game mode, they would be asked to rate their experience on a Likert scale (1-10), on categories such as how difficult they found the level, how fun it was, how it compared to other levels. This is often complemented by text notes, where the participant can write in things they particularly liked or disliked.
Outside of gaming, surveys can often be found on the internet – such as website’s satisfaction surveys, or on professional survey sites, like Survey Monkey.
Survey Monkey in action
Why are surveys used?
It’s easy to understand why surveys are often used when testing user experience. Most obvious is that they are easy to quantify, since the scores are given as a numeric value, which can then be averaged, and given an overall ‘score’. This can then be stuck on a graph, to impress people too busy and important to be involved with the testing itself. Compared to moderated testing, simple analysis is easy, and ‘results’ can be gained with little effort – particularly if an online survey tool is used.
Similarly, with surveys it’s easy to get a large number of opinions quickly, and in a largely un-moderated setting. Hence, 10 (or 10,000) people can test a game at the same time, with only light moderation, and fill out a survey after to record their views. Surveys also don’t require a large degree of specialist equipment – just a printer, and a pen (or they can be done online). This makes them cheaper than many moderated settings, which require a lab decked out with recording equipment.
Problem with surveys
Surveys sound great, don’t they. Cheap, Easy, and give some hard numbers. However, there are a number of problems with surveys, and one key issue that prevent them being suitable for user experience analysis.
First of all, it’s easy for the data from surveys to be misrepresented (either unintentionally or to further a top secret agenda!). Without hard evidence, such as watching (and recording) an individual player of the game, the analysis becomes reduced to which level ‘scores better’, regardless of the intricacies of the play test. Minor issues become lost within the overarching ‘score’.
Much more importantly, the fundamental problem with attempting to understand user experience with a survey is that they log opinions, and not behaviour. People are (sometimes?) stupid, and don’t know what they think. So a player who has had a positive experience throughout a level, and got stuck near the end, will often be left thinking poorly of the entire level. And without an independent observer to monitor, their in-game opinions are lost, or forgotten. Just like I cannot tell how bad my singing is, a player is too close to the subject matter to gain a full understanding of it.
Its pretty bad...
Essentially, surveys introduce a layer of abstraction from the game that is difficult for a player to follow. It is difficult for them to recognise what parts of a game made it fun, and which parts frustrated them, and it often takes someone else to spot these patterns.
Pride, and psychology can also be a contributing factor – players who have needed 10 attempts to complete a section will still say it was “easy” after finally completing it – psychologically they will often believe it as well, since they have felt the satisfaction of completing the task. Other times they will be too proud to say the section was too difficult, and lie. Again, this rich data is lost through a survey.
What should be used instead?
To gain a truer understanding of the user experience (or player experience) of participants when testing a system, or a game, surveys are therefore inadequate. Instead, a moderated task based analysis session, which is recorded for later analysis, will give a truer understanding of how the participant found the system, and their true experience, unaltered by their own perceptions. I’ve written about recording these sessions before, and will discussed them further in the future.
As we have seen, surveys are cheap and easy, and hence should not be disregarded entirely. However they should not be used exclusively, as they can miss key user experience findings, and require users to know themselves, and their feelings, extensively.
Applying Games UX lessons makes dull tasks fun!
I recently watched Sebastian Deterding’s presentation ‘Just Add Points?’. It covers applying lessons learnt from games to software, to make software more enjoyable to use. The talk then goes on to cover where this model traditionally falls down, before rebuilding a model with new rules. The presentation was engaging, very well designed and a good extension of the principles within Ralph Koster’s book, applying its lessons to the real world, and therefore well worth a look.
The presentation first covers ways in which the UX lessons learnt from games have, or can, be applied to dull tasks to incentivise people to do them. Some examples of this can be found on Volkswagen’s thefuntheory.com website, such as turning a staircase into a piano to encourage people to take the stairs, or turning a bottle bank into a game to encourage people to recycle.
all the encouragement I need...
Deterding does provide some critical analysis of this model – what happens on day 2, for example? Is it still fun to recycle? I also question the justification provided by Volkswagen that the bottle bank performed better than a standard one. Although it did in the example, when the user was provided with a choice between two geographically-close bottle banks, this fails to be a conclusive proof of the fun bottle bank being more effective at encouraging recycling. (would the ‘dull’ one receive an equal amount of recycling to the fun one if there was no alternative – what about over a number of months?)
The typical theory of fun is that ‘adding points’ will magically make dull activities fun, because of It adds competition, re-playability, and a new ‘meta-game’ to the activity taking place. However, Deterding’s presentation challenges this, and says that ‘just adding points’ is a too simplistic understanding of the application of fun to menial tasks. Instead, games present an optimized version of many positive psychological features of real life, and through the recognition of this, real life can be optimised.
As I discussed in my review of Ralph Koster’s book, ‘fun’ is the act of learning and successfully applying, and adapting the knowledge learnt, and typically games present an adaption of this. Games optimise ‘fun’ because:
- They allow the construction of clear, realistic goals, with measurable progress
- The goals are presented in a manageable manner, with a clear ‘call to action’, indicating what is to be done, and when it has been achieved
- The player’s current status is clear, and their progress towards the end goal is indicated
- New tasks are built upon knowledge already gained
- Social comparison can be made with your friends to compare progress
So the obvious way of making dull tasks fun would presumably be to integrate these principles from games? However this conflicts with software, and menial tasks, core goals of efficiency. As I’ve noted before, ‘press a button to win’ is effective, but not fun.
Unlike games, software (and menial tasks) doesn’t give designers full control over the environment – instead the user defines the goal (such as ‘write a letter to the TV Licensing people’). This makes direct application of the features from games difficult.
We have taken re _ _ _ sses_ _ _ of your _ a _ s _ _ r
Instead, Deterding presents us with a list of ‘patterns, models and words for emotion and rule design’, that he has derived from games. Unfortunately, they are not as simple as ‘just add points’!.
I highly recommend watching Deterding’s presentation, it is an effective synopsis of a debate that is very much still in progress, and shows us why a simplified or direct application of Ralph Koster’s rules doesn’t work with non-games, despite what Volkswagen have been showing us. Instead, Deterding presents his own models, which are not as simple, or easy, and yet may turn out to be a more practical lesson for how we can apply knowledge from games to improve the user experience of mundane tasks.
