Getting the whole team involved should be a priority even with the smallest scale tests. Not only is it a vital opportunity to sell the UX process, but it’ll give an undeniably clear example of the benefits that UX testing can bring, and help secure funding to ensure the next round of usability and user experience testing will not be on such a small scale.

Phase 2 of Testing

How to Do it:

So how can we get the whole team involved when user testing is taking place? This can be divided by time frame into 3 key areas.

Before a user testing session:

• Inform everyone that the tests will be happening – send a group email, including details on what will be happening, and when, to all interested parties. This could be sent a week in advance, and on the day of the tests, which will increase awareness of user testing and allow you to….
• Invite people to spectate – Let people know that they can watch the user testing, and that their input would be valued. Perhaps doughnuts will prove a big enough incentive to get people to give up some of their time to spectate.
• Ask priorities and ‘goals’ for investigation – Encourage people to spectate and be invested in the process by asking what they’re interested in finding out, and incorporating this into the user tests. This will be followed up by a debrief, described later.

Getting the preparation before a user test correct will help increase awareness of what you do, and how you can help people, break down misconceptions about user testing, and get the team invested in the process. If a team feel like their priorities matter to you, and that they can help shapre the process to ensure it will help them, they’ll become proponents of user experience and usability testing.

During a session

• Set up a remote viewing session – This should ideally be in a shared conference room, which allows people to spectate the tests in progress. This can be done using cheap/free equipment such as webcams, team viewer, and IM clients. Encourage attendance by advertising free food, and letting people know their priorities will be incorporated.
• Make space in the test for questions from the team – At the end of the session, check with the team if there is anything they want to be asked, for example did they want to know more about a subject’s thoughts when displaying behaviour the team found interesting. Note that it’s important that these questions go through you, as this’ll prevent potentially leading questions getting through the net, and increase the validity of the participant’s responses.

After a session

• Run a group debrief after each session – After the user testing session, meet with all the spectators, and give them freedom to discuss what they saw, and their conclusions from it. This allows the team to share their findings, and will encourage them to get involved again, especially since you will…
• Incorporate their findings into your final report – and give credit!

You don't want to be accused of stealing credit...

Advantages

So why go to all the effort of getting people involved with user testing?

The first advantage is that it will give credibility to your findings – not only will the team have helped shaped the conclusions, but they will have seen firsthand the evidence of that behaviour. It’s pretty hard to deny that, for example, the player doesn’t notice when they pick up a power up, having seen them miss it all day.

A secondary advantage through involving them with the process is that they’re likely to be more invested and enthused about the process – the sort of buzz that can help you gain an investment of more time or money in user testing.

The most useful advantage of getting the wider team involved is that they’ll add their own expertise and critical eye, highlighting areas of importance to them, and helping make observations in areas that you may miss. This can only help your final report, and gives a much wider legitimacy to the user testing.

Making the whole team aware, and involved in the user testing process therefore provides advantages to everyone – not only yourself, but to the whole team. It’ll help promote the process, and sell further user testing, and therefore is a key aspect of selling UX testing to games companies.

Related posts:

1. Selling UX in games – everyone else is doing it.
2. Test with real users – not your team
3. Why you should always video players when testing games

I’d be interested in hearing your thoughts on this – please use the comments to add critiques or alternative viewpoints, as I’m sure there will be many! Anyhow, onwards with the 30 minute facebook game usability test.
I’ve been thinking about the social games found on facebook, and I think the most important part is the first 15 minutes of a new player’s experience, e.g. what happens in those fifteen minutes, do they understand the game, and do they want to come back and play again.

This means the most important aspect of user testing is understanding and improving the ‘new user’ experience. For this you’d need some people who’ve never played the game before (and who are close to the target audience of players you want to attract), and simply get them to play the game from the start (without special instructions, just how they would if it was on facebook). Maybe a small incentive (like pizza!) would be enough to recruit people for these short sessions.

a small incentive

Explain to the player that you want them to just play the game as they would at home, and that you are testing the game – not them. Pre planning some notes on this introduction speech will make it easier. It’s probably a legal requirement to let them know if the session is being recorded.

You should have interested developers watch them play through a remote desktop tool (but they shouldn’t interrupt the player!), or record the session to review later. Free tools, such as team viewer, are available to do this. This will be invaluable for getting their buy-in for further user testing, and will prove the benefit of user testing to an often dubious audience.

It’s common to use a ‘think aloud’ methodology, where you ask the player to speak their thoughts aloud as they play. It’s not perfect, but it’ll give some insight into what they’re thinking. You could prompt them by asking non-leading questions such as “what are you doing now” or “what are you thinking” if they’re silent for too long!

what are you thinking?

Then after the fifteen minutes (or completing the tutorial), you can ask them questions to gauge how they understood the game – did they know what they were doing, were they confused by the game, did they know what to do next, would they like to continue playing, etc. Maybe you saw some interesting behaviours while they were playing that you’d like to ask about. Often people don’t remember what they did, and so you’d have to prompt them, or maybe the developers watching have some questions to ask.

Hopefully this quick methodology will show where the user’s are getting confused, or not understanding what to do next, or getting bored, or any other factors that turn player’s away from the game in that crucial first play.

Jakob Nielsen said that most major usability problems can be found by 3-5 users, so you wouldn’t need to run this test many times per iteration of the game. As to when this should be done, its best to get started as soon as there is something playable, as changes will be easier to make the earlier they are found, and then a similar test can be repeated with the next version of the game. Many social games go through an agile design process, with many iterations, and so this process will fit in well.

What do you think? Does this sound helpful? Or do you think that the ‘new user’ experience is not the most important part of a facebook game – maybe converting players to paying players is instead… let me know your thoughts in the comments here, and we can improve this 30 minute facebook game test.

Related posts:

1. Test with real users – not your team
2. How real-world game usability testing is changing
3. Game Usability: Advancing the Player Experience Book Review

What is the environment?

With user testing, the environment comes down to the context in which the software or product is used. This includes where the user is, when they are using something, and who they are using it with.  It’s important to not forget the factor that the environment plays when testing a product, otherwise your final results, conclusions and design decisions may end up way off the mark.

Considering the environment for testing is important for both usability and user experience.  For usability testing, a realistic setting is needed to understand how a system will perform in that environment when out in the field. User experience focuses on how a system makes the user feel, and to truly understand this, you need to test users in an equivalent setting. This is why you’ll find comfy sofas wherever games are tested, since games are typically played at home in front of the TV.

pizza boxes are optional

Getting it wrong

Common mistakes with user testing involve inaccurately representing the environment in which a product is to be used (poor “ecological validity”).

This could include an unfair representation of the conditions in which the product would be used – for example testing a car radio without having the user drive would not give a true understanding of how the radio would function when the user’s attention is diverted.

Another common mistake is failing to test things in a time sensitive manner. For example, ticket machines are often used in high stress situations, minutes before the train is going to leave. Giving the user ample time in a comfortable lab setting does not recreate a typical interaction with the ticket machine, and would give an unrealistic impression of its usability.

User tests should also be separated, though concurrent, to the design process, and the development team should not be allowed to run the user sessions (though it’s a good idea to have them spectate from afar). Having the team in the same room as the player can intimidate them into praising the game or product, and reduce the validity of the results.

you liked it... got that?

Getting it right

So how can you be sure to get the environment right for testing?  The following things must be kept in mind:

• The right place: The ideal place would be the actual environment in which the game or product would be used. However this isn’t always possible, and so it’s important to recreate the environment as closely as possible. It’s also a good tip with games to test somewhere away from where the games are made – a typical player would be excited to be at a game studio, and their opinions would be nicer, and more eager than a typical home experience.
• The right mood: The sofa, and snacks mentioned in the introduction are all part of making the environment comfortable for the user, so that they feel at ease when testing the product or game. Another advantage to the previously mentioned idea of running these tests off-site, away from the people who made the product, is that the user won’t feel intimidated into praising the product, and they’ll be free to give a true opinion.
• The right people: It’s important to recruit the realistically, and test with real users. I’ve written before about how you should test with real users, not your team, and it is one small step which can increase the the validity of your results massively.
• The right friends: Remember when testing multiplayer games that a new environment will initially make the users uncomfortable, and this is magnified if they are playing with strangers. If you want to see how your game is received when played among friends, or with families, there is no shortcut – you have to test with groups of friends to get a true representation.

Taking care when selecting the right environment is important whether you are making a game, a ticket machine, or a washing machine for your cat.  A suitable environment for user tests will increase the validity of your results, and ultimately help make informed design decisions.

Related posts:

1. Test with real users – not your team
2. No user testing? Oops! – The Digiscent iSmell
3. Why you should always video players when testing games

Game development teams need a constant supply of fresh users to test the ‘new user’ experience with. I’ve seen teams keep their project secret from their colleagues, not for official reasons, but so their colleagues can be tested as ‘new users’. Other teams test their games with their HR and secretarial staff, since they are unlikely to have had much exposure to the game.

However it’s a good guideline to never test with your team (unless of course you are building something for them). It’s understandable why this situation arises – often budgets are too tight for intensive user testing, forcing teams to perform ad-hoc tests with their colleagues; however this often causes problems further down the line:

such as ...accidents

Dont test with your team

• Your team are not your users – Unless you are in a very specialist field, or are developing an internal project like an intranet, it’s unlikely that your team are the same people as your users. And they are very unlikely to act in the same way a typical user would.
• Your team know things users wouldn’t – It’s likely your team will have had prior exposure to your game or application that a new user wouldn’t, and will be bringing prior knowledge to the testing session. This also applies to people who do not work directly on your team. To get a true outside perspective, you need to seek outside users.
• Your team know you – Unlike a stranger, your team are already know you, and (hopefully) like you. Their answers, and interactions will be biased to please you, and tell you what you want to hear based on what they know about your job, the project your working on, or your beliefs (for example, attempting to validate your design choices).

Advantages of testing with real users.

How people act can often be surprising. If this wasn’t the case, there would be little point in user testing. That’s why it’s extremely important to gather real data, from the people who will actually be using your product. Only real users will approach your product from an authentic ‘new user’ angle, and give an insight into how your product will be perceived and used.

Getting real users involved with product development will get them engaged with the product. Asking their opinions, and being interested in their experience will make the user feel positive about you, and your product, and will mean they will be more likely to purchase it when it’s ready. In newsrooms, this has been widely known for years – hence the proliferation of lists of names in local papers.

Most importantly, involving users will get them talking about your product, generating true grass roots ‘word of mouth’ promotion (hopefully without breaking any NDA’s!). Giving customers an early exposure to your product can build excitement, and market your product for free!

saving millions on megaphones

Conclusion

Finding real users can be cheaper than you think. Not only is it possible to pull people off the street, using the methods suggested above, but new usability testing methods such as remote user testing allow you to find and test real users from the comfort of your office, for very low cost. These days there’s almost no excuse not to test with real users, and it can be just as easy as testing with your team, with much more rewarding returns.

Related posts:

1. The thirty minute facebook game usability test
2. The right environment for user testing
3. How real-world game usability testing is changing

Surveys are a common tool used to evaluate a participant’s opinions of the user experience, and usability of a system. I’ve written about how to make good questionnaires before, and have often seen them used as a tool when analysing a large group of participants. However, as a method of understanding users, they are imperfect, and not just because they are poorly designed – instead it’s a fundamental problem with surveys. Let’s look at why this is the case, and why people are tempted to use surveys despite this.

Where are surveys used?

When I’ve been involved with user tests for games, I’ve often seen surveys used as a way of recording the player’s experience. For example, after completing a level, or game mode, they would be asked to rate their experience on a Likert scale (1-10), on categories such as how difficult they found the level, how fun it was, how it compared to other levels. This is often complemented by text notes, where the participant can write in things they particularly liked or disliked.

Outside of gaming, surveys can often be found on the internet – such as website’s satisfaction surveys, or on professional survey sites, like Survey Monkey.

Survey Monkey in action

Why are surveys used?

It’s easy to understand why surveys are often used when testing user experience. Most obvious is that they are easy to quantify, since the scores are given as a numeric value, which can then be averaged, and given an overall ‘score’. This can then be stuck on a graph, to impress people too busy and important to be involved with the testing itself. Compared to moderated testing, simple analysis is easy, and ‘results’ can be gained with little effort – particularly if an online survey tool is used.

Similarly, with surveys it’s easy to get a large number of opinions quickly, and in a largely un-moderated setting. Hence, 10 (or 10,000) people can test a game at the same time, with only light moderation, and fill out a survey after to record their views. Surveys also don’t require a large degree of specialist equipment – just a printer, and a pen (or they can be done online). This makes them cheaper than many moderated settings, which require a lab decked out with recording equipment.

Problem with surveys

Surveys sound great, don’t they. Cheap, Easy, and give some hard numbers. However, there are a number of problems with surveys, and one key issue that prevent them being suitable for user experience analysis.

First of all, it’s easy for the data from surveys to be misrepresented (either unintentionally or to further a top secret agenda!). Without hard evidence, such as watching (and recording) an individual player of the game, the analysis becomes reduced to which level ‘scores better’, regardless of the intricacies of the play test. Minor issues become lost within the overarching ‘score’.

Much more importantly, the fundamental problem with attempting to understand user experience with a survey is that they log opinions, and not behaviour. People are (sometimes?) stupid, and don’t know what they think. So a player who has had a positive experience throughout a level, and got stuck near the end, will often be left thinking poorly of the entire level. And without an independent observer to monitor, their in-game opinions are lost, or forgotten. Just like I cannot tell how bad my singing is, a player is too close to the subject matter to gain a full understanding of it.

Its pretty bad...

Essentially, surveys introduce a layer of abstraction from the game that is difficult for a player to follow. It is difficult for them to recognise what parts of a game made it fun, and which parts frustrated them, and it often takes someone else to spot these patterns.

Pride, and psychology can also be a contributing factor – players who have needed 10 attempts to complete a section will still say it was “easy” after finally completing it – psychologically they will often believe it as well, since they have felt the satisfaction of completing the task. Other times they will be too proud to say the section was too difficult, and lie. Again, this rich data is lost through a survey.

What should be used instead?

To gain a truer understanding of the user experience (or player experience) of participants when testing a system, or a game, surveys are therefore inadequate. Instead, a moderated task based analysis session, which is recorded for later analysis, will give a truer understanding of how the participant found the system, and their true experience, unaltered by their own perceptions. I’ve written about recording these sessions before, and will discussed them further in the future.

As we have seen, surveys are cheap and easy, and hence should not be disregarded entirely. However they should not be used exclusively, as they can miss key user experience findings, and require users to know themselves, and their feelings, extensively.

Related posts:

1. Why you should always video players when testing games
2. A Terrible User Experience & how to fix it– Zoomerang.com
3. The right environment for user testing

Don't judge it by it's cover...

Why would you consider remote research?

Written by advocates of remote research, the book highlights many of the potential advantages that remote research gives compared to a more traditional lab based study. These advantages are fleshed out throughout the book through testimonies of experts who have experience in this field, who offer real world examples to emphasise these points.

Some key advantages are:

• Access to a geographically diverse user base. Unlike traditional research, where a moderator would have to be in the same physical location as the subjects, remote research allows a study to be run with anyone who has a high speed internet connection, widely expanding the potential study-group.
• Easy to let stakeholders get involved. Because the research session is being broadcast over the internet, it’s possible to allow stakeholders (i.e. executives and designers) to view the session, and give (moderated) input. This of course increases their engagement with the process, and will be the ‘evidence’ for any conclusions derived from the research.
• Natural browsing environment. The validity of the research can be improved, not only because you are allowing the user to perform the task in a familiar environment (their own home computer), but also some recruitment methods allow you to capture a user performing a task they have selected. For example, recruiting a user who came to the site to buy trousers, for a task based on buying trousers, would provide more accurate results than asking someone to pretend to buy trousers…
• Cheaper (debatably). Not having to pay for travel can keep costs down, however other costs, such as incentives, will still be required, as well as paying for the software.

The remote research book doesn’t advocate killing off lab tests though – instead, it recognises that there are cases when the lab is still appropriate, such as when privacy is a concern. The book also features Andy Budd’s defence of the lab, which argues that remote research fails to pick up aspects of non-verbal behaviour, as well as arguing that remote research doesn’t just remove a selection bias (geography), since it also adds another (internet speed and technical ability). It’s brave of the book to include the case against remote research, and helps project a more trustworthy and reliable image for the book itself.

How to do remote research

The ‘meat’ of the book are the sections dedicated to how-to guides on the different forms of remote research. The book contains step by step instructions on performing moderated or un-moderated research, and includes key topics such as recruitment (and live recruiting), card sorts, and lots of handy hints – such as using IM clients as a chat room for multiple observers to automatically share and timestamp notes.

The book doesn’t just cover basic topics – it goes on to develop novel approaches to user research, such as using ‘reverse screen sharing’ to protect confidential software or data, and using mobile web to gain a new understanding of time-dependant information, outside of the traditional moderated setting.

It also extends the remits of remote research – it doesn’t have to just be websites, but can include doodles or sketches, as well as developing ideas for automatic research with analytics.

Another sort of remote research?

Conclusion

Remote Research is one of the easiest to read UX books I’ve reviewed. Like many Rosenfeld publications, it is laid out well, without appearing dense with text, and has a friendly tone throughout. The book can be likened to Krug’s writing in its style, and presentation.

The book is also practical and realistic, and deals with real world issues, like ‘fakers’ (who can be outed by using open ended questions to discover motives), legal issues, and common challenges such as reluctant stakeholders.

Most importantly for the practical UX practitioner, the book is not dogmatic. This is especially evident in the last chapter which admits that usability shouldn’t be the exclusive goal of product design, and needs to be coupled with initiative, and innovation to develop great things.

Overall this book is a great introduction, and how-to guide to the growing field of remote research, and will be an important tool for anyone trying to keep up to date with the latest research methods.

Related posts:

1. Rocket Surgery Made Easy by Steve Krug – Book Review
2. Evaluating User Experience in Games – Book Review
3. Selling Usability by John Rhodes Book Review

The UXBrighton event was presented in a new format as a series of short talks, from Harry Brignull’s tips on time stamping notes, to Danny Hope’s templates for understanding user roles. Also interesting was a talk on using google analytics, although the length of the talk meant that topic could only be skimmed, dissapointing as I’m an analytics fan. The most interesting idea presented was iCrossing’s presentation on “The iCrossing Connected Brand index: how to measure a brand’s effectiveness online”, given by Ifraz Mughal.

Expert Evaluation

As I’ve mentioned before an expert evaluation is a useful tool for getting an insight into potential usability and user experience issues on a website, or game, with limited resources. Although it can never replace running tests with real users, it can provide a quick approximation, and help highlight the biggest issues.

The ‘method’ for an expert evaluation is simple. Get an expert to look at the site, or game, and tell the client what they think. Job done.

My expert eye tells me you need smarter users...

However an expert evaluation can only ever be subjective, and this is it’s biggest weakness. A client can look at your page full of recommendations, and dismiss it as the opinion of one person. There’s no easy way to see progress with changes, and a comparison with other sites can only ever be abstract.

Quantifying an Expert Evaluation

iCrossing’s solution is to quantify their expert evaluation. As part of their ‘Connected Brand Index’ idea, they rate their clients sites (and competitors), on UX-centric areas such as “usefulness”, “usability” and “desirability”.

A traditional expert evaluation would give a qualitative rating, and give examples to back this up, i.e. “Poor – little emphasis, and diffused call to actions”. Instead iCrossing will give the site a score, on a scale of -2 to 2 (2 being very good). This of course can be backed up with examples in a more in depth report.

after the first few pages, the report can just be pictures of kittens. No-one reads that far.

The advantages:

There are numerous reasons why a client would prefer a scored ‘rating’, rather than comments.

• A ‘score’ makes it easy to benchmark, and compare your own scores against competitors. By dividing the expert evaluation into separate topics, and scoring each, a finely grained comparison can be made, and communicated
• Similarly, a score makes it easy for a client to see progress. If they scored -1 before hiring you, and 1 after, your work can be justified (as long as no-one questions who is doing the scoring!)
• Because this produces a concrete score, clients will be able to handle and communicate the data. Graphs can be made, which wouldn’t be possible for subjective comments. These can be invaluable for justifying and communicating with managers and project sponsors, who do not need to see the details, just get a high-level overview.
• This expert evaluation can be encompassed as one aspect of a larger ‘score’ given to websites, or games. This is the idea behind iCrossing’s connected brands index.

Conclusions:

There is an argument this can be seen as a bit of a scam. Giving arbitrary numbers to your opinions doesn’t make them any less subjective. This method of presenting the data could be misleading if presented incorrectly, and the client should be made aware of the method behind the score system. This could become an issue when running comparative studies before and after your work, since you’d be biased towards giving the site a better score after you’ve worked on it.

The point of this method is to aid communication with the client, and give them data in a format that is useful to them. As I discussed in the review of Selling Usability, management and non-technical people would typically much rather see pretty graphs, and statistics, than a list of comments. This method helps manage client expectations, and gives them what they want.

To make the method more valid, it would be useful to perform a study to ensure the method is sound. Perhaps get a wide range of experts to independently rate a wide range of websites on this scale, and note the correlations between the scores. It’d be first step in countering complaints that this method is still inherently subjective, and help make an art into a science.

Related posts:

1. Conducting an Expert Review
2. How to present a UX Report
3. Understanding players through biometrics

The task was relatively simple – design some worksheets, including textboxes, and pictures, and lay them out in an eye-pleasing manner. I’m sure there are many more appropriate packages to make this in than Word, but it was the user’s software of choice, due to familiarity, and the task shouldn’t be beyond MS Word. I observed, and let them lead the interaction, but advised when they asked for help (naughty I know, but it wasn’t a formal lab setting!)

The lab was busy that day anyway...

How my ‘less-technical user’ used Word:

I noted down (obviously away from the user) some of the more ‘interesting’ characteristics of how they used Word.

• Used the ‘cut’ function as a ‘delete’ (with no understanding of how it links to paste). Taken out of context from “cut and paste”, ‘cut’ would more likely imply removing or ending something, and so this mistake is understandable. Incidentally this method has some pluses. I still don’t know how to remove a table easily (not just the information within it), and cut seems to do this.
• No knowledge of the alignment tools, and so using spaces as a method to align text to the center or right. Obviously ran into problems when editing the text later, as changes would make the text run over the end of the line, ruining the formatting.
• Drew horizontal lines, across the page (i.e. a space to write in your own answer) with –‘s. Seems a pretty effective method, even though I’m sure Word has its own way of doing this. Is there a better way of doing it?
• Displayed difficulty moving images in Word. Is it right that you have to click on an image twice to move it? The first click just gives you resize options, which confused the user.
• Had difficulty with resizing objects. What happens if you make an image so big that it falls off the edge of the paper, and you cannot see the border to make it small again? I guess you could format picture, and manually change the size, but this is an entirely different method of resizing, and isn’t cognitively related to the standard way.
• Constant (constant!) rewriting of words, when word autocapitalised/auto formatted them in an undesired way (which was seemingly every autoformat). User had to delete the word, and re-write each time.

What could word do to improve?

This immediately throws up some questions about how Word was developed. It’s clear that the tools available, such as the alignment, or horizontal lines, are not making their functionality transparent to new users. It wasn’t clear to my user that they existed, or how they should be functioning. Obviously just having the icon on the toolbar isn’t enough, and this should be rethought.

This was also the case with image manipulation. The functions that the user needed do exist in Word (i.e. resizing, moving), but are modal in nature, and so are difficult to find, and don’t offer a consistent user experience to someone who is not familiar with Word’s nuances.

It’s also clear with auto format in particular that the system isn’t adapting to the user’s needs. The constant changes that Word was making to the user’s document, which were then undone each time only created a large degree of frustration in the user. The software should be learning how the user wants auto format to work, and adjust to their preference. In this user’s case, it was causing trouble, and should have turned itself off (or at least given the option)

What they need is some sort of helpful assistant

What should we learn from this?

It occurred to me that these issues were not unique to the user I watched since I encounter similar problems with Word. The difference is I’ve had enough familiarity to learn the workarounds, or solutions to these problems that Word throws at you. For example, it’s an unthinking reaction to press Ctrl+Z after Word incorrectly auto-formats things incorrectly. My user just hadn’t used the program for long enough to train that reaction, and so word’s error became more of a big deal.

Its important when considered usability to realise that users aren’t just like you. If you are in a position to make a difference with usability, it’s very likely you are not an ‘average user’, and as such its difficult to comprehend how ‘average users’ use software.

‘Average users’ are not stupid. They are your mum, and just don’t have the time, or effort to put into learning these workarounds, or making them second nature. The solution, rather than ‘educating’ users, is to make the programs better; make programmers understand who their users are, and how they use the programs. And make them program for the ‘average’ users, rather than the power users. And that is the point of usability.

Related posts:

1. Test with real users – not your team
2. The user experience of Ticketmaster ruins Christmas

Krug’s previous book, Don’t Make Me Think, aimed to introduce the idea of web usability to a general audience, and had a number of key characteristics. It was short, designed to be read during a plane journey, and it was accessible, so that anyone could read it and understand usability. It also had a chapter on performing usability testing, which was removed in the second edition. Rocket Surgery Made Easy is that chapter.

In the decade since the emergence of this first book, Krug has been honing his usability testing skills, and through running numerous workshops, has perfected how to teach usability testing. This book is the summary of that experience, and in a wonderfully brief and readable style, is the only book someone new to user experience will need to start performing usability testing. Looking at that statement in more detail will give a deeper insight into the book.

Pictured: Steve Krug, international man of mystery

“The only book”

Krug’s book is a complete introduction to testing your company’s website, and guides the reader through every step of the process, from recruitment, and pre-test setup (including the equipment required), through to running the session and the debrief. The book also offers all the test-scripts and task cards you’ll need. Throughout the process Krug offers his best practise techniques, useful maxims to remember, and pitfalls to avoid. Essentially, you could start performing usability tests with this book alone as a reference.

“someone new to user experience”

This book aims to be an introduction, and so requires no prior knowledge (although reading his other book wouldn’t be a bad start). It also assumes that you are not a usability professional, just an interested party at a company, and that there isn’t a large degree of company buy-in, and believes that many rounds of semi-amateur testing is superior to one round of professional testing, which is valid. Because it assumes you don’t have a large amount of support from your company, in particular monetarily, it focuses on how to perform usability testing with a large degree of constraints – how to do it cheaply, how to do it quickly (just one morning a month) & how to get people interested through pastry!

“will need to start”

Because of its small size, Krug’s book is only designed as an introduction to usability testing, and won’t contain any new information if you’ve done this before. At best you could pick up a few best practises, but you’re not the intended audience. Just for you, the book has suggestions for further reading “for overachievers only”, with some interesting books to continue studying this field, but the book is largely aimed at newcomers to the field.

“performing usability testing”

Also due to its brevity, the book only covers one on one think aloud tests with websites, and only hints that other forms of usability testing possible. The book ends with a short chapter on remote research, but only hints at the possibilities that we saw at the UX Brighton event. Other types of usability research, such as quantitative methods, expert analysis, or focus groups, are left out entirely of the books remit. Although the book offers a complete solution, it’s a complete solution for one particular type of usability testing, and only covers websites (although the skills could easily be applied elsewhere).

Its not rocket scien.... oh.

Conclusion

Krug’s new book is short, and simple. However neither of these points are negative. It’s short because it has to be, to introduce a wide audience to usability testing, and to emphasise how easy it is to begin. It’s simple, because usability testing can be simple, and done for just a morning a month. If you’re someone who has been doing usability testing for years, or a professional who does a large amount of usability testing, perhaps it’s better to look elsewhere. However if you’re looking for an introduction to usability testing, having never donae it before, or just need a short walkthrough guide to getting usability testing started at your workplace, Krug’s book should definitely be your first choice.

Related posts:

1. Effective UI by the EffectiveUI team – Book Review
2. Remote Research – Book Review
3. Game Usability: Advancing the Player Experience Book Review

Recently I’ve been involved with the testing of the user experience for a number of high profile upcoming games. The results of these sessions have always been very interesting, and undeniably shown that running tests with real players gives a true understanding of the issues in the games, finding issues that often cannot be seen by the developers. However, all the places I’ve worked at have all been making the same, easily fixable mistake of not videoing the player’s reactions while playing.

a mistake never made by this man

There are many possible excuses given why people running player tests on games fail to record the player. In a hectic working environment, it can be difficult to find time to review the video with tight deadlines hanging overhead. Similarly, the initial set up costs, of a camera or software that will record the player seems like an area where cutbacks can be made with a tight budget. Often tester’s will try to justify this – surely they can get the same information, and cheaper, by using surveys and interviews after the event?

Here is just of few of the things missed when you fail to video a player’s experience

• A timeline of the player experience, not just their end of play feelings
• The ability to pinpoint the ‘key’ fun moments
• The ability to pinpoint the dullest moments.
• The ability to verify the opinions given to you by a post-game survey
• A key ‘visual’ way of showing the stakeholders real player’s experiences (useful for presentations)

Would this experience be the same without video?

Having this data leads to the production of better games – the incredibly dull level, that is only redeemed by an exciting boss battle need not be an incredibly dull level, and videoing the players can mean the difference between picking up on these moments and missing them. If you miss them, you’ll never fix these problem moments. It is well worth the time to review what aspects of the game the player especially enjoyed, or hated, without the ‘bias’ introduced by the time delay before getting their opinions. If you are asking them to ‘think aloud’ (and you should be!), a video will also keep notes of the opinions given throughout the session. The problems identified can then be fixed.

User (or player) experience is of utmost importance for game designers – unlike software, which businesses can force upon their employees, everyone who plays games does so because they enjoy doing so. Videoing their reactions can lead to a more enjoyable game, and makes your game more consistently fun, and how can that be a bad thing?

Related posts:

1. Games Usability Testing is not QA!
2. Understanding players through biometrics
3. The right environment for user testing