Archive for the ‘UX Guides’ Category
Test with real users – not your team
‘Free pizza and coke! Just play our game for an hour’. Sounds like a good deal right? And pretty easy to organise, just pulling kids of the street. It can even be done in the pub, for mobile devices. Even this ‘free pizza’ recruitment is better than testing your game (or website, or application) with people from within your office. But why?
Game development teams need a constant supply of fresh users to test the ‘new user’ experience with. I’ve seen teams keep their project secret from their colleagues, not for official reasons, but so their colleagues can be tested as ‘new users’. Other teams test their games with their HR and secretarial staff, since they are unlikely to have had much exposure to the game.
However it’s a good guideline to never test with your team (unless of course you are building something for them). It’s understandable why this situation arises – often budgets are too tight for intensive user testing, forcing teams to perform ad-hoc tests with their colleagues; however this often causes problems further down the line:
such as ...accidents
Dont test with your team
- Your team are not your users – Unless you are in a very specialist field, or are developing an internal project like an intranet, it’s unlikely that your team are the same people as your users. And they are very unlikely to act in the same way a typical user would.
- Your team know things users wouldn’t – It’s likely your team will have had prior exposure to your game or application that a new user wouldn’t, and will be bringing prior knowledge to the testing session. This also applies to people who do not work directly on your team. To get a true outside perspective, you need to seek outside users.
- Your team know you – Unlike a stranger, your team are already know you, and (hopefully) like you. Their answers, and interactions will be biased to please you, and tell you what you want to hear based on what they know about your job, the project your working on, or your beliefs (for example, attempting to validate your design choices).
Advantages of testing with real users.
How people act can often be surprising. If this wasn’t the case, there would be little point in user testing. That’s why it’s extremely important to gather real data, from the people who will actually be using your product. Only real users will approach your product from an authentic ‘new user’ angle, and give an insight into how your product will be perceived and used.
Getting real users involved with product development will get them engaged with the product. Asking their opinions, and being interested in their experience will make the user feel positive about you, and your product, and will mean they will be more likely to purchase it when it’s ready. In newsrooms, this has been widely known for years – hence the proliferation of lists of names in local papers.
Most importantly, involving users will get them talking about your product, generating true grass roots ‘word of mouth’ promotion (hopefully without breaking any NDA’s!). Giving customers an early exposure to your product can build excitement, and market your product for free!
saving millions on megaphones
Conclusion
Finding real users can be cheaper than you think. Not only is it possible to pull people off the street, using the methods suggested above, but new usability testing methods such as remote user testing allow you to find and test real users from the comfort of your office, for very low cost. These days there’s almost no excuse not to test with real users, and it can be just as easy as testing with your team, with much more rewarding returns.
The Problems with Surveys for User Experience Tests
In the run up to Margaret Thatchers’ election victory in 1979, a poll was taken to estimate who would vote for her. Only 1 in 100 said yes. However, as revealed by the final results, 1 in 3 actually voted for her. The poll was inaccurate, and inappropriate for the task.
Surveys are a common tool used to evaluate a participant’s opinions of the user experience, and usability of a system. I’ve written about how to make good questionnaires before, and have often seen them used as a tool when analysing a large group of participants. However, as a method of understanding users, they are imperfect, and not just because they are poorly designed – instead it’s a fundamental problem with surveys. Let’s look at why this is the case, and why people are tempted to use surveys despite this.
Where are surveys used?
When I’ve been involved with user tests for games, I’ve often seen surveys used as a way of recording the player’s experience. For example, after completing a level, or game mode, they would be asked to rate their experience on a Likert scale (1-10), on categories such as how difficult they found the level, how fun it was, how it compared to other levels. This is often complemented by text notes, where the participant can write in things they particularly liked or disliked.
Outside of gaming, surveys can often be found on the internet – such as website’s satisfaction surveys, or on professional survey sites, like Survey Monkey.
Survey Monkey in action
Why are surveys used?
It’s easy to understand why surveys are often used when testing user experience. Most obvious is that they are easy to quantify, since the scores are given as a numeric value, which can then be averaged, and given an overall ‘score’. This can then be stuck on a graph, to impress people too busy and important to be involved with the testing itself. Compared to moderated testing, simple analysis is easy, and ‘results’ can be gained with little effort – particularly if an online survey tool is used.
Similarly, with surveys it’s easy to get a large number of opinions quickly, and in a largely un-moderated setting. Hence, 10 (or 10,000) people can test a game at the same time, with only light moderation, and fill out a survey after to record their views. Surveys also don’t require a large degree of specialist equipment – just a printer, and a pen (or they can be done online). This makes them cheaper than many moderated settings, which require a lab decked out with recording equipment.
Problem with surveys
Surveys sound great, don’t they. Cheap, Easy, and give some hard numbers. However, there are a number of problems with surveys, and one key issue that prevent them being suitable for user experience analysis.
First of all, it’s easy for the data from surveys to be misrepresented (either unintentionally or to further a top secret agenda!). Without hard evidence, such as watching (and recording) an individual player of the game, the analysis becomes reduced to which level ‘scores better’, regardless of the intricacies of the play test. Minor issues become lost within the overarching ‘score’.
Much more importantly, the fundamental problem with attempting to understand user experience with a survey is that they log opinions, and not behaviour. People are (sometimes?) stupid, and don’t know what they think. So a player who has had a positive experience throughout a level, and got stuck near the end, will often be left thinking poorly of the entire level. And without an independent observer to monitor, their in-game opinions are lost, or forgotten. Just like I cannot tell how bad my singing is, a player is too close to the subject matter to gain a full understanding of it.
Its pretty bad...
Essentially, surveys introduce a layer of abstraction from the game that is difficult for a player to follow. It is difficult for them to recognise what parts of a game made it fun, and which parts frustrated them, and it often takes someone else to spot these patterns.
Pride, and psychology can also be a contributing factor – players who have needed 10 attempts to complete a section will still say it was “easy” after finally completing it – psychologically they will often believe it as well, since they have felt the satisfaction of completing the task. Other times they will be too proud to say the section was too difficult, and lie. Again, this rich data is lost through a survey.
What should be used instead?
To gain a truer understanding of the user experience (or player experience) of participants when testing a system, or a game, surveys are therefore inadequate. Instead, a moderated task based analysis session, which is recorded for later analysis, will give a truer understanding of how the participant found the system, and their true experience, unaltered by their own perceptions. I’ve written about recording these sessions before, and will discussed them further in the future.
As we have seen, surveys are cheap and easy, and hence should not be disregarded entirely. However they should not be used exclusively, as they can miss key user experience findings, and require users to know themselves, and their feelings, extensively.
Telling Tales – Stories for promoting user experience.
Stories have long been an important way of recording and imparting information, as evident through the survival of folklore, myths and songs from our past. As a tool for communicating, and retaining information, they are highly valuable, and the principles of this can be applied to the promotion of User Experience.
Why do teams need to understand UX?
When working within a team, whether it’s a small web development company, or a large multi-national company, it’s impossible for one person to oversee every aspect of development. That is why it’s important for everyone to understand the principles of user experience, due to the large variety, and span of tasks that it encompasses. A good user experience designer is not the one who works longer hours, or has more creative control over the final product, instead they are the one who makes sure everyone understands UX goals, and works towards them.
It is this reason that books such as Krug’s Rocket Surgery Made Easy heavily promote the idea of getting everyone, from the CEO to a junior developer, involved when testing with real user’s, as it is these people who will be able to implement user experience ‘fixes’. The solution Krug suggests involves inviting these people to view the user testing sessions, and bribing them with snacks.
Because encouraging successful user experiences requires everyone’s involvement, it’s important that everyone understands the goals of user experience, and can incorporate them into their work. The problem is making them understand what those goals are.
Pictured: not a goal
What are stories good for?
Homer’s Odyssey was composed in the 8th century BC, and yet a canonical written version wasn’t produced for another 300 years. Before this, the tale was remembered, and shared orally, through song. Considering that the Odyssey contains over 12,000 lines, this is no small feat. That’d be like remembering, and reciting on demand, every blog post I’ve done so far… So how was it possible?
The answer lies in the power of stories. Stories, like Homer’s Odyssey, are memorable, due to the plot and the events contained within, and hence easily form visual images within the mind, and aid memory. Plots make stories interesting, and hence easier to remember than non-fictional texts (such as remembering all of my blog posts).
Stories can also be used to impart useful information. These be obvious life lessons (The Three Little Pigs teaching the benefit of putting adequate effort into your tasks), or life saving, such as Ring a Ring o’ Roses imparting symptoms of the Black Death (although this may not be true)
or Cinderella teaching us the dangers of stepmothers
Using stories for UX
So, stories are memorable, can be used to impart useful information, and are easily transferred, as made evident by the 300 years that Homer’s Odyssey lasted without being written down. Hence they seem ideal for the promotion of UX within teams, as they can be used to quickly impart the benefits of UX, and what each individual needs to do to aid a user centred design process, in a memorable and transferable fashion. By bringing the whole team onboard with the principles and goals of UX, a greater degree of coherence can be achieved across the team. Therefore to get better results without manually verifying everything the team does, a User Experience Designer needs to champion user experience principles with all members of the team, and stories are an easy way to do this.
Some examples of UX stories
Many authors have understood how stories help to impart information in a memorable fashion.
- In The Design of Every Day Things, Don Norman uses stories of real world examples to emphasise the dangers of ignoring user centred design, with examples including nuclear accidents and plane crashes.
- Stories based on metaphors are also used by Alan Cooper in The Inmates Are Running the Asylum to emphasise the importance of designing for the end user, by comparing the software design process to film making.
Also of interest, Sam Nixon shared a short story by David Travis called “The Fable of User Centered Design”, which aims to bring clients and team members on board with the benefits of a user centred design approach. Definitely worth the read, and should be shared. The book is available to download from his website.
If anyone has other examples of stories that help define the importance of user experience, remember to leave a comment below!
Quantifying the unquantifiable – Expert Evaluations
At a recent UXBrighton talk, iCrossing presented an interesting idea about applying metrics to expert evaluation. This is a potentially controversial topic, yet has numerous benefits if it can successfully make qualitative data quantitative (and turn impressions and thoughts into numbers). I’ve outlined the method, and my thoughts on the issues around this.
The UXBrighton event was presented in a new format as a series of short talks, from Harry Brignull’s tips on time stamping notes, to Danny Hope’s templates for understanding user roles. Also interesting was a talk on using google analytics, although the length of the talk meant that topic could only be skimmed, dissapointing as I’m an analytics fan. The most interesting idea presented was iCrossing’s presentation on “The iCrossing Connected Brand index: how to measure a brand’s effectiveness online”, given by Ifraz Mughal.
Expert Evaluation
As I’ve mentioned before an expert evaluation is a useful tool for getting an insight into potential usability and user experience issues on a website, or game, with limited resources. Although it can never replace running tests with real users, it can provide a quick approximation, and help highlight the biggest issues.
The ‘method’ for an expert evaluation is simple. Get an expert to look at the site, or game, and tell the client what they think. Job done.
My expert eye tells me you need smarter users...
However an expert evaluation can only ever be subjective, and this is it’s biggest weakness. A client can look at your page full of recommendations, and dismiss it as the opinion of one person. There’s no easy way to see progress with changes, and a comparison with other sites can only ever be abstract.
Quantifying an Expert Evaluation
iCrossing’s solution is to quantify their expert evaluation. As part of their ‘Connected Brand Index’ idea, they rate their clients sites (and competitors), on UX-centric areas such as “usefulness”, “usability” and “desirability”.
A traditional expert evaluation would give a qualitative rating, and give examples to back this up, i.e. “Poor – little emphasis, and diffused call to actions”. Instead iCrossing will give the site a score, on a scale of -2 to 2 (2 being very good). This of course can be backed up with examples in a more in depth report.
after the first few pages, the report can just be pictures of kittens. No-one reads that far.
The advantages:
There are numerous reasons why a client would prefer a scored ‘rating’, rather than comments.
- A ‘score’ makes it easy to benchmark, and compare your own scores against competitors. By dividing the expert evaluation into separate topics, and scoring each, a finely grained comparison can be made, and communicated
- Similarly, a score makes it easy for a client to see progress. If they scored -1 before hiring you, and 1 after, your work can be justified (as long as no-one questions who is doing the scoring!)
- Because this produces a concrete score, clients will be able to handle and communicate the data. Graphs can be made, which wouldn’t be possible for subjective comments. These can be invaluable for justifying and communicating with managers and project sponsors, who do not need to see the details, just get a high-level overview.
- This expert evaluation can be encompassed as one aspect of a larger ‘score’ given to websites, or games. This is the idea behind iCrossing’s connected brands index.
Conclusions:
There is an argument this can be seen as a bit of a scam. Giving arbitrary numbers to your opinions doesn’t make them any less subjective. This method of presenting the data could be misleading if presented incorrectly, and the client should be made aware of the method behind the score system. This could become an issue when running comparative studies before and after your work, since you’d be biased towards giving the site a better score after you’ve worked on it.
The point of this method is to aid communication with the client, and give them data in a format that is useful to them. As I discussed in the review of Selling Usability, management and non-technical people would typically much rather see pretty graphs, and statistics, than a list of comments. This method helps manage client expectations, and gives them what they want.
To make the method more valid, it would be useful to perform a study to ensure the method is sound. Perhaps get a wide range of experts to independently rate a wide range of websites on this scale, and note the correlations between the scores. It’d be first step in countering complaints that this method is still inherently subjective, and help make an art into a science.
The Uncanny Valley of Wireframes
The uncanny valley is a theory describing how, as games and robotics produce more accurate representations of humanity, people’s reactions towards them are increasingly negative. This is also true with the production of wireframes, and in user experience testing and is something user centred designers need to be aware of.
The uncanny valley was originally discovered in the field of robotics, but also frequently applies to video games. It describes a phenomenon with replications of humans, whether they are life-like androids or avatars on a computer game. Initially, as the reproduction of a human and its movement becomes more lifelike we react more positively towards the object, so we’d like Lara Croft more than Leisure Suit Larry. However a point is reached, when the reproduction becomes too life-like, and the emotional response drops rapidly, meaning we feel repulsed from the object. Consider Keanu Reeves’ acting. Almost human, but utterly repulsive!
The term ‘uncanny valley’ therefore comes from plotting a graph showing our emotional response against how lifelike the reproduction is, with a sharp ‘valley’ appearing in the emotional response between a very lifelike reproduction, and the real thing.
Got that?
A similar phenomena can be seen in the production of wireframes, and hence is of critical importance to UX designers. After spending hours producing beautiful wireframes in Omnigraffle, I presented them to a client, to show how their ‘event registration’ pages would function. They came back and said “yeah it looks good, but we need to change that label text… and we need to make the dropdown arrow bolder… and can we make the heading font bigger”. This was their first view of the wireframes, to approve whether they functioned correctly, and it’s obvious what I’d done wrong.
The time and effort I’d put into making the wireframes look good, and look like a real website weren’t just wasted, they were actually hindering the process – since the mock up looked like a real webpage, the client was focusing on the small presentational details, and not the functionality itself. They expected it to look and function like the final product. If instead, I’d done a rough sketch on paper to demonstrate how the registration process should work, the client would have focused on the functionality instead. A design that looks to be in the early stages will encourage more far-reaching comments and criticism, rather than ‘fine-tuning’.
This is most important when you’re trying to focus the user experience when performing tests, especially with people not overly familiar with your site or game. Performing tests to ascertain the correct information architecture, or user’s experiences with a website’s functionality would be useless if all your comments ended up being about the site’s colour scheme. To make it clear that the designs are rough, and the presentation is not the focus, it is important not to create overly realistic designs.
Similarly care should be taken to pick an appropriate prototyping method. Paper isn’t used just because it’s quick and easy, but it also helps manage the client’s expectations. If you spent the time making the webpage on a computer, they’d be expecting it to work like a real product. On paper, this isn’t the case. Like the uncanny valley, getting too close to the real thing will be detrimental to the client’s perception of your work.
And you wouldn't want your work to look like this...
So, what steps do we need to take to ensure that the client, or user will focus on the right areas of your wireframes and designs?
- When performing initial designs, use lo-fi methods, like paper, post-stick notes, and whiteboards, where possible
- If using design software, like Pidoco or Omnigraffle, use a ‘sketches’ template, which renders your design in a pseudo-drawn method.
- Avoid drawing/designing unnecessary parts of the design – focus only on the essential
- Use filler text, and don’t work on the copy until later.
- Make it clear to the user/client that these are rough, disposable prototypes.
So, whether you’re working with a client to design their site, or conducting user testing, take care not to over-present the design, in order to manage expectations, and prevent unnecessary complaints!
