Posts Tagged ‘HCI’
Conducting an Expert Review
Within our HCI classes, we have started reviewing the UX of an upcoming multi-platform game from a prominent client, and are performing an expert review on it. An expert review, as opposed to a user-based study, involves having usability experts play the game themselves, and uses tools and their expertise to find faults. This is different to a user-based study, where the expert would observe another player playing the game. Because of the time constraints involved, we selected an expert review as the most effective method to review the UX of this game.
To get the best results possible, and be as helpful as possible to the client, we had to choose our methodology carefully. In this blog post, I’ll discuss how we chose to approach this task, why we chose these methods, and what the alternatives are.
The first rule placed on us is that we are to work in groups of 3. As described in an article by Laitinen on performing expert evaluations, the evaluation reaches its optimal group size between 3 and 4. Less experts than this may miss things. More experts than this fail to find a significantly larger number of faults.
plus too many cooks spoil the broth
The other restraint placed upon us is that we would only have a short amount of time with the game. We decided to use this time to play and evaluate the games separately, and then come together to discuss our findings. The alternatives to this would have been having one person play, and the other two take notes, or to have each person play for a bit (as we did), but the experts not playing would take notes then. All of these sessions would involve filming the game screen, and the participant.
Two experts watching one player
Advantages:
- One longer complete play through, so can see player development
- Experts can ask the player questions during their play session
Disadvantages:
- Only one play through, so difficult to see if issues are common or just for this user
- Questions asked during play through may distract/alter playing experience
Three experts playing together, in turns
Advantages:
- Three sessions played through, so can see reoccurring issues
- Experts can get a greater understanding of the game mechanics through playing it
Disadvantages:
- Players wouldn’t get as far as they would with a long session from one player
- Second and third experts play experience will be biased from the experience of the first
Three experts playing separately
Advantages:
- Each player gets an authentic ‘new player’ experience
- Comparing after can show what issues naturally arose for all
Disadvantages:
- Players wouldn’t get as far as in one long play through
- Have to perform expert evaluation after the game play, rather than during.
Since the sessions were all being recorded, we opted to do the last one, and hence have the ‘purest’ play experience recorded for each. There is, of course, no right answer – many other groups chose different approaches, and I’m sure they found equally valid issues. I’d welcome comments below if anyone has reasons for a preference with how to perform an expert evaluation.
Now having a video of a play test, we are individually analyzing them. I’m approaching it using heuristics, such as those made by Nielsen, Nokia, and the work of Federoff as a guide. Having identified the issues, I will then attempt to rate them by severity – the extent to which they will hinder the user’s enjoyment of the game. Then, in a group session with my team members, we will evaluate which issues we all agreed where particularly prominent and severe, and amalgamate our results, ending up with a list of issues with the game.
We will then have to present our data to the client. I posted before about writing a UX report, but the circumstances for this report will differ – Geographical location, and time constraints mean that this report will be an in-person presentation, with some take-aways. I will blog about these soon….!
The Humane Interface by Jef Raskin
Along with Alan Cooper’s book, when starting studying Human Computer Interaction, we were recommended to read Jef Raskin’s The Humane Interface. Having recently finished The Humane Interface, written by a designer of the original Mac (credited with the design of the one button mouse), I will briefly summarise its topics, and give my impressions.
My immediate thoughts are to compare this to Alan Cooper’s the inmates are running the asylum. This book is a harder read than Cooper’s – often going deep into highly technical topics (like how he would like to notate mouse clicks), and lacking the wit or lightness of Inmates. The most readable parts of Raskin’s books are the anecdotes about the development of the Mac and Canon Cat, and these are too few. However, this is likely due to a change in the intended audience, as Cooper’s book intends to sell usability concepts to a business audience, whereas Raskin aims his book directly at computing professionals.
Another key difference between Cooper and Raskin is they favour different methods of investigating the quality of an interface design. Whereas Cooper’s book favours qualitative data and methodology, through the establishment of persona’s and attempting to get inside user’s heads, Raskin favours quantitative methods. He includes a chapter on GOMS, a method of assigning arbitrary times for actions such as typing a keystroke, moving a mouse, thinking and moving from the mouse to the keyboard. Then by adding up the times it takes to do these actions, you can compare interaction methods by the time taken. (Its important to note that these times will not relate to the real world, as user’s act at different speeds, and can only be used to compare against other GOMS scores.)
My initial impression of this form of quantitative research is that it would highlight the speed/efficiency of an interaction, but not the quality – which is not necessarily the same thing. If a task takes a few seconds more, but is considered a lot more fulfilling, GOMS wouldn’t record this. This is particularly relevant to the field of videogames, where a purely GOMS based method to check interaction quality would lead to games such as this below:
Maybe the computer could press the button for you?
GOMS can be a useful tool to help compare interaction times, but should not be used exclusively.
Raskin also documents a number of problems with current interaction, with a particular dislike for modes (i.e. interactions that do different things in different concepts). A simple way to explain modes is the ‘caps lock’ key; turning on this mode will make ‘TEXT LIKE THIS’, despite my keystrokes being the same as when making ‘text like this’. He advocates an elimination of modes, as they introduce cognitive dissonance, and make it harder to form habits. A useful compromise, Raskin say’s is quasimodes, which is a mode that requires a constant input to achieve (and hence can be part of habit formation). This would include holding the shift key to produce capitals.
The elimination of modes extends into the elimination of applications – typing ‘SUM 7 + 6’ should produce ‘13’ everywhere, not just in a calculator. This improves the quality of interaction by allowing the user to be clear that the methods they have learnt will work anywhere. I believe this trend can be seen in current operating systems (such as the amalgamation of windows explorer and IE), and this is one of Google’s main aims with their OS.
Raskin also advocates an unlimited undo feature (even through closing and re-opening documents), and the elimination of dialog boxes asking ‘are you sure?’ These two are linked, giving that level of undo freedom would make ‘are you sure’ unnecessary, and is more technically feasible now than when the book was written. I assume it’s a matter of conventions, and momentum which would hinder people advocating these new interaction methods, and it is this mindset Raskin is trying to overturn.
An even more radical suggestion is Raskin’s radical redesign to information architecture. Looking at the hierarchical, folder methodology we have of storing files currently, Raskin notes that it is inefficient – from any point, you cannot see what’s in the folders below, or in the level above you. Since the book was published in 2000 we can see efforts have been made to combat these criticisms – in Windows, folder icons now show the file types inside (and previews if they are pictures), and have made it easier to go up a level. On Macs, they have additional folder view types that make it possible to see ‘up’ the hierarchy.
Raskin however has a more radical suggestion, which he calls ‘Zoom World’. Imagine, flying over a world with a series of zones, ‘I.e. pictures, home, work’. Then you zoom in on pictures (while still being able to see the others), and note that closer up we can see the pictures has it own sub zones, entitled ‘pictures of France’, ‘pictures of the dog’, ‘pictures of lily’ etc. Zooming in on ‘pictures of the dog’, now we are close we can see some individual pictures, one of the dog smiling, one of it playing with a ball
One of the dog playing counterstrike
Zooming in further on this picture would let us read and alter it, but we always have the option to quickly and freely zoom out and see any area of ZoomWorld. The advantage of this system is it solves the issues with being able to see the files above and below at any point, and not be restricted to your current folder. It has been implemented in ‘Archy’, which includes many features Raskin advocates in this book.
Ultimately its interesting seeing how many of the ideas Raskin advocates are ahead of their time, and were included in later revisions of Macs, and in general interaction. (such as searching starting from the first character you type, rather than waiting for you to press ‘enter’). As a book though, it’s harder to get through than Inmates, and does go on in exceptional depth about less than inspiring topics. Raskin talks endlessly about the Canon Cat, a system from the eighties with which he had tried to implement many of his interface ideas. He notes however that it met resistance from users who were used to the existing human computer interaction paradigm, and was not commercially successful. Perhaps, with the moves made by the leading Operating Systems, and Google OS breaking down the barriers between an OS and a browser, people would now be more susceptible to higher quality interaction with computers, and are prepared to unlearn their bad habits.
7 aspects of successful usability questionnaires
This week in HCI we’ve been thinking about questionnaires. They can be an important usability tool, although there are also many limitations. Primarily questionnaires are used as a quantitative data collection method (i.e. it will give back a large amount of responses), and so, compared to a qualitative methodology, are useful in pinpointing where problems exist, but less helpful in helping us understand why. As such, it is best to combine both forms of research, perhaps by starting off with questionnaires to identify frequent problem areas, and generalized opinions of systems, before moving into a qualitative method to understand why these areas are problems. An advantage of questionnaires include the fact that they are cheaper and quicker to get results from than many other methods, but this is balanced by some drawbacks – the data you record is more subjectively influenced by the researcher and participants opinions than in other methods, such as direct observation.
Nonetheless it is an important usability tool, and it is important that the responses received from questionnaires are of high quality, and useful. So, I’m going to share some of the areas that I, and other HCCS students, have identified as potential problems when dealing with questionnaires, in order to help you make better questionnaires. And since this is the internet, we’ll be presenting them in the form of a list, as everyone on the internet loves lists!
everyone on the internet also loves pictures of cats
So, here are seven important aspects to consider when creating questionnaires.
1. Answers can only be as good as your questions
When preparing a questionnaire, you need to think at length about the aspect of the subject you want to investigate, and go in knowing what you need to find out. Generalized questions, or being vague on the topic, won’t give useful data, and so it’s important to make sure the questions are actually asking relevant things. For example if you wanted to find out about… the most popular aisles in Sainsbury’s, asking questions about whether people prefer the supermarket to its rivals wouldn’t get closer to this goal. Also we all know it’s the cereal aisle. So, know what you want to find out from the questionnaire.
2. The questions need to cover the areas in depth.
When getting opinions, it helps to be specific. Don’t just ask ‘did you like this’, but follow it up with either a question asking for reasons why, or (if you’re after a data set that can be analyzed more uniformly), ask them to rate on a number of scales why they did or didn’t like it (i.e. “to what extent did the look of the webpage affect your opinion of it”). Not doing this will lead to closed answers (Did you like this? “no”), when it would be possible to get a much richer set of data from the participant. Whether you select an open question ‘why’ or a closed question (based on scales), depends on whether you are after purely quantitative data, or also want to include qualitative data as well.
3. Changing the questions mid-implementation taints your qualitative data
Halfway through a study, the results may start to show interesting trends that you’d want to find more about. Take caution when altering the questionnaire to investigate these trends. Adding more questions should be fine (except for the tired participants!), but when editing a question that already exists (i.e. from ‘did you like the look and feel of the website’ to ‘did you like the look and feel of the first page of the website’), keep in mind that this will invalidate getting a quantitative response (i.e. ‘85% of people liked the look and feel of the first page of the website’) from the entire dataset for that question, as the participants have been answering different questions.
4. Subjective answers need to be standardized
Remember, when asking whether something was ‘easy’ or ‘hard’, that answers to theses questions are going to be subjective. People are likely to have a wide range of expectations about how a system should be, and a wide range of experience, and so will be judging on separate scales.
Dr Graham McAllister tells a story related to this. When doing usability testing, he asked ‘did anyone have any problems with the program’… no reply. So he asked instead ‘did anyone think that someone else may have problems with this program’, and a whole host of replies were given from the same people.
Don’t forget that pride can be a factor preventing people from saying they found task’s hard. Shifting the focus of the questions from the participant to the medium can help prevent this.
Also, terms such as ‘often’ or ‘rarely’ mean different things to different people. Try and replace them with specific terms ‘every day’, ‘every week’ etc.
5. The questions reflect your opinion
Because of the close controlled environment that a questionnaire creates (i.e. the participants can only answer the questions they have been asked) it is important to make sure that the researchers opinions do not show through the questions. For example, leading questions, which make it easier to answer one way than the other. I saw an advert recently, for some sort of Christian business, that asked ‘Does god exist?’ with tick boxes for ‘Yes’ ‘Probably’ and ‘No’. This is a leading question – the only indefinite reply implies agreement. Where is ‘probably not’, ‘neither agree or disagree’ or ‘don’t know’? (Answer: not on an advert paid for by the church)
6. You need to give people a reason to participate
now thats an incentive
Before I go on with this list, I was wondering if you’d be happy to answer 25 questions on your opinions of southern English fauna and shrubbery. Please click here to fill it out.
Did I mention that filling out the survey gets you a £25 amazon voucher? Do you want that link again?
The point, as I’m sure you guessed, was that you need to offer an incentive for people to participate in your questionnaire, otherwise only people really interested in the subject will reply. Suitable incentives would be discounts, free products, a prize draw, or something related to the field you are investigating.
7. The data can be skewed towards extreme opinions
Failing to give a good enough incentive or no incentive at all, will end up with unrepresentative data – only people who feel so strongly about the subject matter to reply will bother to. In practise this will either be people who are really angry about it, or people who love it, and this will skew your data towards the extremes. To ensure you get a natural selection of participants, steps need to be taken, such as pre-selecting participants, or offering incentives as covered above.
So there we have it. Seven tips to help you make effective questionnaires. Enjoy asking people things!
Understanding Cognition, User Experience Winners & Losers, and a Design Failure.
Understanding Cognition
In order to design a positive user experience, it is important to understand how humans work from a cognitive perspective. There are three aspects of this attention, perception and memory. By understanding what humans can and can’t do, we can create models of behaviour and guidelines that will aid the user experience design, and avoid asking tasks of users that they will find difficult or impossible (like remembering a long, arbitrary sequence of numbers).
are you sure this is how i save a file?
Attention:
Primarily, we need to understand how users are directed through a task, and the tools we can use to focus or divert attention. This is commonly seen on websites as a ‘call to action’. For example, Google’s sparse design draws immediate attention to its ‘search’ functionality, as this is what Google want user’s to do.
Colour, ordering, spacing and animation can all be used to help attract or divert attention, as well as avoiding over cluttering. Humans are particularly drawn to moving items (like the T-Rex in Jurassic Park), and so this is a tool we can use to attract attention
click where to claim my free prize?
Eye scans showing that people read websites in an ‘F pattern’. Like the shape of a capital F, people will scan along the first line, and then down the page, highlighting mainly on the first word of each sentence, and only occasionally reading a sentence all the way across. This means that people will skim past alot of important sentences, like this one where reading to the end could win you money. It won’t. But you should have read it anyway.
Perception:
Perception seems a bit obvious – it can be visual, auditory or tactile (i.e. force feedback), and changes based on context. There are a few things we need to take into account based on the limits of our perception – text should be legible, icons should be easy to interpret, that sort of thing. Studies done into advertising and copywriting show us simple rules we can follow to make websites and programs more readable – for example it’s easier to read high contrast items, and black on white is read quicker than white on black. Furthermore white space is important in readability.
Memory:
People find recognition easier than recall. To aid this, we want to make things visually consistent, so that people will recognise features they’ve used before, rather than having to remember them. In the before-times, we use to have to remember commands for a terminal (i.e. erase all) and type them in to execute the command. Now GUI’s allow us to present these options on a screen, and have a user select them. Images will aid this, as people remember images easier than they remember words. The user will recognise an icon they’ve used before, and select it to perform the same task.
GIS tells us this is a memorable icon
So, who’s done this right? We’ve been asked to identify 3 products we like using, and 3 we don’t. I’ll also be seeing how they fit in with the cognitive processes talked about above
3 User Experience Winners
- Tweetie.app – Twitter client Tweetie is a success because of it use of context dependant features, and its seamless integration of key features. For example, the ‘new tweet’ button will either start a completely new tweet, or an ‘@’ reply, or a Direct Message, based on where I am when I select it. By assuming the feature the user is most likely to need, it prevents the need for users to remember how to access these features individually.
- Trainline.app – From the first screen of the trainline app for the iPhone you can see the clear ‘call to actions’ that the app has been built around. The app remembers my home station, and can work out which station I’m nearest, and hence can provide me with a single ‘next journey home’ button, which will instantly show me the train to get home. Not only this, but it remembers my frequent journeys, and presents these as buttons as well, making it clear how to get to the options I’m after. By remembering what I use it for, and treating my data as important, the trainline app is a success.
- The vending machine at Sussex Innovations Center – Wow, this thing is amazing – I’ll make a video for one week! When making your selection, a robot arm comes and picks up your selection, and carries it to the drop bin area. (no just throwing your drink/snack to the floor like other vending machines!). Then, the drop bin opens up automatically and a little light comes on (a good way of attracting attention!). Overall, a positive user experience, it makes me feel like it cares about my selection, and highlights where my attention needs to be at any time.
3 User Experience Losers.
- Texting with the iPhone – Only minor gripes here, both of which my girlfriend (a new iPhone user) has fallen foul of in the last week. First the placement of the send button – right above the keyboard, and less than a centimetre above ‘o’ and ‘p’ – hence very easy to press halfway through writing a text. Second, the placement of the ‘cancel’ button, right above where recipients are added. Sending a group text (for example saying ‘I have a new iPhone, heres my number) to multiple recipients requires clicking multiple times in the top right, again less than a centimetre from the cancel button. And the cancel button doesn’t ask for confirmation (from a blank text), just loses your list of recipients. Denied.
- Ticketmaster – I’ve mentioned in brief my grief with Ticketmaster before, and will briefly touch on it again here (I still intend to go into depth on this topic soon!) The problem with Ticketmaster is it hides its information. If I want to go to a gig, I have to select which date, where I’d like to sit, how much I’d like to pay, before it tells me that they have nothing that matches my selection. So I have to start again from the beginning of the process. Repeatedly. And then eventually find that all the dates, and all the seats, have sold out. Ticketmaster’s system surely knows that the event is sold out, and yet the website keeps this secret, instead you have to try every possible option and deduce for yourself that tickets are sold out. And if the option you pick is sold out, it forgets everything about what date or what price I want to see, and makes you start the process from the beginning. Surely it could give you alternatives based on what is available? From a cognitive perspective, the system expects me to remember what information I’ve entered before, and avoid entering the same details again.
- My pen – My pen is magic, and has a rubber on the end hat can erase ink. But is also very cheap. And so the rubber is barely held on simply resting in a gap on the top of the pen. When used to rub things out, the rubber breaks. Without the rubber attached, it’s impossible to extend/retract the pen, without poking it with something else (like another pen). This isn’t really a cognitive failure – just bad design (the device cant be used for the function its been designed for)
Similarities & Differences
So what did the ones that were good do?
- Assumed the tasks the user wanted to do, and made it easy to do them
- Stopped the user from making mistakes, either by hiding dangerous options or through making options undoable.
- Remembered me, and my preferences.
In contrast, the items that I considered had a negative user experience
- Allowed me to lose my data, and make permanent mistakes
- Doesn’t give you all the information to reach your goal, and hides information that is in the system, which would be useful to me.
- Requires repetition of tasks, and forgets things you’ve already told it, asking me to put in information again.
Design failure – Water Machine
Just a small design failure this week, but a mystery to me (would welcome comments with solutions). This is the water machine at work:
water machine
We can see there are 5 buttons, all with icons. It’s these icons I’m having trouble with.
The big button has a picture of some drops of water. I’m assuming, due to its prominence, and the fact that I tried it, that this button makes water come out. Great
Now for the four smaller buttons.
Two, very confusingly, have the same picture on (a glass with some ice cubes in). I can’t tell whether these buttons do the same thing, or different things. The same icon would suggest the same function, but maybe not.
The icons on the other side – a steaming mug, and a steaming mug with a plus on, I assume do hot water. And… hotter water? Does that mean that the two ice cube buttons do cold water, and colder water? But then why no plus on one of those buttons? And which one would be colder?
It’s a mystery to me!
Evaluating existing technologies, paper prototypes in action, Windows 7 and the disappointing user experience of my DVD player!
This week our HCI course featured an introduction to starting the design process by evaluating existing technologies, and the key advantages of this were made apparent when we started to build paper prototypes.
In an abstract sense, the advantages of evaluating existing technology is that aids redesign – you can see what elements turned out successful, and which did not. What this means in practise is that you can steal the best bits of the competition, and fix the bits that users complain about (and hence find raw user experience data on the internet). This is particularly pertinent with this week’s release of Windows 7.
Aside from multitouch (surely useless for most home users), two of the main ‘new’ features of Windows 7 are likely to be the result of evaluating existing technologies – mainly OSX.
The task bar now has persistent icons, so they don’t go away when you close the program. Programs that are closed have a slightly different visual effect applied to them on the task bar. OSX users will of course remember this from their own Dock.
Windows 7 Taskbar
Or is it this one?
Another borrowed feature is the new way show desktop works on Windows Seven. Hover over the bottom right hand corner of the screen, and it will look like this:
show desktop
As OSX users will know, this is the same as leopards’ exposé feature, which allows users to show all open windows, all application windows, or (as in windows 7) the desktop, by moving the mouse into the corner of the screen. When I used to work on Mac’s, I thought this was a great feature, and its no surprise why Microsoft borrowed it.
Both of these are examples of taking ideas that were either popular or productive, from a rival operating system, and integrating them into your own design.
Also this week is the launch of the first Microsoft Store. Again, they evaluated the existing user experience of an Apple store (sleek design, answers bar, the layout) and… nicked it. Great success!
This isn’t just Microsoft bashing by the way, its just relevant examples of evaluating existing ideas, and taking the good bits. Everyone knows that Apple nicked all its ideas from Xerox anyhow!
So, in the spirit of ‘evaluating existing ideas’, we have been involved in the design of a new system for the University. User Experience will be important here, for the feature we are designing is likely to be used by students in high stress situations. We had half an hour to paper prototype some ideas on how to build the system (which it quickly become apparent was not enough time). Then we swapped, and compared our own designs to those produced by other groups. My personal favourite design (by Hassan and James ) had emulated key features from eBay, particularly how they categorised entries by the status (i.e. on eBay, ‘items being bid for’, ‘items I’m watching’, ‘items sold’ appear on separate lists).
This practical exercise brought home the advantages of paper prototyping, and evaluating other designs. Paper prototyping had proved to be:
- Fast
- Useful for brainstorming ideas
- Able to be changed quickly
- Adequate at demonstrating the key features of a website
and evaluating existing designs had not only helped our group find a better solution than the one we’d implemented, but also had been a key part of James and Hassan’s design process.
My ‘design gripe’ this week was my DVD player. It has many problems, a few of which I’ll share here.
- If there is no DVD in the player, it displays ‘NULL’
- The ‘time played’ counter (the default on the front of the player while playing a DVD) only counts to an hour. Then it goes back to 0 minutes again.
- The sound volume of a DVD is roughly half that of the normal audio output of the TV (and this is with the player’s DVD volume settings set to max)
- If multiple camera angles are available on the DVD, the icon for it (a little film camera) appears on the TV screen always. Its quite distracting when watching a film, and cannot be turned off. Same for if you use the ‘zoom’ functionality.
From this, we can deduce the following points:
- User experience wasn’t a priority, as ‘Null’ would mean nothing to a non-geek audience
- The player is probably intentionally badly designed, to drive the user to a higher priced one (I assume the company make more expensive ones). This may be counter productive though, as I’d be unlikely to go for the same brand.
- You shouldn’t buy the second cheapest DVD player Argos sell. (I’d hate to see the functionality of the cheapest one)
my next dvd player
I’ve started reading Alan Cooper’s The Inmates are Running the Asylum, which diagnoses the problems with technology currently as a lack of understanding by the business of the need to separate programmers and designers. Programmers are great at thinking logically, and making tools they themselves could use effectively, however they need to be given clear goals and design models by people who are better placed to understand user needs, and that is where our role as designers come in. I have to say, it’s a lot more readable than the HCI textbook, and Alan Cooper has a great degree of insight into the subject. He recounts the following (rather old) joke, which is quite relevant to our field of user experience.
He shouts down “I’m lost, where am I?”.
The man replies “You are in an airplane, 100 feet above the ground”.
After hearing that, the pilot immediately flies off, and lands successfully without a problem.
“How did you do it?” he was later asked.
The pilot replies “Well, with an answer like that, I knew I must be at the Microsoft building, and I know my way back from there”.
As we can see, the answer was true, but not helpful. This is what we want to avoid as user experience architects.
I was going to feature here the terrible user experience of Ticketmaster’s website, but I think it’s so bad it could fill an entire blog post on its own. Expect this soon (especially since I’m bitter I didn’t get my Paul McCartney tickets)!
