Don't judge it by it's cover...

Why would you consider remote research?

Written by advocates of remote research, the book highlights many of the potential advantages that remote research gives compared to a more traditional lab based study. These advantages are fleshed out throughout the book through testimonies of experts who have experience in this field, who offer real world examples to emphasise these points.

Some key advantages are:

• Access to a geographically diverse user base. Unlike traditional research, where a moderator would have to be in the same physical location as the subjects, remote research allows a study to be run with anyone who has a high speed internet connection, widely expanding the potential study-group.
• Easy to let stakeholders get involved. Because the research session is being broadcast over the internet, it’s possible to allow stakeholders (i.e. executives and designers) to view the session, and give (moderated) input. This of course increases their engagement with the process, and will be the ‘evidence’ for any conclusions derived from the research.
• Natural browsing environment. The validity of the research can be improved, not only because you are allowing the user to perform the task in a familiar environment (their own home computer), but also some recruitment methods allow you to capture a user performing a task they have selected. For example, recruiting a user who came to the site to buy trousers, for a task based on buying trousers, would provide more accurate results than asking someone to pretend to buy trousers…
• Cheaper (debatably). Not having to pay for travel can keep costs down, however other costs, such as incentives, will still be required, as well as paying for the software.

The remote research book doesn’t advocate killing off lab tests though – instead, it recognises that there are cases when the lab is still appropriate, such as when privacy is a concern. The book also features Andy Budd’s defence of the lab, which argues that remote research fails to pick up aspects of non-verbal behaviour, as well as arguing that remote research doesn’t just remove a selection bias (geography), since it also adds another (internet speed and technical ability). It’s brave of the book to include the case against remote research, and helps project a more trustworthy and reliable image for the book itself.

How to do remote research

The ‘meat’ of the book are the sections dedicated to how-to guides on the different forms of remote research. The book contains step by step instructions on performing moderated or un-moderated research, and includes key topics such as recruitment (and live recruiting), card sorts, and lots of handy hints – such as using IM clients as a chat room for multiple observers to automatically share and timestamp notes.

The book doesn’t just cover basic topics – it goes on to develop novel approaches to user research, such as using ‘reverse screen sharing’ to protect confidential software or data, and using mobile web to gain a new understanding of time-dependant information, outside of the traditional moderated setting.

It also extends the remits of remote research – it doesn’t have to just be websites, but can include doodles or sketches, as well as developing ideas for automatic research with analytics.

Another sort of remote research?

Conclusion

Remote Research is one of the easiest to read UX books I’ve reviewed. Like many Rosenfeld publications, it is laid out well, without appearing dense with text, and has a friendly tone throughout. The book can be likened to Krug’s writing in its style, and presentation.

The book is also practical and realistic, and deals with real world issues, like ‘fakers’ (who can be outed by using open ended questions to discover motives), legal issues, and common challenges such as reluctant stakeholders.

Most importantly for the practical UX practitioner, the book is not dogmatic. This is especially evident in the last chapter which admits that usability shouldn’t be the exclusive goal of product design, and needs to be coupled with initiative, and innovation to develop great things.

Overall this book is a great introduction, and how-to guide to the growing field of remote research, and will be an important tool for anyone trying to keep up to date with the latest research methods.

Related posts:

1. Rocket Surgery Made Easy by Steve Krug – Book Review
2. Evaluating User Experience in Games – Book Review
3. Selling Usability by John Rhodes Book Review

We’ll be looking at what biometric research is, how it can be applied to games research, and the problems that became apparent with this method.

What is biometric research?

Biometrics are traditionally an automated way of recognising, or recording, peoples physiological data, or characteristics. To apply this to video-games readings were taken by Vertical Slice by hooking players up  to machines which record their heart rate, brainwaves, or galvanic skin response (…how sweaty their skin is, presumably).  It’s proposed that there is some correlation between how their bodies react, and how the player is feeling – such as how a player’s heart will beat faster while fighting Gunther Hermann’s Skull Gun or scoring a tetris.

Pejman Mirza-Babaei has been investigating how this can be applied to games research. Working with Vertical Slice, he is interested in measuring the player experience – how to know when players are having fun, or becoming frustrated, and so has been performing studies to see the feasibility of measuring this with biometric data. By having players play either Haze, or Modern Warfare 2, while hooked up to this machine, maybe it’s possible to gain a greater insight into the player’s thoughts, and how they feel when playing.

And in a non obtrusive way...

What did biometric research show?

When playing the games, the player’s heart rate and GSR ratings (that sweatiness rating) were recorded along with a video of the player, and of their screen.  What was found from the biometric readings, at the simplest level, was when the player’s heart rate went up. The researcher would then conduct an interview after the gaming session, and ask why the heart rate went up at those points, for the player to justify the measurements.

We saw examples of these spikes when the player enjoyed, or was frustrated by, a task (such as using a machine gun, or getting stuck looking for a vehicle), and were given the player’s justifications for feeling like this.

The most obvious advantage of this method is that it allows a more natural environment to be created for the player. Since biometrics doesn’t require distracting the player by asking them to perform a  think-aloud, or by interrupting their game by asking questions and yet still having a degree of insight into how they are feeling, a more natural game play experience can be achieved, without stopping useful data from being gathered.

Problems with biometric research on games

However, some limitations on the application of this technology became obvious through the presentation. Biometric data (in its current form) doesn’t give any insight into why the player’s heart rate has spiked, just that it has. This problem is exacerbated by the single range of readings it can give – there is no way to distinguish between stress and happiness (or any other reasons a heart rate can spike).

such as exciting vehicles

Because of this, biometric readings have to be justified by another method, to give some understanding as to why the heart rate spikes at certain moments. Traditional UX methods, such as a post-test interview, are therefore needed in addition to biometric readings. However this reintroduces traditional UX problems. A player may not be able to accurately remember why they felt excited at a certain moment, and as pointed out by Sam Nixon, may simply justify their opinion by what they see on screen.

For example, the player may explain a heart rate spike caused by audio cues as being caused by the enemy visible on screen when the clip is shown later, missing the real reason for their excitement.

Conclusion

So biometric readings alone cannot tell us what a player is thinking. Hence they cannot (currently) be a replacement for traditional UX methods.

What biometric readings can do, is aid the application of current UX methodologies. When combined with tools such as think aloud, or interviews, they can add weight to the findings. For a think aloud, it can tell you which parts of the game particularly affected the player, and hence what comments to pay attention too. Similarly with interviews, biometric research can pinpoint the areas that the player should be asked about. When used in combination with typical UX tools, biometric research can be justified and have some understanding applied to its findings.

There is amazing potential in the application of biometric data to games. Currently, the ‘AI director’ in Left For Dead controls the game based on how the player is doing – giving less zombies to fight if the player is doing poorly, or making the game harder, and giving the player some nasty surprises, if they are doing well. Imagine if a system like this could take biometric data into account, and change the game experience based on how the player was feeling. Vertical Slice have begun to show us the potential of this technology, and I feel we’re at the start of an exciting journey.

Related posts:

1. Why you should always video players when testing games
2. Evaluating User Experience in Games – Book Review
3. Quantifying the unquantifiable – Expert Evaluations

The UXBrighton event was presented in a new format as a series of short talks, from Harry Brignull’s tips on time stamping notes, to Danny Hope’s templates for understanding user roles. Also interesting was a talk on using google analytics, although the length of the talk meant that topic could only be skimmed, dissapointing as I’m an analytics fan. The most interesting idea presented was iCrossing’s presentation on “The iCrossing Connected Brand index: how to measure a brand’s effectiveness online”, given by Ifraz Mughal.

Expert Evaluation

As I’ve mentioned before an expert evaluation is a useful tool for getting an insight into potential usability and user experience issues on a website, or game, with limited resources. Although it can never replace running tests with real users, it can provide a quick approximation, and help highlight the biggest issues.

The ‘method’ for an expert evaluation is simple. Get an expert to look at the site, or game, and tell the client what they think. Job done.

My expert eye tells me you need smarter users...

However an expert evaluation can only ever be subjective, and this is it’s biggest weakness. A client can look at your page full of recommendations, and dismiss it as the opinion of one person. There’s no easy way to see progress with changes, and a comparison with other sites can only ever be abstract.

Quantifying an Expert Evaluation

iCrossing’s solution is to quantify their expert evaluation. As part of their ‘Connected Brand Index’ idea, they rate their clients sites (and competitors), on UX-centric areas such as “usefulness”, “usability” and “desirability”.

A traditional expert evaluation would give a qualitative rating, and give examples to back this up, i.e. “Poor – little emphasis, and diffused call to actions”. Instead iCrossing will give the site a score, on a scale of -2 to 2 (2 being very good). This of course can be backed up with examples in a more in depth report.

after the first few pages, the report can just be pictures of kittens. No-one reads that far.

The advantages:

There are numerous reasons why a client would prefer a scored ‘rating’, rather than comments.

• A ‘score’ makes it easy to benchmark, and compare your own scores against competitors. By dividing the expert evaluation into separate topics, and scoring each, a finely grained comparison can be made, and communicated
• Similarly, a score makes it easy for a client to see progress. If they scored -1 before hiring you, and 1 after, your work can be justified (as long as no-one questions who is doing the scoring!)
• Because this produces a concrete score, clients will be able to handle and communicate the data. Graphs can be made, which wouldn’t be possible for subjective comments. These can be invaluable for justifying and communicating with managers and project sponsors, who do not need to see the details, just get a high-level overview.
• This expert evaluation can be encompassed as one aspect of a larger ‘score’ given to websites, or games. This is the idea behind iCrossing’s connected brands index.

Conclusions:

There is an argument this can be seen as a bit of a scam. Giving arbitrary numbers to your opinions doesn’t make them any less subjective. This method of presenting the data could be misleading if presented incorrectly, and the client should be made aware of the method behind the score system. This could become an issue when running comparative studies before and after your work, since you’d be biased towards giving the site a better score after you’ve worked on it.

The point of this method is to aid communication with the client, and give them data in a format that is useful to them. As I discussed in the review of Selling Usability, management and non-technical people would typically much rather see pretty graphs, and statistics, than a list of comments. This method helps manage client expectations, and gives them what they want.

To make the method more valid, it would be useful to perform a study to ensure the method is sound. Perhaps get a wide range of experts to independently rate a wide range of websites on this scale, and note the correlations between the scores. It’d be first step in countering complaints that this method is still inherently subjective, and help make an art into a science.

Related posts:

1. Conducting an Expert Review
2. How to present a UX Report
3. Understanding players through biometrics

I’m in the process of designing a GPS game, and am currently discovering the functional requirements for the project. As UX practitioners, we know that involving the user is of critical importance at this stage, hence we designed a questionnaire to establish peoples experience, and perception of GPS games, and what they’d like a GPS game to be like. (linked here)

When at university, our internet access goes through a proxy server, which blocks unsuitable content. For some reason, this includes surveymonkey.com, a site I’ve used a few times in the past to construct online surveys. Interestingly, the ‘site blocked’ dialogue said “for survey sites, try zoomerang.com”. However, when I search for some hardcore action, it never gives me alternate suggestions for that. Have I uncovered a conspiracy? Nonetheless, I followed the link.

And so I ended up on zoomerang.com. Being fair, there is one key advantage to zoomerang which immediately put me in a good mood. On surveymonkey, for a free account, you are limited to ten questions. On Zoomerang, you can ask 30 questions before you have to pay. This meant we didn’t have to redo, or concatenate our questions, and made me smile inside

Pictured: an inside smile

Problems with Zoomerang.com

This goodwill was shortlived, when I tried to use the site to implement my questionnaire. Heres why:

1. The workflow isn’t clear when making a survey, and so I entirely missed the step where you add your questions. Clicking through the process actually caused me publish a blank questionnaire. Which wouldn’t be a problem, except…
2. …You can’t edit an existing survey. Once its published, you cannot add/remove/change questions. Surveymonkey allows this. So I was stuck with my blank survey, and had to start again from scratch.
3. Having figured out how to add questions, I got started, and selected “insert question”. It added a header, which then had to be changed to type question. I guessed that was because it was my first item, but no, it always defaults to inserting a header (odd, since you’d only need 1 per page, whereas you’d need multiple questions).
4. So I finally got to add a question, and this is when the terribleness of the design struck me. I selected a question where a radio button would select from a number of answers, and typed in my list of 15 or so alternative answers into a rich text field. I hit submit, and … got an error, saying “answers can only be 1000 characters, including HTML”, and even worse…
5. …It deleted the data I had entered in that field. All 15 answers. This is a critical failure of any system, since the data a user inputs should be considered sacred.
6. There was no counter telling me how many characters I had entered, so I had to retry a few times. Eventually I realised that I could only enter 5 potential one word answers before it’d error that I was over 1000 characters. That had to be a mistake? I investigated further…
7. …Looking at the HTML, it turned out that the rich text editor was writing rubbish html. At the start of each answer, it’d add needless style tags, often multiple times. Heres an example of the HTML it generated for my one word answer “complicated”
   
   <p><span style=”font-family: Arial; color: #000000; font-size: small;”><span style=”font-family: Arial; color: #000000; font-size: small;”><span style=”font-family: Arial; color: #000000; font-size: small;”>Complicated</span></span></span></p>
8. …no wonder it was hitting the character limit after 4 or 5 words. I had to manually enter the html for all the possible answers, just so I could get round this.
9. My last fault with zoomerang.com is just a suspicion. I look after my email accounts, and so have never received spam in my current primary address. After signing up for zoomerang last week, I received my first random spam email. Might just be a coincidence, but I didn’t sign up for anything else that week!

I had to call this guy to fix my survey

How to fix zoomerang.com

To improve their user experience, they should look at red-routing the goals the user needs to achieve:

1. Make the progression through survey design clearer, highlighting which step questions are added in
2. Also make it clear how far through the design process you are, and what steps remain
3. Restrict what the user can do, so they cannot post a blank questionnaire. Its obvious if they are about to do this that they’ve made a mistake, tell them!
4. Don’t make question types default to “header”. Surely users will only use this type once at most, whereas they’re going to have more than one question on the questionnaire. Make it default to that!
5. Fix the WYSIWYG code generator, so that the user doesn’t have to manually code the answers in HTML. A lot of user’s would get stuck at this point!
6. Don’t send me spam!

And what can you do, until these fixes are made? Use surveymonkey.com. Or, if you’ve found anything better, leave a comment and let me know!

Related posts:

1. The user experience of Ticketmaster ruins Christmas
2. 7 aspects of successful usability questionnaires
3. The Problems with Surveys for User Experience Tests

What is the Likert scale?

Lets start with an example.

Most people have seen a Likert scale before. Do you agree with this statement?

• Strongly agree
• Agree
• Neither agree or disagree
• Disagree
• Strongly disagree

And the responses should be balanced... unless you have an agenda

Often used to gauge opinions, they are especially important for people involved with measuring usability or player experience, as they can help quantify subjective things like a user’s experiences. They are usually in the form of a statement, followed by a selection of statements, to indicate how far someone agrees with the statement. They can often be used to quantify things like ease-of-use, or fun, which would be impossible to quantify through other methods. Hence they are of particularly important for us, since user experience is essentially abstract.

Different kinds of Likert scales.

The essential question when it comes to implementing a Likert scale, is how many responses to offer.

‘Forced Choice’ scales are those which have an even number of options. Essentially this means missing out the ‘neither agree or disagree’ option, and forcing the participant to make a selection (see what they did with the name? very clever!). This would be done to force participants to show an opinion, but there are dangers inherent with this. Forcing a response may give a larger degree of ‘static’ in the responses, reducing their accuracy, since the responses may not map their opinions. People who don’t agree or disagree may not be happy about being forced to give an opinion, reducing their likelihood to answer later questions accurately. However if your aim is to support a conclusion that people do/do-not like a system, you may be willing to risk these to prove your point when designing the survey.

Forced choice means its hard to tell who is neutral, and who doesn’t want to participate

If you select to use a scale with an odd number of options, there are a few issues that should be kept in mind when deciding between a five or seven point scale. The most obvious difference is that a finer grain of responses can be analysed from a seven point scale, as it can represent a wider range of views. Also, take into account that it’s been shown participants shy away from the ‘edges’, the extreme like and dislike options offered. This means a five point scale will likely only get responses in the ‘slightly’ columns from all except the most ardent fanatics. Again, you have to consider whether a wider range of responses is useful to the topic you are exploring.

Should you use a Likert Scale

Ultimately if you are trying to track opinions, a Likert scale is a good method of accessing this data. There is no all-encompassing correct answer for which scale is appropriate, the context of use and what you want to find out will all affect this. As long as you keep in mind that not only the phrasing of the question, but the range and number of responses you offer will affect the results, and anticipate this affect, you can’t really go wrong. Happy surveying!

Related posts:

1. 7 aspects of successful usability questionnaires
2. The Problems with Surveys for User Experience Tests
3. Five essential iPhone apps for usability professionals

Straight back onto UX issues -
For my HCI Coursework, I’m designing a new product, with a focus on evaluating the user requirements -
To do this, I’ve made a survey (keeping in mind the issues I raised on making a UX Survey), to help me understand user’s needs.

It’d make my decade if you’d fill out the survey, it only takes 2-3 minutes and doesnt require any personal details

Click here to fill it out.

Feel free to comment here aswell, if you find any issues with the survey itself.

I have lots of ideas for UX blog posts for 2010, so expect alot more to come soon…

Steve

Related posts:

1. Who are you?

Working with usability requires a wide range of abilities, and hence these tools reflect the wide range of roles a usability professional may be performing. When talking about each app, I’ll try to cover where in the usability process it may come in useful, and how it can help you to increase efficiency and get better results.

but don't forget lo-fi alternatives

1. Pidoco°

Starting off, this isn’t strictly an iPhone app, but Pidoco’s software has been developed to be compatible with mobile browsers and the iPhone. I saw them demo the software at a recent UX Brighton event, and they impressed upon me that it was built with usability in mind. Pidoco offers a quick way to collaborate on the design of wire frames, and quickly show these to the user or client, without the need to be in the same geographic location as them. The easy to use nature of the system means it’d be ideal when iteratively testing an interface with users, and is a mid-fi alternative to paper prototyping. Currently offering a month’s free trial, there’s no excuse not to give it a go!

Pidoco°’s website: https://pidoco.com/en/

2. Mocha VNC

Need to mock up a mid-fi prototype of an iPhone App? You obviously don’t have time to make the app in Xcode, buy the iPhone developers license, link the customer’s phone to the computer and copy the app over. Instead, how about making a prototype on your Mac, or PC and using a VNC client to make the iPhone look at the prototype. The user will still be able to interact with the prototype by clicking, or navigating around, and the ‘controller’ sat at the server can display the correct pages to the user. Easy mid-fi prototyping, on an iPhone, without the long set up costs and investments required to actually have the app running on the iPhone.

Mocha VNC’s website: http://www.mochasoft.dk/iphone_vnc.htm
 

3. Camera

Watching someone perform a task is an important part qualitative usability testing. What about those moments where you weren’t looking, or noticed something that you wanted to ask about later, but forgot about? Looks like you need to be recording your user tests. Camera comes with all iPhones, but is only available for video recording on the 3GS. Since you are always likely to have your phone with you, it’s a handy mobile solution that allows you to record your usability sessions without requiring any additional equipment, so that they can be analysed later. Maybe buy a stand for it though!

4. exZact Data Collection

Quantitative research is also an important part of establishing user requirements, and working towards a design that meets their goals. The Data Collection iPhone app aims to allow you to create, and answer, custom surveys on the iPhone, collecting information like geographic location, and responses in a variety of forms (1-10 ratings, drop down lists, yes/no responses, and comment boxes). They then offer real time analysis, so you can see the results as they come in, and claim to be massively scalable. Offering a month’s free trial, it seems an effective solution if you require your data collection method to be mobile, and accessible anywhere.

exZact’s website: http://www.iformbuilder.com

5. Insight – Basecamp on iPhone

You’ve seen from the diversity of the apps above that a usability specialist has to be experienced in a lot of areas, and its often hard to keep on top of these. A tool that I have found invaluable in project management is Basecamp, which allows you to collaborate on projects, assign tasks and see what everyone is up to. Insight is a mobile version of Basecamp, and will allow you to keep up to date with how everyone working with you on the usability project is progressing from one easy interface, and keep on top of projects. As a busy person, it’s likely to be an invaluable tool to any usability specialist or contractor!

Insight’s website: http://www.encampapp.com/

Conclusion

Its clear from looking for the most useful usability apps for the iPhone that this is a largely untapped market – although there are tools available to help with the individual steps in the process, there is no one ‘go-to’ app for usability and UX professionals. What is needed is a more holistic solution, one that can work with a usability professional in every step of the process, from the initial requirement gathering stage, through building up personas of the customers, to the iterative design process and beyond. Until then we’ll have to make use of the wide range of apps covered today to achieve these tasks.

Related posts:

1. iPad Usability Study
2. The importance of usability in Mobile Geolocation games.
3. 7 aspects of successful usability questionnaires

My immediate thoughts are to compare this to Alan Cooper’s the inmates are running the asylum. This book is a harder read than Cooper’s – often going deep into highly technical topics (like how he would like to notate mouse clicks), and lacking the wit or lightness of Inmates. The most readable parts of Raskin’s books are the anecdotes about the development of the Mac and Canon Cat, and these are too few. However, this is likely due to a change in the intended audience, as Cooper’s book intends to sell usability concepts to a business audience, whereas Raskin aims his book directly at computing professionals.

Another key difference between Cooper and Raskin is they favour different methods of investigating the quality of an interface design. Whereas Cooper’s book favours qualitative data and methodology, through the establishment of persona’s and attempting to get inside user’s heads, Raskin favours quantitative methods. He includes a chapter on GOMS, a method of assigning arbitrary times for actions such as typing a keystroke, moving a mouse, thinking and moving from the mouse to the keyboard. Then by adding up the times it takes to do these actions, you can compare interaction methods by the time taken. (Its important to note that these times will not relate to the real world, as user’s act at different speeds, and can only be used to compare against other GOMS scores.)

My initial impression of this form of quantitative research is that it would highlight the speed/efficiency of an interaction, but not the quality – which is not necessarily the same thing. If a task takes a few seconds more, but is considered a lot more fulfilling, GOMS wouldn’t record this. This is particularly relevant to the field of videogames, where a purely GOMS based method to check interaction quality would lead to games such as this below:

Maybe the computer could press the button for you?

GOMS can be a useful tool to help compare interaction times, but should not be used exclusively.

Raskin also documents a number of problems with current interaction, with a particular dislike for modes (i.e. interactions that do different things in different concepts). A simple way to explain modes is the ‘caps lock’ key; turning on this mode will make ‘TEXT LIKE THIS’, despite my keystrokes being the same as when making ‘text like this’. He advocates an elimination of modes, as they introduce cognitive dissonance, and make it harder to form habits. A useful compromise, Raskin say’s is quasimodes, which is a mode that requires a constant input to achieve (and hence can be part of habit formation). This would include holding the shift key to produce capitals.

The elimination of modes extends into the elimination of applications – typing ‘SUM 7 + 6’ should produce ‘13’ everywhere, not just in a calculator. This improves the quality of interaction by allowing the user to be clear that the methods they have learnt will work anywhere. I believe this trend can be seen in current operating systems (such as the amalgamation of windows explorer and IE), and this is one of Google’s main aims with their OS.

Raskin also advocates an unlimited undo feature (even through closing and re-opening documents), and the elimination of dialog boxes asking ‘are you sure?’ These two are linked, giving that level of undo freedom would make ‘are you sure’ unnecessary, and is more technically feasible now than when the book was written. I assume it’s a matter of conventions, and momentum which would hinder people advocating these new interaction methods, and it is this mindset Raskin is trying to overturn.

An even more radical suggestion is Raskin’s radical redesign to information architecture. Looking at the hierarchical, folder methodology we have of storing files currently, Raskin notes that it is inefficient – from any point, you cannot see what’s in the folders below, or in the level above you. Since the book was published in 2000 we can see efforts have been made to combat these criticisms – in Windows, folder icons now show the file types inside (and previews if they are pictures), and have made it easier to go up a level. On Macs, they have additional folder view types that make it possible to see ‘up’ the hierarchy.

Raskin however has a more radical suggestion, which he calls ‘Zoom World’. Imagine, flying over a world with a series of zones, ‘I.e. pictures, home, work’. Then you zoom in on pictures (while still being able to see the others), and note that closer up we can see the pictures has it own sub zones, entitled ‘pictures of France’, ‘pictures of the dog’, ‘pictures of lily’ etc. Zooming in on ‘pictures of the dog’, now we are close we can see some individual pictures, one of the dog smiling, one of it playing with a ball

One of the dog playing counterstrike

Zooming in further on this picture would let us read and alter it, but we always have the option to quickly and freely zoom out and see any area of ZoomWorld. The advantage of this system is it solves the issues with being able to see the files above and below at any point, and not be restricted to your current folder. It has been implemented in ‘Archy’, which includes many features Raskin advocates in this book.

Ultimately its interesting seeing how many of the ideas Raskin advocates are ahead of their time, and were included in later revisions of Macs, and in general interaction. (such as searching starting from the first character you type, rather than waiting for you to press ‘enter’). As a book though, it’s harder to get through than Inmates, and does go on in exceptional depth about less than inspiring topics. Raskin talks endlessly about the Canon Cat, a system from the eighties with which he had tried to implement many of his interface ideas. He notes however that it met resistance from users who were used to the existing human computer interaction paradigm, and was not commercially successful. Perhaps, with the moves made by the leading Operating Systems, and Google OS breaking down the barriers between an OS and a browser, people would now be more susceptible to higher quality interaction with computers, and are prepared to unlearn their bad habits.

Related posts:

1. Alan Cooper’s The Inmates are Running the Asylum
2. Evaluating existing technologies, paper prototypes in action, Windows 7 and the disappointing user experience of my DVD player!

Nonetheless it is an important usability tool, and it is important that the responses received from questionnaires are of high quality, and useful. So, I’m going to share some of the areas that I, and other HCCS students, have identified as potential problems when dealing with questionnaires, in order to help you make better questionnaires. And since this is the internet, we’ll be presenting them in the form of a list, as everyone on the internet loves lists!

everyone on the internet also loves pictures of cats

So, here are seven important aspects to consider when creating questionnaires.  

1. Answers can only be as good as your questions

When preparing a questionnaire, you need to think at length about the aspect of the subject you want to investigate, and go in knowing what you need to find out. Generalized questions, or being vague on the topic, won’t give useful data, and so it’s important to make sure the questions are actually asking relevant things. For example if you wanted to find out about… the most popular aisles in Sainsbury’s, asking questions about whether people prefer the supermarket to its rivals wouldn’t get closer to this goal. Also we all know it’s the cereal aisle. So, know what you want to find out from the questionnaire.

2. The questions need to cover the areas in depth.

            When getting opinions, it helps to be specific. Don’t just ask ‘did you like this’, but follow it up with either a question asking for reasons why, or (if you’re after a data set that can be analyzed more uniformly), ask them to rate on a number of scales why they did or didn’t like it (i.e. “to what extent did the look of the webpage affect your opinion of it”). Not doing this will lead to closed answers (Did you like this? “no”), when it would be possible to get a much richer set of data from the participant. Whether you select an open question ‘why’ or a closed question (based on scales), depends on whether you are after purely quantitative data, or also want to include qualitative data as well.

3. Changing the questions mid-implementation taints your qualitative data

Halfway through a study, the results may start to show interesting trends that you’d want to find more about. Take caution when altering the questionnaire to investigate these trends. Adding more questions should be fine (except for the tired participants!), but when editing a question that already exists (i.e. from ‘did you like the look and feel of the website’ to ‘did you like the look and feel of the first page of the website’), keep in mind that this will invalidate getting a quantitative response (i.e. ‘85% of people liked the look and feel of the first page of the website’) from the entire dataset for that question, as the participants have been answering different questions.

4. Subjective answers need to be standardized

Remember, when asking whether something was ‘easy’ or ‘hard’, that answers to theses questions are going to be subjective. People are likely to have a wide range of expectations about how a system should be, and a wide range of experience, and so will be judging on separate scales.

Dr Graham McAllister tells a story related to this. When doing usability testing, he asked ‘did anyone have any problems with the program’… no reply. So he asked instead ‘did anyone think that someone else may have problems with this program’, and a whole host of replies were given from the same people.

Don’t forget that pride can be a factor preventing people from saying they found task’s hard. Shifting the focus of the questions from the participant to the medium can help prevent this.

Also, terms such as ‘often’ or ‘rarely’ mean different things to different people. Try and replace them with specific terms ‘every day’, ‘every week’ etc.

5. The questions reflect your opinion

Because of the close controlled environment that a questionnaire creates (i.e. the participants can only answer the questions they have been asked) it is important to make sure that the researchers opinions do not show through the questions. For example, leading questions, which make it easier to answer one way than the other. I saw an advert recently, for some sort of Christian business, that asked ‘Does god exist?’ with tick boxes for ‘Yes’ ‘Probably’ and ‘No’. This is a leading question – the only indefinite reply implies agreement. Where is ‘probably not’, ‘neither agree or disagree’ or ‘don’t know’? (Answer: not on an advert paid for by the church)

6. You need to give people a reason to participate

now thats an incentive

Before I go on with this list, I was wondering if you’d be happy to answer 25 questions on your opinions of southern English fauna and shrubbery. Please click here to fill it out.

Did I mention that filling out the survey gets you a £25 amazon voucher? Do you want that link again?

The point, as I’m sure you guessed, was that you need to offer an incentive for people to participate in your questionnaire, otherwise only people really interested in the subject will reply. Suitable incentives would be discounts, free products, a prize draw, or something related to the field you are investigating.

7. The data can be skewed towards extreme opinions

Failing to give a good enough incentive or no incentive at all, will end up with unrepresentative data – only people who feel so strongly about the subject matter to reply will bother to. In practise this will either be people who are really angry about it, or people who love it, and this will skew your data towards the extremes. To ensure you get a natural selection of participants, steps need to be taken, such as pre-selecting participants, or offering incentives as covered above.

So there we have it. Seven tips to help you make effective questionnaires. Enjoy asking people things!

Related posts:

1. A Terrible User Experience & how to fix it– Zoomerang.com
2. The Likert scale – Or “How I learnt to stop worrying, and ‘strongly enjoy’ the bomb”.
3. Five essential iPhone apps for usability professionals