Posts Tagged ‘questionnaires’
The Problems with Surveys for User Experience Tests
In the run up to Margaret Thatchers’ election victory in 1979, a poll was taken to estimate who would vote for her. Only 1 in 100 said yes. However, as revealed by the final results, 1 in 3 actually voted for her. The poll was inaccurate, and inappropriate for the task.
Surveys are a common tool used to evaluate a participant’s opinions of the user experience, and usability of a system. I’ve written about how to make good questionnaires before, and have often seen them used as a tool when analysing a large group of participants. However, as a method of understanding users, they are imperfect, and not just because they are poorly designed – instead it’s a fundamental problem with surveys. Let’s look at why this is the case, and why people are tempted to use surveys despite this.
Where are surveys used?
When I’ve been involved with user tests for games, I’ve often seen surveys used as a way of recording the player’s experience. For example, after completing a level, or game mode, they would be asked to rate their experience on a Likert scale (1-10), on categories such as how difficult they found the level, how fun it was, how it compared to other levels. This is often complemented by text notes, where the participant can write in things they particularly liked or disliked.
Outside of gaming, surveys can often be found on the internet – such as website’s satisfaction surveys, or on professional survey sites, like Survey Monkey.
Survey Monkey in action
Why are surveys used?
It’s easy to understand why surveys are often used when testing user experience. Most obvious is that they are easy to quantify, since the scores are given as a numeric value, which can then be averaged, and given an overall ‘score’. This can then be stuck on a graph, to impress people too busy and important to be involved with the testing itself. Compared to moderated testing, simple analysis is easy, and ‘results’ can be gained with little effort – particularly if an online survey tool is used.
Similarly, with surveys it’s easy to get a large number of opinions quickly, and in a largely un-moderated setting. Hence, 10 (or 10,000) people can test a game at the same time, with only light moderation, and fill out a survey after to record their views. Surveys also don’t require a large degree of specialist equipment – just a printer, and a pen (or they can be done online). This makes them cheaper than many moderated settings, which require a lab decked out with recording equipment.
Problem with surveys
Surveys sound great, don’t they. Cheap, Easy, and give some hard numbers. However, there are a number of problems with surveys, and one key issue that prevent them being suitable for user experience analysis.
First of all, it’s easy for the data from surveys to be misrepresented (either unintentionally or to further a top secret agenda!). Without hard evidence, such as watching (and recording) an individual player of the game, the analysis becomes reduced to which level ‘scores better’, regardless of the intricacies of the play test. Minor issues become lost within the overarching ‘score’.
Much more importantly, the fundamental problem with attempting to understand user experience with a survey is that they log opinions, and not behaviour. People are (sometimes?) stupid, and don’t know what they think. So a player who has had a positive experience throughout a level, and got stuck near the end, will often be left thinking poorly of the entire level. And without an independent observer to monitor, their in-game opinions are lost, or forgotten. Just like I cannot tell how bad my singing is, a player is too close to the subject matter to gain a full understanding of it.
Its pretty bad...
Essentially, surveys introduce a layer of abstraction from the game that is difficult for a player to follow. It is difficult for them to recognise what parts of a game made it fun, and which parts frustrated them, and it often takes someone else to spot these patterns.
Pride, and psychology can also be a contributing factor – players who have needed 10 attempts to complete a section will still say it was “easy” after finally completing it – psychologically they will often believe it as well, since they have felt the satisfaction of completing the task. Other times they will be too proud to say the section was too difficult, and lie. Again, this rich data is lost through a survey.
What should be used instead?
To gain a truer understanding of the user experience (or player experience) of participants when testing a system, or a game, surveys are therefore inadequate. Instead, a moderated task based analysis session, which is recorded for later analysis, will give a truer understanding of how the participant found the system, and their true experience, unaltered by their own perceptions. I’ve written about recording these sessions before, and will discussed them further in the future.
As we have seen, surveys are cheap and easy, and hence should not be disregarded entirely. However they should not be used exclusively, as they can miss key user experience findings, and require users to know themselves, and their feelings, extensively.
A Terrible User Experience & how to fix it– Zoomerang.com
When running a website, its important to make sure that the user can achieve their goal with the minimal fuss. This is especially important if you are selling, or trying to sell, a commercial service. I recently had to use Zoomerang.com, a survey site, and had a few notes about the user experience. As you’ll remember, I don’t rant often…
I’m in the process of designing a GPS game, and am currently discovering the functional requirements for the project. As UX practitioners, we know that involving the user is of critical importance at this stage, hence we designed a questionnaire to establish peoples experience, and perception of GPS games, and what they’d like a GPS game to be like. (linked here)
When at university, our internet access goes through a proxy server, which blocks unsuitable content. For some reason, this includes surveymonkey.com, a site I’ve used a few times in the past to construct online surveys. Interestingly, the ‘site blocked’ dialogue said “for survey sites, try zoomerang.com”. However, when I search for some hardcore action, it never gives me alternate suggestions for that. Have I uncovered a conspiracy? Nonetheless, I followed the link.
And so I ended up on zoomerang.com. Being fair, there is one key advantage to zoomerang which immediately put me in a good mood. On surveymonkey, for a free account, you are limited to ten questions. On Zoomerang, you can ask 30 questions before you have to pay. This meant we didn’t have to redo, or concatenate our questions, and made me smile inside
Pictured: an inside smile
Problems with Zoomerang.com
This goodwill was shortlived, when I tried to use the site to implement my questionnaire. Heres why:
- The workflow isn’t clear when making a survey, and so I entirely missed the step where you add your questions. Clicking through the process actually caused me publish a blank questionnaire. Which wouldn’t be a problem, except…
- …You can’t edit an existing survey. Once its published, you cannot add/remove/change questions. Surveymonkey allows this. So I was stuck with my blank survey, and had to start again from scratch.
- Having figured out how to add questions, I got started, and selected “insert question”. It added a header, which then had to be changed to type question. I guessed that was because it was my first item, but no, it always defaults to inserting a header (odd, since you’d only need 1 per page, whereas you’d need multiple questions).
- So I finally got to add a question, and this is when the terribleness of the design struck me. I selected a question where a radio button would select from a number of answers, and typed in my list of 15 or so alternative answers into a rich text field. I hit submit, and … got an error, saying “answers can only be 1000 characters, including HTML”, and even worse…
- …It deleted the data I had entered in that field. All 15 answers. This is a critical failure of any system, since the data a user inputs should be considered sacred.
- There was no counter telling me how many characters I had entered, so I had to retry a few times. Eventually I realised that I could only enter 5 potential one word answers before it’d error that I was over 1000 characters. That had to be a mistake? I investigated further…
- …Looking at the HTML, it turned out that the rich text editor was writing rubbish html. At the start of each answer, it’d add needless style tags, often multiple times. Heres an example of the HTML it generated for my one word answer “complicated”
<p><span style=”font-family: Arial; color: #000000; font-size: small;”><span style=”font-family: Arial; color: #000000; font-size: small;”><span style=”font-family: Arial; color: #000000; font-size: small;”>Complicated</span></span></span></p> - …no wonder it was hitting the character limit after 4 or 5 words. I had to manually enter the html for all the possible answers, just so I could get round this.
- My last fault with zoomerang.com is just a suspicion. I look after my email accounts, and so have never received spam in my current primary address. After signing up for zoomerang last week, I received my first random spam email. Might just be a coincidence, but I didn’t sign up for anything else that week!
I had to call this guy to fix my survey
How to fix zoomerang.com
To improve their user experience, they should look at red-routing the goals the user needs to achieve:
- Make the progression through survey design clearer, highlighting which step questions are added in
- Also make it clear how far through the design process you are, and what steps remain
- Restrict what the user can do, so they cannot post a blank questionnaire. Its obvious if they are about to do this that they’ve made a mistake, tell them!
- Don’t make question types default to “header”. Surely users will only use this type once at most, whereas they’re going to have more than one question on the questionnaire. Make it default to that!
- Fix the WYSIWYG code generator, so that the user doesn’t have to manually code the answers in HTML. A lot of user’s would get stuck at this point!
- Don’t send me spam!
And what can you do, until these fixes are made? Use surveymonkey.com. Or, if you’ve found anything better, leave a comment and let me know!
The Likert scale – Or “How I learnt to stop worrying, and ‘strongly enjoy’ the bomb”.
As a practitioner of usability or user experience, a common way that you will attempt to investigate a user (or player, or customer)’s perceptions is through designing and implementing a survey. In designing a survey, its important to consider the format that questions come in, especially with common question types such as “How frustrating did you find this level?.” Today we’ll look at one of the most common question formats, the Likert scale, and the implications that using it has on your studies.
What is the Likert scale?
Lets start with an example.
Most people have seen a Likert scale before. Do you agree with this statement?
- Strongly agree
- Agree
- Neither agree or disagree
- Disagree
- Strongly disagree
And the responses should be balanced... unless you have an agenda
Often used to gauge opinions, they are especially important for people involved with measuring usability or player experience, as they can help quantify subjective things like a user’s experiences. They are usually in the form of a statement, followed by a selection of statements, to indicate how far someone agrees with the statement. They can often be used to quantify things like ease-of-use, or fun, which would be impossible to quantify through other methods. Hence they are of particularly important for us, since user experience is essentially abstract.
Different kinds of Likert scales.
The essential question when it comes to implementing a Likert scale, is how many responses to offer.
‘Forced Choice’ scales are those which have an even number of options. Essentially this means missing out the ‘neither agree or disagree’ option, and forcing the participant to make a selection (see what they did with the name? very clever!). This would be done to force participants to show an opinion, but there are dangers inherent with this. Forcing a response may give a larger degree of ‘static’ in the responses, reducing their accuracy, since the responses may not map their opinions. People who don’t agree or disagree may not be happy about being forced to give an opinion, reducing their likelihood to answer later questions accurately. However if your aim is to support a conclusion that people do/do-not like a system, you may be willing to risk these to prove your point when designing the survey.
Forced choice means its hard to tell who is neutral, and who doesn’t want to participate
If you select to use a scale with an odd number of options, there are a few issues that should be kept in mind when deciding between a five or seven point scale. The most obvious difference is that a finer grain of responses can be analysed from a seven point scale, as it can represent a wider range of views. Also, take into account that it’s been shown participants shy away from the ‘edges’, the extreme like and dislike options offered. This means a five point scale will likely only get responses in the ‘slightly’ columns from all except the most ardent fanatics. Again, you have to consider whether a wider range of responses is useful to the topic you are exploring.
Should you use a Likert Scale
Ultimately if you are trying to track opinions, a Likert scale is a good method of accessing this data. There is no all-encompassing correct answer for which scale is appropriate, the context of use and what you want to find out will all affect this. As long as you keep in mind that not only the phrasing of the question, but the range and number of responses you offer will affect the results, and anticipate this affect, you can’t really go wrong. Happy surveying!
Happy New Decade
Welcome to 2010, hope everyone had a great holiday.
Straight back onto UX issues -
For my HCI Coursework, I’m designing a new product, with a focus on evaluating the user requirements -
To do this, I’ve made a survey (keeping in mind the issues I raised on making a UX Survey), to help me understand user’s needs.
It’d make my decade if you’d fill out the survey, it only takes 2-3 minutes and doesnt require any personal details
Feel free to comment here aswell, if you find any issues with the survey itself.
I have lots of ideas for UX blog posts for 2010, so expect alot more to come soon…
Steve
Five essential iPhone apps for usability professionals
As you may have noticed, I like my iPhone. But the iPhone is not just for playing games, or twitter, but can also be a useful tool for every stage of usability testing, from the discovery of customer needs to the iterative testing of potential designs and implementation. I’ve tried to compile five essential apps for usability professionals, available on the iPhone now. If anyone has any suggestions/additions, or disagrees with the choices I’ve made, please add your comments!
Working with usability requires a wide range of abilities, and hence these tools reflect the wide range of roles a usability professional may be performing. When talking about each app, I’ll try to cover where in the usability process it may come in useful, and how it can help you to increase efficiency and get better results.
but don't forget lo-fi alternatives
1. Pidoco°
Starting off, this isn’t strictly an iPhone app, but Pidoco’s software has been developed to be compatible with mobile browsers and the iPhone. I saw them demo the software at a recent UX Brighton event, and they impressed upon me that it was built with usability in mind. Pidoco offers a quick way to collaborate on the design of wire frames, and quickly show these to the user or client, without the need to be in the same geographic location as them. The easy to use nature of the system means it’d be ideal when iteratively testing an interface with users, and is a mid-fi alternative to paper prototyping. Currently offering a month’s free trial, there’s no excuse not to give it a go!
Pidoco°’s website: https://pidoco.com/en/
2. Mocha VNC
Need to mock up a mid-fi prototype of an iPhone App? You obviously don’t have time to make the app in Xcode, buy the iPhone developers license, link the customer’s phone to the computer and copy the app over. Instead, how about making a prototype on your Mac, or PC and using a VNC client to make the iPhone look at the prototype. The user will still be able to interact with the prototype by clicking, or navigating around, and the ‘controller’ sat at the server can display the correct pages to the user. Easy mid-fi prototyping, on an iPhone, without the long set up costs and investments required to actually have the app running on the iPhone.
Mocha VNC’s website: http://www.mochasoft.dk/iphone_vnc.htm
3. Camera
Watching someone perform a task is an important part qualitative usability testing. What about those moments where you weren’t looking, or noticed something that you wanted to ask about later, but forgot about? Looks like you need to be recording your user tests. Camera comes with all iPhones, but is only available for video recording on the 3GS. Since you are always likely to have your phone with you, it’s a handy mobile solution that allows you to record your usability sessions without requiring any additional equipment, so that they can be analysed later. Maybe buy a stand for it though!
4. exZact Data Collection
Quantitative research is also an important part of establishing user requirements, and working towards a design that meets their goals. The Data Collection iPhone app aims to allow you to create, and answer, custom surveys on the iPhone, collecting information like geographic location, and responses in a variety of forms (1-10 ratings, drop down lists, yes/no responses, and comment boxes). They then offer real time analysis, so you can see the results as they come in, and claim to be massively scalable. Offering a month’s free trial, it seems an effective solution if you require your data collection method to be mobile, and accessible anywhere.
exZact’s website: http://www.iformbuilder.com
5. Insight – Basecamp on iPhone
You’ve seen from the diversity of the apps above that a usability specialist has to be experienced in a lot of areas, and its often hard to keep on top of these. A tool that I have found invaluable in project management is Basecamp, which allows you to collaborate on projects, assign tasks and see what everyone is up to. Insight is a mobile version of Basecamp, and will allow you to keep up to date with how everyone working with you on the usability project is progressing from one easy interface, and keep on top of projects. As a busy person, it’s likely to be an invaluable tool to any usability specialist or contractor!
Insight’s website: http://www.encampapp.com/
Conclusion
Its clear from looking for the most useful usability apps for the iPhone that this is a largely untapped market – although there are tools available to help with the individual steps in the process, there is no one ‘go-to’ app for usability and UX professionals. What is needed is a more holistic solution, one that can work with a usability professional in every step of the process, from the initial requirement gathering stage, through building up personas of the customers, to the iterative design process and beyond. Until then we’ll have to make use of the wide range of apps covered today to achieve these tasks.
