{"id":233,"date":"2009-11-11T15:11:05","date_gmt":"2009-11-11T14:11:05","guid":{"rendered":"http:\/\/www.stevebromley.com\/blog\/?p=233"},"modified":"2019-11-06T17:45:04","modified_gmt":"2019-11-06T16:45:04","slug":"7-aspects-of-successful-usability-questionnaires","status":"publish","type":"post","link":"https:\/\/www.stevebromley.com\/blog\/2009\/11\/11\/7-aspects-of-successful-usability-questionnaires\/","title":{"rendered":"7 aspects of successful usability questionnaires"},"content":{"rendered":"

This week in HCI we\u2019ve been thinking about questionnaires. They can be an important usability tool, although there are also many limitations. Primarily questionnaires are used as a quantitative data collection method (i.e. it will give back a large amount of responses), and so, compared to a qualitative methodology, are useful in pinpointing where problems exist, but less helpful in helping us understand why. As such, it is best to combine both forms of research, perhaps by starting off with questionnaires to identify frequent problem areas, and generalized opinions of systems, before moving into a qualitative method to understand why these areas are problems. An advantage of questionnaires include the fact that they are cheaper and quicker to get results from than many other methods, but this is balanced by some drawbacks \u2013 the data you record is more subjectively influenced by the researcher and participants opinions than in other methods, such as direct observation.<\/p>\n

Nonetheless it is an important usability tool, and it is important that the responses received from questionnaires are of high quality, and useful. So, I\u2019m going to share some of the areas that I, and other HCCS students, have identified as potential problems when dealing with questionnaires, in order to help you make better questionnaires. And since this is the internet, we\u2019ll be presenting them in the form of a list, as everyone on the internet loves lists!<\/p>\n

\"everyone

everyone on the internet also loves pictures of cats<\/p><\/div><\/div>\n

So, here are seven important aspects to consider when creating questionnaires.\u00a0\u00a0<\/p>\n

1. Answers can only be as good as your questions<\/h4>\n

When preparing a questionnaire, you need to think at length about the aspect of the subject you want to investigate, and go in knowing what you need to find out. Generalized questions, or being vague on the topic, won\u2019t give useful data, and so it\u2019s important to make sure the questions are actually asking relevant things. For example if you wanted to find out about… the most popular aisles in Sainsbury\u2019s, asking questions about whether people prefer the supermarket to its rivals wouldn\u2019t get closer to this goal. Also we all know it\u2019s the cereal aisle. So, know what you want to find out from the questionnaire.<\/p>\n

2. The questions need to cover the areas in depth.<\/h4>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 When getting opinions, it helps to be specific. Don\u2019t just ask \u2018did you like this\u2019, but follow it up with either a question asking for reasons why, or (if you\u2019re after a data set that can be analyzed more uniformly), ask them to rate on a number of scales why they did or didn\u2019t like it (i.e. \u201cto what extent did the look of the webpage affect your opinion of it\u201d). Not doing this will lead to closed answers (Did you like this? \u201cno\u201d), when it would be possible to get a much richer set of data from the participant. Whether you select an open question \u2018why\u2019 or a closed question (based on scales), depends on whether you are after purely quantitative data, or also want to include qualitative data as well.<\/p>\n

3. Changing the questions mid-implementation taints your qualitative data<\/h4>\n

Halfway through a study, the results may start to show interesting trends that you\u2019d want to find more about. Take caution when altering the questionnaire to investigate these trends. Adding more questions should be fine (except for the tired participants!), but when editing a question that already exists (i.e. from \u2018did you like the look and feel of the website\u2019 to \u2018did you like the look and feel of the first page of the website\u2019), keep in mind that this will invalidate getting a quantitative response (i.e. \u201885% of people liked the look and feel of the first page of the website\u2019) from the entire dataset for that question, as the participants have been answering different questions.<\/p>\n

4. Subjective answers need to be standardized<\/h4>\n

Remember, when asking whether something was \u2018easy\u2019 or \u2018hard\u2019, that answers to theses questions are going to be subjective. People are likely to have a wide range of expectations about how a system should be, and a wide range of experience, and so will be judging on separate scales.<\/p>\n

Dr Graham McAllister<\/a> tells a story related to this. When doing usability testing, he asked \u2018did anyone have any problems with the program\u2019\u2026 no reply. So he asked instead \u2018did anyone think that someone else may have problems with this program\u2019, and a whole host of replies were given from the same people.<\/p>\n

Don\u2019t forget that pride can be a factor preventing people from saying they found task\u2019s hard. Shifting the focus of the questions from the participant to the medium can help prevent this.<\/p>\n

Also, terms such as \u2018often\u2019 or \u2018rarely\u2019 mean different things to different people. Try and replace them with specific terms \u2018every day\u2019, \u2018every week\u2019 etc.<\/p>\n

5. The questions reflect your opinion<\/h4>\n

Because of the close controlled environment that a questionnaire creates (i.e. the participants can only answer the questions they have been asked) it is important to make sure that the researchers opinions do not show through the questions. For example, leading questions, which make it easier to answer one way than the other. I saw an advert recently, for some sort of Christian business, that asked \u2018Does god exist?\u2019 with tick boxes for \u2018Yes\u2019 \u2018Probably\u2019 and \u2018No\u2019. This is a leading question \u2013 the only indefinite reply implies agreement. Where is \u2018probably not\u2019, \u2018neither agree or disagree\u2019 or \u2018don\u2019t know\u2019? (Answer: not on an advert paid for by the church)<\/p>\n

6. You need to give people a reason to participate<\/h4>\n
\"now

now thats an incentive<\/p><\/div><\/div>\n

Before I go on with this list, I was wondering if you\u2019d be happy to answer 25 questions on your opinions of southern English fauna and shrubbery. Please click here<\/a> to fill it out.<\/p>\n

Did I mention that filling out the survey gets you a \u00a325 amazon voucher? Do you want that link again?<\/p>\n

The point, as I\u2019m sure you guessed, was that you need to offer an incentive for people to participate in your questionnaire, otherwise only people really interested in the subject will reply. Suitable incentives would be discounts, free products, a prize draw, or something related to the field you are investigating.<\/p>\n

7. The data can be skewed towards extreme opinions<\/h4>\n

Failing to give a good enough incentive or no incentive at all, will end up with unrepresentative data \u2013 only people who feel so strongly about the subject matter to reply will bother to. In practise this will either be people who are really angry about it, or people who love it, and this will skew your data towards the extremes. To ensure you get a natural selection of participants, steps need to be taken, such as pre-selecting participants, or offering incentives as covered above.<\/p>\n

So there we have it. Seven tips to help you make effective questionnaires. Enjoy asking people things!<\/p>\n","protected":false},"excerpt":{"rendered":"

This week in HCI we\u2019ve been thinking about questionnaires. They can be an important usability tool, although there are also many limitations. Primarily questionnaires are used as a quantitative data collection method (i.e. it will give back a large amount of responses), and so, compared to a qualitative methodology, are useful in pinpointing where problems […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[118],"tags":[],"class_list":["post-233","post","type-post","status-publish","format-standard","hentry","category-user-research","grve-entry-item","grve-blog-item"],"yoast_head":"\n7 aspects of successful usability questionnaires - Steve Bromley - User Research<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/11\/7-aspects-of-successful-usability-questionnaires\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"7 aspects of successful usability questionnaires - Steve Bromley - User Research\" \/>\n<meta property=\"og:description\" content=\"This week in HCI we\u2019ve been thinking about questionnaires. They can be an important usability tool, although there are also many limitations. Primarily questionnaires are used as a quantitative data collection method (i.e. it will give back a large amount of responses), and so, compared to a qualitative methodology, are useful in pinpointing where problems […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/11\/7-aspects-of-successful-usability-questionnaires\/\" \/>\n<meta property=\"og:site_name\" content=\"Steve Bromley - User Research\" \/>\n<meta property=\"article:published_time\" content=\"2009-11-11T14:11:05+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-11-06T16:45:04+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/www.stevebromley.com\/blog\/wp-content\/uploads\/2009\/11\/computer-cat11.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#website\",\"url\":\"https:\/\/www.stevebromley.com\/blog\/\",\"name\":\"Steve Bromley - User Research\",\"description\":\"Usability and User Research for Websites, Software and Games\",\"publisher\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#\/schema\/person\/9c0be0bbd079c086677d422d1fd9c8c7\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/www.stevebromley.com\/blog\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/11\/7-aspects-of-successful-usability-questionnaires\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"http:\/\/www.stevebromley.com\/blog\/wp-content\/uploads\/2009\/11\/computer-cat11.png\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/11\/7-aspects-of-successful-usability-questionnaires\/#webpage\",\"url\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/11\/7-aspects-of-successful-usability-questionnaires\/\",\"name\":\"7 aspects of successful usability questionnaires - Steve Bromley - User Research\",\"isPartOf\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/11\/7-aspects-of-successful-usability-questionnaires\/#primaryimage\"},\"datePublished\":\"2009-11-11T14:11:05+00:00\",\"dateModified\":\"2019-11-06T16:45:04+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/11\/7-aspects-of-successful-usability-questionnaires\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/11\/7-aspects-of-successful-usability-questionnaires\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/11\/7-aspects-of-successful-usability-questionnaires\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/\",\"url\":\"https:\/\/www.stevebromley.com\/blog\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/11\/7-aspects-of-successful-usability-questionnaires\/\",\"url\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/11\/7-aspects-of-successful-usability-questionnaires\/\",\"name\":\"7 aspects of successful usability questionnaires\"}}]},{\"@type\":\"Article\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/11\/7-aspects-of-successful-usability-questionnaires\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/11\/7-aspects-of-successful-usability-questionnaires\/#webpage\"},\"author\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#\/schema\/person\/9c0be0bbd079c086677d422d1fd9c8c7\"},\"headline\":\"7 aspects of successful usability questionnaires\",\"datePublished\":\"2009-11-11T14:11:05+00:00\",\"dateModified\":\"2019-11-06T16:45:04+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/11\/7-aspects-of-successful-usability-questionnaires\/#webpage\"},\"commentCount\":4,\"publisher\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#\/schema\/person\/9c0be0bbd079c086677d422d1fd9c8c7\"},\"image\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/11\/7-aspects-of-successful-usability-questionnaires\/#primaryimage\"},\"articleSection\":\"User Research\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/11\/7-aspects-of-successful-usability-questionnaires\/#respond\"]}]},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#\/schema\/person\/9c0be0bbd079c086677d422d1fd9c8c7\",\"name\":\"Steve Bromley\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/4dfbbfc5a00187fd6f5fd405361347b2698a65a866f49de07f9486895b6c7029?s=96&d=mm&r=g\",\"caption\":\"Steve Bromley\"},\"logo\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#personlogo\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/posts\/233","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/comments?post=233"}],"version-history":[{"count":6,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/posts\/233\/revisions"}],"predecessor-version":[{"id":635,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/posts\/233\/revisions\/635"}],"wp:attachment":[{"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/media?parent=233"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/categories?post=233"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/tags?post=233"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}