{"id":941,"date":"2010-07-07T08:45:17","date_gmt":"2010-07-07T07:45:17","guid":{"rendered":"http:\/\/www.stevebromley.com\/blog\/?p=941"},"modified":"2019-11-06T17:45:02","modified_gmt":"2019-11-06T16:45:02","slug":"the-problems-with-surveys-for-user-experience-tests","status":"publish","type":"post","link":"https:\/\/www.stevebromley.com\/blog\/2010\/07\/07\/the-problems-with-surveys-for-user-experience-tests\/","title":{"rendered":"The Problems with Surveys for User Experience Tests"},"content":{"rendered":"

In the run up to Margaret Thatchers\u2019 election victory in 1979, a poll was taken to estimate who would vote for her. Only 1 in 100 said yes. However, as revealed by the final results, 1 in 3 actually voted for her. The poll was inaccurate, and inappropriate for the task. <\/p>\n

Surveys are a common tool used to evaluate a participant\u2019s opinions of the user experience, and usability of a system. I\u2019ve written about how to make good questionnaires<\/a> before, and have often seen them used as a tool when analysing a large group of participants. However, as a method of understanding users, they are imperfect, and not just because they are poorly designed \u2013 instead it\u2019s a fundamental problem with surveys. Let\u2019s look at why this is the case, and why people are tempted to use surveys despite this.<\/p>\n

Where are surveys used?<\/H4><\/p>\n

When I\u2019ve been involved with user tests for games, I\u2019ve often seen surveys used as a way of recording the player\u2019s experience. For example, after completing a level, or game mode, they would be asked to rate their experience on a Likert scale (1-10), on categories such as how difficult they found the level, how fun it was, how it compared to other levels. This is often complemented by text notes, where the participant can write in things they particularly liked or disliked.<\/p>\n

Outside of gaming, surveys can often be found on the internet \u2013 such as website\u2019s satisfaction surveys, or on professional survey sites, like Survey Monkey.<\/p>\n

\n
\"Monkey

Survey Monkey in action<\/p><\/div>\n<\/div>\n

Why are surveys used?<\/H4><\/p>\n

It\u2019s easy to understand why surveys are often used when testing user experience. Most obvious is that they are easy to quantify, since the scores are given as a numeric value, which can then be averaged, and given an overall \u2018score\u2019. This can then be stuck on a graph, to impress people too busy and important to be involved with the testing itself. Compared to moderated testing, simple analysis is easy, and \u2018results\u2019 can be gained with little effort \u2013 particularly if an online survey tool is used.<\/p>\n

Similarly, with surveys it\u2019s easy to get a large number of opinions quickly, and in a largely un-moderated setting. Hence, 10 (or 10,000) people can test a game at the same time, with only light moderation, and fill out a survey after to record their views. Surveys also don\u2019t require a large degree of specialist equipment \u2013 just a printer, and a pen (or they can be done online). This makes them cheaper than many moderated settings, which require a lab decked out with recording equipment.<\/p>\n

Problem with surveys<\/H4><\/p>\n

Surveys sound great, don\u2019t they. Cheap, Easy, and give some hard numbers. However, there are a number of problems with surveys, and one key issue that prevent them being suitable for user experience analysis.<\/p>\n

First of all, it\u2019s easy for the data from surveys to be misrepresented (either unintentionally or to further a top secret agenda!). Without hard evidence, such as watching (and recording) an individual player of the game, the analysis becomes reduced to which level \u2018scores better\u2019, regardless of the intricacies of the play test. Minor issues become lost within the overarching \u2018score\u2019.<\/p>\n

Much more importantly, the fundamental problem with attempting to understand user experience with a survey is that they log opinions, and not behaviour. People are (sometimes?) stupid, and don\u2019t know what they think. So a player who has had a positive experience throughout a level, and got stuck near the end, will often be left thinking poorly of the entire level. And without an independent observer to monitor, their in-game opinions are lost, or forgotten. Just like I cannot tell how bad my singing is, a player is too close to the subject matter to gain a full understanding of it.<\/p>\n

\n
\"Guitar

Its pretty bad...<\/p><\/div>\n<\/div>\n

Essentially, surveys introduce a layer of abstraction from the game that is difficult for a player to follow. It is difficult for them to recognise what parts of a game made it fun, and which parts frustrated them, and it often takes someone else to spot these patterns.<\/p>\n

Pride, and psychology can also be a contributing factor \u2013 players who have needed 10 attempts to complete a section will still say it was \u201ceasy\u201d after finally completing it \u2013 psychologically they will often believe it as well, since they have felt the satisfaction of completing the task. Other times they will be too proud to say the section was too difficult, and lie. Again, this rich data is lost through a survey.<\/p>\n

What should be used instead?<\/H4><\/p>\n

To gain a truer understanding of the user experience (or player experience) of participants when testing a system, or a game, surveys are therefore inadequate. Instead, a moderated task based analysis session, which is recorded for later analysis, will give a truer understanding of how the participant found the system, and their true experience, unaltered by their own perceptions. I\u2019ve written about recording these sessions<\/a> before, and will discussed them further in the future. <\/p>\n

As we have seen, surveys are cheap and easy, and hence should not be disregarded entirely. However they should not be used exclusively, as they can miss key user experience findings, and require users to know themselves, and their feelings, extensively.<\/p>\n","protected":false},"excerpt":{"rendered":"

In the run up to Margaret Thatchers\u2019 election victory in 1979, a poll was taken to estimate who would vote for her. Only 1 in 100 said yes. However, as revealed by the final results, 1 in 3 actually voted for her. The poll was inaccurate, and inappropriate for the task. Surveys are a common […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[118],"tags":[],"class_list":["post-941","post","type-post","status-publish","format-standard","hentry","category-user-research","grve-entry-item","grve-blog-item"],"yoast_head":"\nThe Problems with Surveys for User Experience Tests - Steve Bromley - User Research<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.stevebromley.com\/blog\/2010\/07\/07\/the-problems-with-surveys-for-user-experience-tests\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The Problems with Surveys for User Experience Tests - Steve Bromley - User Research\" \/>\n<meta property=\"og:description\" content=\"In the run up to Margaret Thatchers\u2019 election victory in 1979, a poll was taken to estimate who would vote for her. Only 1 in 100 said yes. However, as revealed by the final results, 1 in 3 actually voted for her. The poll was inaccurate, and inappropriate for the task. Surveys are a common […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.stevebromley.com\/blog\/2010\/07\/07\/the-problems-with-surveys-for-user-experience-tests\/\" \/>\n<meta property=\"og:site_name\" content=\"Steve Bromley - User Research\" \/>\n<meta property=\"article:published_time\" content=\"2010-07-07T07:45:17+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-11-06T16:45:02+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/www.stevebromley.com\/blog\/wp-content\/uploads\/2010\/07\/survey.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#website\",\"url\":\"https:\/\/www.stevebromley.com\/blog\/\",\"name\":\"Steve Bromley - User Research\",\"description\":\"Usability and User Research for Websites, Software and Games\",\"publisher\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#\/schema\/person\/9c0be0bbd079c086677d422d1fd9c8c7\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/www.stevebromley.com\/blog\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2010\/07\/07\/the-problems-with-surveys-for-user-experience-tests\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"http:\/\/www.stevebromley.com\/blog\/wp-content\/uploads\/2010\/07\/survey.png\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2010\/07\/07\/the-problems-with-surveys-for-user-experience-tests\/#webpage\",\"url\":\"https:\/\/www.stevebromley.com\/blog\/2010\/07\/07\/the-problems-with-surveys-for-user-experience-tests\/\",\"name\":\"The Problems with Surveys for User Experience Tests - Steve Bromley - User Research\",\"isPartOf\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2010\/07\/07\/the-problems-with-surveys-for-user-experience-tests\/#primaryimage\"},\"datePublished\":\"2010-07-07T07:45:17+00:00\",\"dateModified\":\"2019-11-06T16:45:02+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2010\/07\/07\/the-problems-with-surveys-for-user-experience-tests\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.stevebromley.com\/blog\/2010\/07\/07\/the-problems-with-surveys-for-user-experience-tests\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2010\/07\/07\/the-problems-with-surveys-for-user-experience-tests\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/\",\"url\":\"https:\/\/www.stevebromley.com\/blog\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2010\/07\/07\/the-problems-with-surveys-for-user-experience-tests\/\",\"url\":\"https:\/\/www.stevebromley.com\/blog\/2010\/07\/07\/the-problems-with-surveys-for-user-experience-tests\/\",\"name\":\"The Problems with Surveys for User Experience Tests\"}}]},{\"@type\":\"Article\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2010\/07\/07\/the-problems-with-surveys-for-user-experience-tests\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2010\/07\/07\/the-problems-with-surveys-for-user-experience-tests\/#webpage\"},\"author\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#\/schema\/person\/9c0be0bbd079c086677d422d1fd9c8c7\"},\"headline\":\"The Problems with Surveys for User Experience Tests\",\"datePublished\":\"2010-07-07T07:45:17+00:00\",\"dateModified\":\"2019-11-06T16:45:02+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2010\/07\/07\/the-problems-with-surveys-for-user-experience-tests\/#webpage\"},\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#\/schema\/person\/9c0be0bbd079c086677d422d1fd9c8c7\"},\"image\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2010\/07\/07\/the-problems-with-surveys-for-user-experience-tests\/#primaryimage\"},\"articleSection\":\"User Research\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.stevebromley.com\/blog\/2010\/07\/07\/the-problems-with-surveys-for-user-experience-tests\/#respond\"]}]},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#\/schema\/person\/9c0be0bbd079c086677d422d1fd9c8c7\",\"name\":\"Steve Bromley\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/4dfbbfc5a00187fd6f5fd405361347b2698a65a866f49de07f9486895b6c7029?s=96&d=mm&r=g\",\"caption\":\"Steve Bromley\"},\"logo\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#personlogo\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/posts\/941","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/comments?post=941"}],"version-history":[{"count":8,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/posts\/941\/revisions"}],"predecessor-version":[{"id":2102,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/posts\/941\/revisions\/2102"}],"wp:attachment":[{"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/media?parent=941"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/categories?post=941"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/tags?post=941"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}