{"id":326,"date":"2009-11-30T17:47:25","date_gmt":"2009-11-30T16:47:25","guid":{"rendered":"http:\/\/www.stevebromley.com\/blog\/?p=326"},"modified":"2019-11-06T17:45:03","modified_gmt":"2019-11-06T16:45:03","slug":"conducting-an-expert-review","status":"publish","type":"post","link":"https:\/\/www.stevebromley.com\/blog\/2009\/11\/30\/conducting-an-expert-review\/","title":{"rendered":"Conducting an Expert Review"},"content":{"rendered":"<p>Within our HCI classes, we have started reviewing the UX of an upcoming multi-platform game from a prominent client, and are performing an expert review on it.\u00a0 An expert review, as opposed to a user-based study, involves having usability experts play the game themselves, and uses tools and their expertise to find faults. This is different to a user-based study, where the expert would observe another player playing the game. Because of the time constraints involved, we selected an expert review as the most effective method to review the UX of this game.<\/p>\n<p>To get the best results possible, and be as helpful as possible to the client, we had to choose our methodology carefully. In this blog post, I\u2019ll discuss how we chose to approach this task, why we chose these methods, and what the alternatives are.<\/p>\n<p>The first rule placed on us is that we are to work in groups of 3. As described in an article by Laitinen on performing expert evaluations, the evaluation reaches its optimal group size between 3 and 4. Less experts than this may miss things. More experts than this fail to find a significantly larger number of faults.<\/p>\n<div align=\"center\">\n<div id=\"attachment_327\" style=\"width: 340px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-327\" src=\"http:\/\/www.stevebromley.com\/blog\/wp-content\/uploads\/2009\/11\/too-many-cooks-spoil-the-br.png\" alt=\"plus too many cooks spoil the broth\" title=\"too-many-cooks-spoil-the-br\" width=\"330\" height=\"345\" class=\"size-full wp-image-327\" srcset=\"https:\/\/www.stevebromley.com\/blog\/wp-content\/uploads\/2009\/11\/too-many-cooks-spoil-the-br.png 330w, https:\/\/www.stevebromley.com\/blog\/wp-content\/uploads\/2009\/11\/too-many-cooks-spoil-the-br-286x300.png 286w\" sizes=\"auto, (max-width: 330px) 100vw, 330px\" \/><p id=\"caption-attachment-327\" class=\"wp-caption-text\">plus too many cooks spoil the broth<\/p><\/div>\n<\/div>\n<p>The other restraint placed upon us is that we would only have a short amount of time with the game. We decided to use this time to play and evaluate the games separately, and then come together to discuss our findings. The alternatives to this would have been having one person play, and the other two take notes, or to have each person play for a bit (as we did), but the experts not playing would take notes then. All of these sessions would involve filming the game screen, and the participant.<\/p>\n<p><strong>Two experts watching one player<\/strong><\/p>\n<p>Advantages:<\/p>\n<ul>\n<li>One      longer complete play through, so can see player development<\/li>\n<li>Experts      can ask the player questions during their play session<\/li>\n<\/ul>\n<p>Disadvantages:<\/p>\n<ul>\n<li>Only      one play through, so difficult to see if issues are common or just for      this user<\/li>\n<li>Questions      asked during play through may distract\/alter playing experience<\/li>\n<\/ul>\n<p><strong>Three experts playing together, in turns<\/strong><\/p>\n<p>Advantages:<\/p>\n<ul>\n<li>Three      sessions played through, so can see reoccurring issues<\/li>\n<li>Experts      can get a greater understanding of the game mechanics through playing it<\/li>\n<\/ul>\n<p>Disadvantages:<\/p>\n<ul>\n<li>Players      wouldn\u2019t get as far as they would with a long session from one player<\/li>\n<li>Second      and third experts play experience will be biased from the experience of      the first<\/li>\n<\/ul>\n<p><strong>Three experts playing separately<\/strong><\/p>\n<p>Advantages:<\/p>\n<ul>\n<li>Each      player gets an authentic \u2018new player\u2019 experience<\/li>\n<li>Comparing      after can show what issues naturally arose for all<\/li>\n<\/ul>\n<p>Disadvantages:<\/p>\n<ul>\n<li>Players      wouldn\u2019t get as far as in one long play through<\/li>\n<li>Have      to perform expert evaluation after the game play, rather than during.<\/li>\n<\/ul>\n<p>Since the sessions were all being recorded, we opted to do the last one, and hence have the \u2018purest\u2019 play experience recorded for each. \u00a0There is, of course, no right answer \u2013 many other groups chose different approaches, and I\u2019m sure they found equally valid issues. I\u2019d welcome comments below if anyone has reasons for a preference with how to perform an expert evaluation.<\/p>\n<p>Now having a video of a play test, we are individually analyzing them. I\u2019m approaching it using heuristics, such as those made by <a href=\"http:\/\/www.useit.com\/papers\/heuristic\/heuristic_list.html\">Nielsen<\/a>, <a href=\"http:\/\/library.forum.nokia.com\/index.jsp?topic=\/Design_and_User_Experience_Library\/GUID-E012A8B2-B705-475A-A2CD-92BDFE146CDA.html\">Nokia<\/a>, and the work of <a href=\"http:\/\/melissafederoff.com\/\">Federoff<\/a> as a guide. Having identified the issues, I will then attempt to rate them by severity \u2013 the extent to which they will hinder the user\u2019s enjoyment of the game. Then, in a group session with my team members, we will evaluate which issues we all agreed where particularly prominent and severe, and amalgamate our results, ending up with a list of issues with the game.<\/p>\n<p>We will then have to present our data to the client. I posted before about <a href=\"http:\/\/www.stevebromley.com\/blog\/2009\/11\/17\/how-to-present-a-ux-report\/\">writing a UX report<\/a>, but the circumstances for this report will differ \u2013 Geographical location, and time constraints mean that this report will be an in-person presentation, with some take-aways. I will blog about these soon\u2026.!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Within our HCI classes, we have started reviewing the UX of an upcoming multi-platform game from a prominent client, and are performing an expert review on it.\u00a0 An expert review, as opposed to a user-based study, involves having usability experts play the game themselves, and uses tools and their expertise to find faults. This is [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[118],"tags":[],"class_list":["post-326","post","type-post","status-publish","format-standard","hentry","category-user-research","grve-entry-item","grve-blog-item"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v15.0 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Conducting an Expert Review - Steve Bromley - User Research<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/30\/conducting-an-expert-review\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Conducting an Expert Review - Steve Bromley - User Research\" \/>\n<meta property=\"og:description\" content=\"Within our HCI classes, we have started reviewing the UX of an upcoming multi-platform game from a prominent client, and are performing an expert review on it.\u00a0 An expert review, as opposed to a user-based study, involves having usability experts play the game themselves, and uses tools and their expertise to find faults. This is [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/30\/conducting-an-expert-review\/\" \/>\n<meta property=\"og:site_name\" content=\"Steve Bromley - User Research\" \/>\n<meta property=\"article:published_time\" content=\"2009-11-30T16:47:25+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-11-06T16:45:03+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/www.stevebromley.com\/blog\/wp-content\/uploads\/2009\/11\/too-many-cooks-spoil-the-br.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#website\",\"url\":\"https:\/\/www.stevebromley.com\/blog\/\",\"name\":\"Steve Bromley - User Research\",\"description\":\"Usability and User Research for Websites, Software and Games\",\"publisher\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#\/schema\/person\/9c0be0bbd079c086677d422d1fd9c8c7\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/www.stevebromley.com\/blog\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/30\/conducting-an-expert-review\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"http:\/\/www.stevebromley.com\/blog\/wp-content\/uploads\/2009\/11\/too-many-cooks-spoil-the-br.png\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/30\/conducting-an-expert-review\/#webpage\",\"url\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/30\/conducting-an-expert-review\/\",\"name\":\"Conducting an Expert Review - Steve Bromley - User Research\",\"isPartOf\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/30\/conducting-an-expert-review\/#primaryimage\"},\"datePublished\":\"2009-11-30T16:47:25+00:00\",\"dateModified\":\"2019-11-06T16:45:03+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/30\/conducting-an-expert-review\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/30\/conducting-an-expert-review\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/30\/conducting-an-expert-review\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/\",\"url\":\"https:\/\/www.stevebromley.com\/blog\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/30\/conducting-an-expert-review\/\",\"url\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/30\/conducting-an-expert-review\/\",\"name\":\"Conducting an Expert Review\"}}]},{\"@type\":\"Article\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/30\/conducting-an-expert-review\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/30\/conducting-an-expert-review\/#webpage\"},\"author\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#\/schema\/person\/9c0be0bbd079c086677d422d1fd9c8c7\"},\"headline\":\"Conducting an Expert Review\",\"datePublished\":\"2009-11-30T16:47:25+00:00\",\"dateModified\":\"2019-11-06T16:45:03+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/30\/conducting-an-expert-review\/#webpage\"},\"commentCount\":2,\"publisher\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#\/schema\/person\/9c0be0bbd079c086677d422d1fd9c8c7\"},\"image\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/30\/conducting-an-expert-review\/#primaryimage\"},\"articleSection\":\"User Research\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.stevebromley.com\/blog\/2009\/11\/30\/conducting-an-expert-review\/#respond\"]}]},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#\/schema\/person\/9c0be0bbd079c086677d422d1fd9c8c7\",\"name\":\"Steve Bromley\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/4dfbbfc5a00187fd6f5fd405361347b2698a65a866f49de07f9486895b6c7029?s=96&d=mm&r=g\",\"caption\":\"Steve Bromley\"},\"logo\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#personlogo\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/posts\/326","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/comments?post=326"}],"version-history":[{"count":4,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/posts\/326\/revisions"}],"predecessor-version":[{"id":633,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/posts\/326\/revisions\/633"}],"wp:attachment":[{"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/media?parent=326"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/categories?post=326"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/tags?post=326"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}