{"id":1971,"date":"2015-03-19T16:48:53","date_gmt":"2015-03-19T15:48:53","guid":{"rendered":"http:\/\/www.stevebromley.com\/blog\/?p=1971"},"modified":"2019-11-06T17:44:57","modified_gmt":"2019-11-06T16:44:57","slug":"some-things-ive-learned-about-observing-playtesting-sessions","status":"publish","type":"post","link":"https:\/\/www.stevebromley.com\/blog\/2015\/03\/19\/some-things-ive-learned-about-observing-playtesting-sessions\/","title":{"rendered":"Some things I’ve learned about observing playtesting sessions"},"content":{"rendered":"

Being the observer is often seen as the less glamorous side of usability testing. While the moderator gets to meet and interact directly with the player, the observer is trapped behind the one-way mirror. However it is actually the more important of the two roles \u2013 the observer is the one who is identifying and recording the usability issues that occur during tests.<\/p>\n

To be a good observer, you need to know the game\u2019s code inside out, and be able to identify issues (as the moderator may not be in contact to highlight them for you). You also need to handle this while looking after the development team, who are likely to be watching the test next to you.<\/p>\n

These are some techniques I\u2019ve picked up from other researchers at PlayStation, which I feel have improved my own skills as an observer of user tests. This post follows on from my previous post about moderating one-to-one user test sessions<\/a>, and aims to be tips that may not seem obvious at first, but greatly improved my own note-taking once I started implementing them.<\/p>\n

Timestamping Notes<\/strong><\/p>\n

Assuming that you don\u2019t have a fancy video system that automatically associates your notes with the video, it comes down to the observer to keep track of when issues occurred. Mirweis Sangin<\/a> has been a great proponent of doing this on our team, and the benefits have been huge.<\/p>\n

How to time stamp notes is simple:<\/p>\n

Start a timer as soon as recording starts at the beginning of the session. Then, when issues occur, add to your note the time that it occurred, from the timer. It doesn\u2019t matter if your time isn\u2019t exact, or you don\u2019t think of it until 20 seconds later, just write down what you have.<\/p>\n

It\u2019s an easy thing to do, but it will save you a huge amount of time later during the analysis phase when you need to review occurrences of the usability issues. The time stamps make finding the issue again much easier on the video than scrubbing through hours of footage, and so it becomes simple to find screenshots or videos for the report or a highlight reel.<\/p>\n

It can also help re-cap issues when the notes are unclear, or if you need more information to explain what happened (for example, you\u2019d missed what the outcome of the issue had been). And on that subject\u2026<\/p>\n

 <\/p>\n

Always write what happened at the end<\/strong><\/p>\n

Our team uses an adaption of userfocus\u2019 method for prioritising usability issues (http:\/\/www.userfocus.co.uk\/articles\/prioritise.html<\/a> ). In order to prioritise reliably and consistently against this model, we have to capture what the outcome of every issue is.<\/p>\n

If you had just written an issue like \u201cuser had difficulty finding the next objective\u201d, this would be useless to you when looking to determine the impact a few days later. Did they find it? How long did it take? Did they backtrack? Did they need help? Where did they look to see the next objective? Why?<\/p>\n

With all issues, remember to capture the outcome, such as \u201chow long did it take to resolve\u201d, \u201cdid the moderator have to help them?\u201d, \u201cwhat prompted them towards overcoming the issue\u201d. This sort of detail is essential for the team to be able to make the correct fix, and recording this information now will save you a lot of time, help you prioritise the issues later, and ensure that you don\u2019t have to spend days re-watching the videos!<\/p>\n

 <\/p>\n

Ignore (some) opinions<\/strong><\/p>\n

Most qualitative studies do not have enough users to find reliable opinion data. Instead questions about what people thought of the game should be saved for a quantitative study.<\/p>\n

When a user does express an opinion during a one-to-one study, it should be captured it if it\u2019s a result of a usability issue (e.g. \u201cI\u2019m annoyed because I can\u2019t find where to go\u201d). However if it\u2019s an opinion about a subjective thing (\u201cI don\u2019t like how it looks\u201d), the observer can ignore it.<\/p>\n

When opinions do arise, a good moderator should reply to opinions with a probing statement like \u201cwhy do you feel that?\u201d to identify if a usability issue is causing the opinion, which will help increase the quality of the notes taken.<\/p>\n

Opinions can be great for a route into usability issues, or be used to help illustrate the effects of usability issues in the final report. However on small sample sizes, they are unreliable when reported directly, as they may create a misleading impression of the game\u2019s state.<\/p>\n

 <\/p>\n

Aggregate while the test is running<\/strong><\/p>\n

We use a mind mapping software to capture our notes during the test. Not only does this allow issues to be arranged spatially, and navigated quickly, but it allows quick and simple aggregation of issues.<\/p>\n

Before the test begins, create logical areas in which the issues will fall in the mindmap. This can be level by level, or a list of the core mechanics, usually this is obvious depending on what type of game it is and the objectives of the test.<\/p>\n

The first time an issue occurs during a session, navigate to the relevant section in the mindmap, write the issue and tag it with the user\u2019s number. After the second occurrence of an issue, instead of writing it out again, just tag the original occurrence with this second user\u2019s number, and add any extra details as a sub point.<\/p>\n

By aggregating and clustering the notes during the test, the analysis time is greatly reduced. This will mean the report will be ready quicker, and the results can be shared with the team sooner, which is crucial in the fast-moving environment of games development.<\/p>\n

 <\/p>\n

The observers role during the test of identifying and capturing usability issues is often more difficult than the moderator\u2019s role, as they need to identify and capture the usability issues on the fly. These tips have helped me improve my own skills note-taking during a user test, and I hope they will help you too!<\/p>\n","protected":false},"excerpt":{"rendered":"

Being the observer is often seen as the less glamorous side of usability testing. While the moderator gets to meet and interact directly with the player, the observer is trapped behind the one-way mirror. However it is actually the more important of the two roles \u2013 the observer is the one who is identifying and […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[117],"tags":[62],"class_list":["post-1971","post","type-post","status-publish","format-standard","hentry","category-games-user-research","tag-best-practise","grve-entry-item","grve-blog-item"],"yoast_head":"\nSome things I've learned about observing playtesting sessions - Steve Bromley - User Research<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.stevebromley.com\/blog\/2015\/03\/19\/some-things-ive-learned-about-observing-playtesting-sessions\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Some things I've learned about observing playtesting sessions - Steve Bromley - User Research\" \/>\n<meta property=\"og:description\" content=\"Being the observer is often seen as the less glamorous side of usability testing. While the moderator gets to meet and interact directly with the player, the observer is trapped behind the one-way mirror. However it is actually the more important of the two roles \u2013 the observer is the one who is identifying and […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.stevebromley.com\/blog\/2015\/03\/19\/some-things-ive-learned-about-observing-playtesting-sessions\/\" \/>\n<meta property=\"og:site_name\" content=\"Steve Bromley - User Research\" \/>\n<meta property=\"article:published_time\" content=\"2015-03-19T15:48:53+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-11-06T16:44:57+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#website\",\"url\":\"https:\/\/www.stevebromley.com\/blog\/\",\"name\":\"Steve Bromley - User Research\",\"description\":\"Usability and User Research for Websites, Software and Games\",\"publisher\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#\/schema\/person\/9c0be0bbd079c086677d422d1fd9c8c7\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/www.stevebromley.com\/blog\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2015\/03\/19\/some-things-ive-learned-about-observing-playtesting-sessions\/#webpage\",\"url\":\"https:\/\/www.stevebromley.com\/blog\/2015\/03\/19\/some-things-ive-learned-about-observing-playtesting-sessions\/\",\"name\":\"Some things I've learned about observing playtesting sessions - Steve Bromley - User Research\",\"isPartOf\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#website\"},\"datePublished\":\"2015-03-19T15:48:53+00:00\",\"dateModified\":\"2019-11-06T16:44:57+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2015\/03\/19\/some-things-ive-learned-about-observing-playtesting-sessions\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.stevebromley.com\/blog\/2015\/03\/19\/some-things-ive-learned-about-observing-playtesting-sessions\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2015\/03\/19\/some-things-ive-learned-about-observing-playtesting-sessions\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/\",\"url\":\"https:\/\/www.stevebromley.com\/blog\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2015\/03\/19\/some-things-ive-learned-about-observing-playtesting-sessions\/\",\"url\":\"https:\/\/www.stevebromley.com\/blog\/2015\/03\/19\/some-things-ive-learned-about-observing-playtesting-sessions\/\",\"name\":\"Some things I’ve learned about observing playtesting sessions\"}}]},{\"@type\":\"Article\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2015\/03\/19\/some-things-ive-learned-about-observing-playtesting-sessions\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2015\/03\/19\/some-things-ive-learned-about-observing-playtesting-sessions\/#webpage\"},\"author\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#\/schema\/person\/9c0be0bbd079c086677d422d1fd9c8c7\"},\"headline\":\"Some things I’ve learned about observing playtesting sessions\",\"datePublished\":\"2015-03-19T15:48:53+00:00\",\"dateModified\":\"2019-11-06T16:44:57+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/2015\/03\/19\/some-things-ive-learned-about-observing-playtesting-sessions\/#webpage\"},\"commentCount\":1,\"publisher\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#\/schema\/person\/9c0be0bbd079c086677d422d1fd9c8c7\"},\"keywords\":\"best practise\",\"articleSection\":\"Games User Research\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.stevebromley.com\/blog\/2015\/03\/19\/some-things-ive-learned-about-observing-playtesting-sessions\/#respond\"]}]},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#\/schema\/person\/9c0be0bbd079c086677d422d1fd9c8c7\",\"name\":\"Steve Bromley\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/4dfbbfc5a00187fd6f5fd405361347b2698a65a866f49de07f9486895b6c7029?s=96&d=mm&r=g\",\"caption\":\"Steve Bromley\"},\"logo\":{\"@id\":\"https:\/\/www.stevebromley.com\/blog\/#personlogo\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/posts\/1971","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/comments?post=1971"}],"version-history":[{"count":6,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/posts\/1971\/revisions"}],"predecessor-version":[{"id":1978,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/posts\/1971\/revisions\/1978"}],"wp:attachment":[{"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/media?parent=1971"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/categories?post=1971"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.stevebromley.com\/blog\/wp-json\/wp\/v2\/tags?post=1971"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}