{"id":2069,"date":"2016-06-03T19:07:59","date_gmt":"2016-06-03T18:07:59","guid":{"rendered":"http:\/\/www.stevebromley.com\/blog\/?p=2069"},"modified":"2019-11-06T17:44:57","modified_gmt":"2019-11-06T16:44:57","slug":"running-user-tests-for-virtual-reality","status":"publish","type":"post","link":"https:\/\/www.stevebromley.com\/blog\/2016\/06\/03\/running-user-tests-for-virtual-reality\/","title":{"rendered":"Running User Tests for Virtual Reality"},"content":{"rendered":"

I recently gave a talk at UXBrighton<\/a> about running user tests for virtual reality. The event was a really fun evening, with some great talks from Deborah Amar and Henry Ryder, and was organised by Harvey<\/a> from Player Research.<\/p>\n

In my talk, I covered some tips that I’d learned from running user research for virtual reality hardware, and many virtual reality experiences. I’ve copied the slides and my notes below for your viewing pleasure. Enjoy!
\n<\/p>\n

\n

\"slide1\"
\nOk, so here we go….<\/p>\n

\n

\"slide2\"
\nFor the last few years, I’ve been working a lot on VR, including running research on the hardware itself, the system software and the launch games for the PlayStation VR headset…<\/p>\n

\n

\"slide3\"
\n…but who cares about that. Today I’ll be sharing some things I’ve learned during running these sessions, in particular for 1:1 moderated lab-based sessions. Some of the points we come across may seem obvious in hindsight, but hopefully they will save you wasting a participant, a research session, or worse by encountering easy to overcome issues.<\/p>\n

\n

\"slide4\"
\nThe first area I wanted to talk about was how to set up the room for VR, which needs some extra thought about it compared to \u2018traditional\u2019 games user research.<\/p>\n

\n

\"slide5\"
\nNormally the set-up of the room isn\u2019t particularly important. This is a picture of our social room, a set up that most games user research labs also have.<\/p>\n

It has a TV, some couches, and is just meant to represent a typical living room set-up. As long as you have the participant, and a TV they can play on, that\u2019s good enough to get good quality usability feedback from.<\/p>\n

\n

\"slide6\"
\nHowever getting the room right is much more important for VR tests, because getting it wrong can cause plenty of issues with the game.<\/p>\n

The first thing to think about is where to sit the player. Both Oculus and PSVR rely on line of sight between the camera and the headset \u2013 they use infra-red so that the game can track the player\u2019s position, and allow them to move their head and body forward\/back in the virtual environment, something not possible on stand-alone systems like GearVR.<\/p>\n

Because of this, there\u2019s an optimal place for the player to sit. If they are too far forward, or too far back, the system won\u2019t be able to see them, and tracking will fail causing other issues in the game.<\/p>\n

Be especially aware if the game will require movement \u2013 some of the demos that PlayStation have publicly shown require players to move between sitting and standing \u2013 so make sure that the player is sat somewhere the camera can see them both sitting and standing. <\/p>\n

If the player is in a poor position, you\u2019ll have plenty of potential usability issues \u2013 e.g. whether players recognise that the tracking is failing, and whether they know how to deal with it. However this is a system level feature, and so not the responsibility of the game you are testing \u2013 so development teams who have put the time and expense into organising the test will not be particularly happy if this occurs during their test, and obscures other potential usability issues in the game.<\/p>\n

So, before starting, ensure the player is in the optimal position to play the game \u2013 which can be particularly an issue in multi-seat set-ups where many playtests are run simultaneously in the same room\u2026 but more of that soon.<\/p>\n

\n

\"slide7\"
\nAnother common issue with usability labs is that they often have 1 way mirrors. Returning to the test we saw earlier ,we can see from the other angle that there is a one-way mirror in the room, which allows a note taker (and members of the development team) to observe the session. This is usually great \u2013 it gives a clear view of the context of the issue, and inviting the game team to come and see the session live allows the development team to come and engage with the session \u2013 paying more attention, and understanding the issues greater than watching later on a video.<\/p>\n

However because many VR headsets use the IR sensors, the one way mirror can be a potential issue. It\u2019ll cause a duplication of the IR sources for the camera, which can cause the quality of positional tracking to degrade, again introducing interference and reducing the quality of the experience, which can impact usability. <\/p>\n

Luckily there\u2019s an easy solution to this. Our lab manager discovered that on Amazon they sell blackout blinds, which are bits of plastic that stick, like cling film, to glass. This takes seconds to attach and can blackout the window preventing the reflection of the IR lights. The development team will once again have to rely on the camera feed, but the players experience is more authentic.<\/p>\n

\n

\"slide8\"
\nAnother thing to be aware of is swivelling chairs. These are very common in an office environment, and so can be found in some playtest labs.<\/p>\n

However they are much less common in the home environment, particularly with home consoles, so are unrepresentative. <\/p>\n

The issue with swivel chairs is that they cause players to lose their orientation, and \u2018forget\u2019 which way is forward (or which way the camera is). This can cause usability issues, but again they are unrealistic because players would be on a sofa at home, where the forward direction is much clearer just from the feel of it.<\/p>\n

They also have wheels on, which can cause them to move, and be a safety issue when working with players who cannot see! <\/p>\n

\n

\"slide9\"
\nThe last thing to mention is in regards to room scale VR. HTC Vive is built to allow players to walk around each experience physically. Oculus and PSVR have also both shown equivalent demos that allow players to physically move in a space. <\/p>\n

For research on room scale VR \u2013 or anything that will encourage the players to move around – you want to keep the room free of obstacles (because they are walking around blindfolded in a room they are not familiar with). <\/p>\n

Also remember not to move things \u2013 if you move a controller, or more importantly a chair, players will not be aware that it\u2019s moved, which can cause safety issues.<\/p>\n

\n

\"slide10\"
\nSo, to summarise this section on the room setup \u2013 extra care needs to be taken to ensure that the room setup isn’t going to cause \u2018unfair\u2019 usability issues that are out of scope for your current research.<\/p>\n

Also remember to keep the user safe \u2013 don\u2019t move things or let them walk into stuff!<\/p>\n

\n

\"slide11\"
\nThe next thing I want to talk about is methodology \u2013 how the test needs to be adapted to run good quality user research sessions.<\/p>\n

\n

\"slide12\"
\nSo, my favourite type of user research is 1:1 qualitative sessions. We usually have a moderator in the room with the participant, who can help ensure the game is running correctly and interview them at the end. We also have a note-taker behind the glass who can observe the usability issues, and write down players behaviours and what they say. This all works fine for VR tests.<\/p>\n

One of my favourite parts of these sessions is probing players to understanding why players are acting in a specific way. For example, you can observe that the player walked straight past the next objective. But without probing, you cannot tell why \u2013 did they not see it? Did they want to collect something else first? Did they think they *needed* to collect something else first? The challenge is asking questions in the right way so that players are not lead to give the answer that they think you expect, so it\u2019s like a puzzle, which is why I enjoy it. However, with VR there\u2019s a problem with this\u2026<\/p>\n

\n

\"slide13\"
\n…You can\u2019t talk to the players as they play.<\/p>\n

With the headset on, they can\u2019t see you. These headsets also have 3D audio (so they can tell the direction of sounds), which requires headphones to work. Meaning they can\u2019t hear you either.<\/p>\n

This means that asking probing questions isn\u2019t easy. There\u2019s a couple of ways to address this.<\/p>\n