Posts Tagged ‘usability’
Test with real users – not your team
‘Free pizza and coke! Just play our game for an hour’. Sounds like a good deal right? And pretty easy to organise, just pulling kids of the street. It can even be done in the pub, for mobile devices. Even this ‘free pizza’ recruitment is better than testing your game (or website, or application) with people from within your office. But why?
Game development teams need a constant supply of fresh users to test the ‘new user’ experience with. I’ve seen teams keep their project secret from their colleagues, not for official reasons, but so their colleagues can be tested as ‘new users’. Other teams test their games with their HR and secretarial staff, since they are unlikely to have had much exposure to the game.
However it’s a good guideline to never test with your team (unless of course you are building something for them). It’s understandable why this situation arises – often budgets are too tight for intensive user testing, forcing teams to perform ad-hoc tests with their colleagues; however this often causes problems further down the line:
such as ...accidents
Dont test with your team
- Your team are not your users – Unless you are in a very specialist field, or are developing an internal project like an intranet, it’s unlikely that your team are the same people as your users. And they are very unlikely to act in the same way a typical user would.
- Your team know things users wouldn’t – It’s likely your team will have had prior exposure to your game or application that a new user wouldn’t, and will be bringing prior knowledge to the testing session. This also applies to people who do not work directly on your team. To get a true outside perspective, you need to seek outside users.
- Your team know you – Unlike a stranger, your team are already know you, and (hopefully) like you. Their answers, and interactions will be biased to please you, and tell you what you want to hear based on what they know about your job, the project your working on, or your beliefs (for example, attempting to validate your design choices).
Advantages of testing with real users.
How people act can often be surprising. If this wasn’t the case, there would be little point in user testing. That’s why it’s extremely important to gather real data, from the people who will actually be using your product. Only real users will approach your product from an authentic ‘new user’ angle, and give an insight into how your product will be perceived and used.
Getting real users involved with product development will get them engaged with the product. Asking their opinions, and being interested in their experience will make the user feel positive about you, and your product, and will mean they will be more likely to purchase it when it’s ready. In newsrooms, this has been widely known for years – hence the proliferation of lists of names in local papers.
Most importantly, involving users will get them talking about your product, generating true grass roots ‘word of mouth’ promotion (hopefully without breaking any NDA’s!). Giving customers an early exposure to your product can build excitement, and market your product for free!
saving millions on megaphones
Conclusion
Finding real users can be cheaper than you think. Not only is it possible to pull people off the street, using the methods suggested above, but new usability testing methods such as remote user testing allow you to find and test real users from the comfort of your office, for very low cost. These days there’s almost no excuse not to test with real users, and it can be just as easy as testing with your team, with much more rewarding returns.
Remote Research – Book Review
Remote Research is a new book by Nate Bolt and Tony Tulathimutte, who have worked with the UX agency Bolt | Peters on a wide range of studies, with clients such as Wikipedia and Electronic Arts (I recommend watching the funny out-takes of Spore user testing).
Their new book sums up their experiences with performing remote research (Tony has previously discussed this subject on this blog, in the comments here), and gives clear instructions on how others can perform a wide range of usability and user experience studies with people who are physically distant, by using the internet.
Don't judge it by it's cover...
Why would you consider remote research?
Written by advocates of remote research, the book highlights many of the potential advantages that remote research gives compared to a more traditional lab based study. These advantages are fleshed out throughout the book through testimonies of experts who have experience in this field, who offer real world examples to emphasise these points.
Some key advantages are:
- Access to a geographically diverse user base. Unlike traditional research, where a moderator would have to be in the same physical location as the subjects, remote research allows a study to be run with anyone who has a high speed internet connection, widely expanding the potential study-group.
- Easy to let stakeholders get involved. Because the research session is being broadcast over the internet, it’s possible to allow stakeholders (i.e. executives and designers) to view the session, and give (moderated) input. This of course increases their engagement with the process, and will be the ‘evidence’ for any conclusions derived from the research.
- Natural browsing environment. The validity of the research can be improved, not only because you are allowing the user to perform the task in a familiar environment (their own home computer), but also some recruitment methods allow you to capture a user performing a task they have selected. For example, recruiting a user who came to the site to buy trousers, for a task based on buying trousers, would provide more accurate results than asking someone to pretend to buy trousers…
- Cheaper (debatably). Not having to pay for travel can keep costs down, however other costs, such as incentives, will still be required, as well as paying for the software.
The remote research book doesn’t advocate killing off lab tests though – instead, it recognises that there are cases when the lab is still appropriate, such as when privacy is a concern. The book also features Andy Budd’s defence of the lab, which argues that remote research fails to pick up aspects of non-verbal behaviour, as well as arguing that remote research doesn’t just remove a selection bias (geography), since it also adds another (internet speed and technical ability). It’s brave of the book to include the case against remote research, and helps project a more trustworthy and reliable image for the book itself.
How to do remote research
The ‘meat’ of the book are the sections dedicated to how-to guides on the different forms of remote research. The book contains step by step instructions on performing moderated or un-moderated research, and includes key topics such as recruitment (and live recruiting), card sorts, and lots of handy hints – such as using IM clients as a chat room for multiple observers to automatically share and timestamp notes.
The book doesn’t just cover basic topics – it goes on to develop novel approaches to user research, such as using ‘reverse screen sharing’ to protect confidential software or data, and using mobile web to gain a new understanding of time-dependant information, outside of the traditional moderated setting.
It also extends the remits of remote research – it doesn’t have to just be websites, but can include doodles or sketches, as well as developing ideas for automatic research with analytics.
Another sort of remote research?
Conclusion
Remote Research is one of the easiest to read UX books I’ve reviewed. Like many Rosenfeld publications, it is laid out well, without appearing dense with text, and has a friendly tone throughout. The book can be likened to Krug’s writing in its style, and presentation.
The book is also practical and realistic, and deals with real world issues, like ‘fakers’ (who can be outed by using open ended questions to discover motives), legal issues, and common challenges such as reluctant stakeholders.
Most importantly for the practical UX practitioner, the book is not dogmatic. This is especially evident in the last chapter which admits that usability shouldn’t be the exclusive goal of product design, and needs to be coupled with initiative, and innovation to develop great things.
Overall this book is a great introduction, and how-to guide to the growing field of remote research, and will be an important tool for anyone trying to keep up to date with the latest research methods.
Games Usability Testing is not QA!
As an emerging field, the aims and characteristics of usability and player experience testing can often be unknown or unrecognised within game companies, outside of a few key industry leaders, such as Valve and Bungie. This can often lead to comparisons being made with QA testing, or confusing usability testing as an element of QA. As an already established area of game development, it’s a common misconception that they are the similar, or the same fields. This is not the case
There are similar elements, as both involve considering the end user’s experience, and involve getting players to physically play the game. However, they have different goals, and this is what I will be covering today, by looking briefly at the aims of QA, the aims of usability testing and how they differ.
What is QA?
Quality Assurance (or Assessment) is an established field within game development. Often performed at both a developer and publisher level, it typically involves a room full of underpaid gamers endlessly playing a game in every conceivable way. They’re looking for bugs, which will be documented and then passed on to the coding or art department.
Typical bugs would be “the hitbox on that model is wrong”, or “when you shoot the tires on that Jaguar E Type, it makes a metal ricochet sound, not a rubber one”. So the task is largely monotonous, and will involve running into every wall in the game, and testing every dialogue choice, to find one which gives an unexpected result. Moving beyond bugs, QA often includes other areas such as localisation, compliance, and compatibility with a range of devices.
...and taste
Throughout the process, QA is trying to make sure the player can’t get themselves stuck, and no bugs in the game prevent them from completing, or enjoying the game.
What is Usability and Player Experience testing?
Similar to QA, usability and player experience testing involves playing the game. However, this typically wouldn’t be in a similar ‘farm’ setting. Instead, usability tests often attempt to recreate a typical playing environment, to emulate how a player would typically play the game (required: a sofa, and a 2 litre bottle of pepsi .)
Player experience is focused on whether a player enjoys what they do in the game, and whether they understand their goals. This can encompass many sub-categories of player experience, such as how is challenge and interest is maintained, why and where players give up, and how they understand the game. Essentially, the aim of player experience testing is to optimise the game so that people want to play the game.
Meanwhile usability testing blurs slightly more with QA, yet has some fundamental differences. Usability testing is focused on whether the game allows the player to achieve their goals, for example do they notice when something in game changes (such as picking up a new item), or do they understand where they have to go to complete the level. This is different to the ‘bugs’ that QA discovers, since these are game features, not ‘mistakes’.
Why do they differ?
Essentially this is the key difference between the two. QA focuses on the unintentional problems with a game, and aims to make the final product as close to the design features documented. A QA success would be a game with no bugs, and with an implementation that matches the designer’s vision.
My vision is money... lots of money
In contrast, usability and player experience testing aims to influence and guide the designer’s vision. By involving players, they gain an insight into how the game will be received by its audience, and help the designer create the reaction they’re after. As such, the focus is on the game’s intentional aspects, not its unintentional bugs.
This leads to different characteristics in the problems found. QA works largely in the edge cases, and tests every hit box, every wall, every enemy and every dialogue choice. Player experience and usability is much more interested in the average experience, to ensure that players ‘get’ it.
We can see a shift in the gaming industries perceptions of player experience and usability testing, having been championed by leaders such as Microsoft and Valve, more companies are starting to recruit user experience professionals. With the rise of free social games which require the player to ‘get’ games quickly, we can see that the field is becoming a key requirement for success, and is becoming a focal point of how games are made. But it’s not QA!
iPad Usability Study
Just a short update this week, sharing some thoughts on the recent Nielsen-Norman report on usability for the iPad. The recently published study was based research from a combination of both expert evaluation and user-testing, and aimed to discover how people interact with the iPad, and what issues typical users would encounter that prevent them from achieving their goals.
An iPad Nano
Jakob Nielsen admits that the study is not as thorough as a typical usability study. However has decided to share it anyway, due to the over-inflated impact that usability studies produced early in a devices’ lifecycle have been seen to have. As an aside, this is an interesting contrary viewpoint to the disadvantages of being first-to-market noted in ‘Inmates’, which argues that being first to market is irrelevant compared to being ‘best’.
The report has some interesting key findings, including that the apps seen on the iPad and the iPhone suffer from the re-emergence of a problem not seen since the early 90’s. Unlike web browsers and desktop software, which has established graphical conventions to highlight buttons and GUI elements, iPhone and iPad software has not implemented standard conventions, such as making a clickable button appear 3D. Hence there is no consistent manner of designating important aspects of the UI, and users just didn’t know what they could click on. Nielsen likens this to the first emergence of graphical interfaces of the early 90s, when anything and everything could be a button.
Its therefore clear that the main recommendation of the study is to standardize common elements, like navigation, among first and third party applications, such as “swipe to turn page”, or “press and hold to delete”. This also links with the studies’ findings that users were unsure what reaction their action would cause, as the apps have yet to find a consistent manner in which to work.
Not just a big iPhone
The study highlights how the iPad is not just a big iPhone, and different usability issues emerged on the larger device – most pertinent is that navigation elements on the bottom of the page, as seen in many iPhone applications, will not work on the iPad. The larger screen means that these elements are too far from the user’s field of vision to be noticed – and hence are not appropriate. What this means for people who make apps is that a custom iPad version is needed, not just relying on the ‘universal’ up scaling of iPhone apps.
The full report is linked below, and worth a look if you’re interested in the usability, the iPad, or designing an application!
Read the full report, here.
100 Rogues – Playability Heuristics Review
In the words of the Fast Show, this week I’ve mostly been playing 100 Rogues. I’ve never previously been able to get into Rogue-likes, having only played games in this genre briefly, before being scared off by the dungeon crawler’s core mechanics of ‘odds stacked against you’, ‘if you die you lose’, and ‘you will die’. However, 100 Rogues aims to present an accessible Rogue-like, ideal for someone who hasn’t played before. As accessibility is one of their key design goals, a heuristic based playability review seems appropriate.
This review aims to evaluate the playability of the game, including pertinent usability issues, and the effect this has on player experience. This is especially important, given 100 Rogues’ mission of making a difficult genre accessible. I’ve based the review on the heuristic points identified by Heather Desurvire’s paper ‘Using Heuristics to evaluate the playability of games’
What this review doesn’t cover is non-usability or player experience issues. Hence, it’s not a review of the game itself (which I gather is a pretty standard Rogue-like). It’s also not QA, and so doesn’t cover bugs in the game. This is especially important as 100 Rogues has infamously been plagued with a number of bugs since its release last week. The first version would crash if the player equipped a shield. The fix for this introduced several new bugs. And I still haven’t been able to get defeat the first boss without the game killing my player after I’ve defeated the boss. I gather the developers are working on this though.
100 Rogues
Game Play
Game play covers the game’s goals, and player’s involvement in achieving these. 100 Rogues succeeds in many areas here. Crucially, it guides the player through their first steps, and introduces them to the range of actions available to them, by immediately giving them the ability to level up their player, which is reinforced by the low cap for the second level up, allowing the player to practise this mechanic twice within the first 2 minutes of the game.
More complex actions are also introduced to the player, through the game’s challenge mode – a series of short scenarios where only the use of an advanced technique (such as ‘tele-stabbing’) will succeed. These introduce the player to some of the more complex moves available to them, in an intuitive way (rather than just… telling them)
The game handles the inevitable death of your character well, also. As mentioned previously, when playing a Rogue-like, you only have one life, no reprieves, and you will die. They explain this by likening the game to Tetris – the goal with your one life is to see how far you can get, not to reach the ‘end’. However death in any specific situation is never inevitable and the game always seems fair. This isn’t Mario Kart, where the CPU will always cheat at the last moment. Instead, after every death you’ll always believe that you could have done something differently and survived, and hence the game encourages a ‘one more go’ attitude to playing, and challenge comes off as a positive experience.
The only game play element which seems poorly balanced, and takes some of the control away from the player, is the ‘food’ mechanic. If you don’t eat food, your character dies. Makes sense. And sometimes you cannot find anything but rotten food, which will fill you up, but reduce your maximum HP. That’s fine too, if it’s a choice between being weakened, and death. But sometimes the game will spawn no food at all. And then you’re stuck, and nothing you can do will save your character from death. This looks like it needs to be balanced in future games, so the game at least spawns some food (rotten or not) when the player is critically hungry. As it stands, the player doesn’t feel in control of their destiny, and has no ability to prevent their character from dying.
Pictured: 100 Rouges
Story
Story defines how your characters’ actions fit into the world you are presented with and the feeling that the game-space exists as a real world, which you can affect, but which exists without you.
The story in 100 Rogues is simple. Satan is at the bottom of the dungeons, and you must kill him. Why? Because he’s Satan.
This story is introduced by a cut scene at the start of the game, and loading text gives character-related background, however this is where the player’s involvement in the story ends. The in game content, and enemies don’t reflect this final goal (aside from the end boss), and the character development isn’t plot related. Occasional infighting among the enemies gives the potential for further depth within the story, however this is not explored further. Ultimately, like Tetris, the story of ‘why’ you are doing this is not a focal point of the game.
Mechanics
Mechanics covers consistency in how the game reacts, how the AI acts, and how the player controls their character. The AI in the game is a success, with the NPC’s acting consistently, and yet occasionally surprising the player – such as when an archer’s miss-fired arrow will hit another enemy, and they will start battling it out. Hence the game balances allowing the player to understand how NPC’s react, without making them entirely predictable.
The game allows the player to track their own progress, through in game score/stats, and a global high-score table. Although implementation of this isn’t perfect, as I’m not convinced scores are being posted to the internet, the bug has been noted and is being ‘worked on’.
An area where the mechanics of 100 Rogues fails is with its controls. The character is controlled by touching the screen – touch the screen at the top to make the player go up, or touch an enemy to attack them. However, with no confirmation before an action is taken, and no indication of the active areas for each target, I found myself ‘miss-clicking’ numerous times, with often deadly consequences. Hence, when trying to click on an enemy for a ranged attack, I’d instead walk towards them, putting my character in danger. Restricting the range of actions available on a single click, or making ‘attack’ a double click, may help alleviate some of these issues.
Must be my sausage fingers...
Usability
Usability concerns how the game gives feedback for inputting actions, and whether they can achieve their goals. A success for 100 Rogues is how it saves the game state on quit, allowing the game to continue from the last point, as I’ve discussed before
As I discussed above, the game also gives direction to a first time player, by giving them an introduction to levelling up on game start. However, it hinders play the second time you start by… giving the same introduction. Since the game demands multiple play-throughs, I feel that I have grasped this mechanic the 20th time it has been introduced to me.
Conclusion
As has been made clear by a heuristic evaluation of 100 Rogues, the game has a high degree of playability, and provides an accessible entry point to a traditionally difficult genre. As noted, there is room for improvement, yet the game offers significant advances on other games in this genre.
However, I’d be hesitant to recommend the game, as it stands, as a positive player experience. Although, playability shouldn’t include bugs, bugs will undoubtedly have an effect on player experience. Hence, as the game stands, the unexpected crashes and deaths will detract from player’s opinions of the game. What incentive do players have to give the time and effort of playing, when their character could be taken away from them through no fault of their own?
That said, the development team have been dedicated to fixing bugs – having released two patches in the week after the game was released, and are promising up to two-three times more content released periodically, which is an advantage of the iPhone as a platform. Within a few more iterations, I can see this game being the definitive introduction to the Rogue-like genre.
