Remote Research – Book Review
Remote Research is a new book by Nate Bolt and Tony Tulathimutte, who have worked with the UX agency Bolt | Peters on a wide range of studies, with clients such as Wikipedia and Electronic Arts (I recommend watching the funny out-takes of Spore user testing).
Their new book sums up their experiences with performing remote research (Tony has previously discussed this subject on this blog, in the comments here), and gives clear instructions on how others can perform a wide range of usability and user experience studies with people who are physically distant, by using the internet.
Don't judge it by it's cover...
Why would you consider remote research?
Written by advocates of remote research, the book highlights many of the potential advantages that remote research gives compared to a more traditional lab based study. These advantages are fleshed out throughout the book through testimonies of experts who have experience in this field, who offer real world examples to emphasise these points.
Some key advantages are:
- Access to a geographically diverse user base. Unlike traditional research, where a moderator would have to be in the same physical location as the subjects, remote research allows a study to be run with anyone who has a high speed internet connection, widely expanding the potential study-group.
- Easy to let stakeholders get involved. Because the research session is being broadcast over the internet, it’s possible to allow stakeholders (i.e. executives and designers) to view the session, and give (moderated) input. This of course increases their engagement with the process, and will be the ‘evidence’ for any conclusions derived from the research.
- Natural browsing environment. The validity of the research can be improved, not only because you are allowing the user to perform the task in a familiar environment (their own home computer), but also some recruitment methods allow you to capture a user performing a task they have selected. For example, recruiting a user who came to the site to buy trousers, for a task based on buying trousers, would provide more accurate results than asking someone to pretend to buy trousers…
- Cheaper (debatably). Not having to pay for travel can keep costs down, however other costs, such as incentives, will still be required, as well as paying for the software.
The remote research book doesn’t advocate killing off lab tests though – instead, it recognises that there are cases when the lab is still appropriate, such as when privacy is a concern. The book also features Andy Budd’s defence of the lab, which argues that remote research fails to pick up aspects of non-verbal behaviour, as well as arguing that remote research doesn’t just remove a selection bias (geography), since it also adds another (internet speed and technical ability). It’s brave of the book to include the case against remote research, and helps project a more trustworthy and reliable image for the book itself.
How to do remote research
The ‘meat’ of the book are the sections dedicated to how-to guides on the different forms of remote research. The book contains step by step instructions on performing moderated or un-moderated research, and includes key topics such as recruitment (and live recruiting), card sorts, and lots of handy hints – such as using IM clients as a chat room for multiple observers to automatically share and timestamp notes.
The book doesn’t just cover basic topics – it goes on to develop novel approaches to user research, such as using ‘reverse screen sharing’ to protect confidential software or data, and using mobile web to gain a new understanding of time-dependant information, outside of the traditional moderated setting.
It also extends the remits of remote research – it doesn’t have to just be websites, but can include doodles or sketches, as well as developing ideas for automatic research with analytics.
Another sort of remote research?
Conclusion
Remote Research is one of the easiest to read UX books I’ve reviewed. Like many Rosenfeld publications, it is laid out well, without appearing dense with text, and has a friendly tone throughout. The book can be likened to Krug’s writing in its style, and presentation.
The book is also practical and realistic, and deals with real world issues, like ‘fakers’ (who can be outed by using open ended questions to discover motives), legal issues, and common challenges such as reluctant stakeholders.
Most importantly for the practical UX practitioner, the book is not dogmatic. This is especially evident in the last chapter which admits that usability shouldn’t be the exclusive goal of product design, and needs to be coupled with initiative, and innovation to develop great things.
Overall this book is a great introduction, and how-to guide to the growing field of remote research, and will be an important tool for anyone trying to keep up to date with the latest research methods.
User Experience or Player Experience?
The common job title for the role of understanding, and optimising how people feel when playing games or using software is ‘User Experience Designer’. Although factually accurate, I believe that this term is neither appropriate, nor flattering for designing experiences, particularly for games. Instead, I prefer the term ‘player experience’.
Read on to discover why! (…and read even further on to find the comments section, and tell me why I’m wrong)
Why ‘User Experience’
‘Experience’ is simple enough – the object is to design how the person who uses your product feels when they use it – essentially, how they experience it. Simples.
My kind of experience
The aspect of the title that I feel isn’t accurate, is ‘User’. The ‘User’ in User Experience is rooted in traditional computing terminology, derived from authentication requirements, like a log-in, to access directories and applications. The computer’s ability to have multiple people access the same system, and hence create a multi-user environment solidified the term, and the effects of the internet, and hence a worldwide network of authentication has made the term common place.
As such, you can see why the generic term to call someone who uses a computer has become ‘user’. And, even when playing games, this term is still currently used. But maybe it shouldn’t…
Why not?
Why don’t I like the term ‘user experience’ for games? More so than with the web, or applications, gaming suffers from the association with the other main connotation of ‘user’ – drug use, and addiction.
Unfairly, computer games have often been compared to addictions and drug use (and not the nice drugs either!). The press often cover stories such as “Gaming as addictive as cocaine”, “Addicted to Xbox”, or Gaming Addiction Clinics.
I hated family holidays...
And everyone’s heard about the people who live in World of Warcraft, ordering food through its pizza delivery service, and eternally grinding, letting their ‘real’ life fall to pieces. Social games in particular, through excessive positive reinforcement and social competition, aim to get player’s ‘hooked’, and keep on playing. So, you can see why gaming comes off unfairly when compared to drug use, through the term ‘user’.
I say unfairly as, like most things, gaming is psychologically addictive. In that, if you like doing it, you’ll do it again. But then, so are all the fun things you enjoy, like petting the cat, or reading a book. Addiction is doing these things to excess, and it’s the excess that’s bad, not the activity. Hard drugs on the other hand are chemically addictive. Which is completely different and creates dependence.
Instead…
So, to avoid these comparisons with the other types of users, I prefer the term ‘Player Experience’. Not only does this remove the unsightly comparison, but it’s more accurate than the term user for playing games.
Computer Gaming has little to do with authentication, and exists independent from the platform – as an activity it is closer to games than computing. The designer is crafting the game to change the player’s experience, rather than crafting the computer, and hence the term should reflect gaming’s prominence in this relationship.
Hence I believe that ‘Player Experience’ is a more accurate, and nice, term for describing what is being designed within games.
(I also have a vested interest in the term as I’m on the first page of Google for ‘player experience’, and miles down the list for ‘User Experience’ – that said, more people come to this blog having searched for User Experience.)
However, for the web, and applications, I’m not so sure. Obviously ‘Player Experience’ doesn’t fit. And does computing in general suffer from the negative public image with regards to addiction that gaming does? If not, maybe the term shouldn’t be changed.
Other alternatives, for software, could be ‘Customer Experience’ – however this is rather corporate, and not universally applicable, or simply using ‘Experience Design’, and dropping the user. I would be interested in hearing your thoughts, or alternative terms, or if you prefer ‘User Experience’ as a term. Comment, or add me on Twitter, and let me know what you think!
Replacing the Desktop?
Just under 40 years ago, the desktop metaphor was devised as a way to allow computer users to understand graphical interactions with their computer. Standard tasks, like using a calculator, or deleting files, were presented in a manner familiar to workers from a traditional office place, as an effort to build their experience of computing upon pre-existing knowledge. And, as is evident by this metaphor’s continued existence today, it was a massive success.
Just under 16 years ago Microsoft attempted to reinvent the desktop metaphor, and bring it up to date. The product, Microsoft Bob, aimed to shift computing from an office metaphor to a home metaphor. And it was a massive failure. My first home computer came with Bob installed, and so today I’ll be looking at why it failed, and what we can learn from this failure.
Usability advantages of the desktop metaphor.
So why has the desktop metaphor proved to be a lasting success? Introduced at a time when graphical methods of interacting with a computer were new, it had several key characteristics that led to its success.
...such as fun games
First of all, the desktop was familiar. Rather than having to learn context specific methods of interacting with a computer, it built upon the user’s pre-existing knowledge. For example, when deleting a file, a user could use their existing understanding of a trash can, and drag the file into it (rather than running a deltree command, which doesn’t map with any real-world knowledge). Therefore the desktop metaphor was easy to figure out, and consistent with real life experience, reducing the learning curve upon adoption. This meets Nielsen’s heuristic on a ‘match between the system and the real world’.
Building upon this familiarity was the appropriateness of the desktop metaphor for the tasks at hand. Before home computing, the workplace was the most likely place for users to use a computer, and the computer would be performing office-based tasks, such as word processing or calculating. Hence the adoption of a workplace metaphor seemed appropriate for developing a graphical user interface, as it registered with the target market. This meets Nielsen’s heuristic on ‘recognition rather than recall’.
Furthermore, the desktop metaphor was wide enough to expand to meet the growing roles that computers played. By extending the workplace metaphor through terms such as ‘cut’ and ‘paste’, and the development of graphical tools emulating image manipulation tasks, the desktop metaphor proved that it wasn’t static, and could extend to reach an ever growing range of requirements.
The desktop metaphor also met the heuristic requirement, of having a wide degree of flexibility, by allowing ‘experienced’ users to automate or speed up tasks, such as by selecting groups of objects, or utilizing keyboard shortcuts.
What did Bob try to do?
In the mid 90’s, Microsoft Bob was devised as the successor to the desktop metaphor. Recognising a growth in home computing, Microsoft aimed to shift the graphical interface model for computing from a business/creative focus, to the ‘home’. It was thought that this would open the computing world to a whole range of ‘novice’ users, who would have found the desktop metaphor inaccessible.
Bob presented the user with ‘their room’, covered with clickable objects, such as bookcases, clocks and a notepad. Clicking these things will launch the relevant program (or help you locate files), and you can add your own programs to the shelves. It’s just like your home! (assuming your home is littered with boxes that say ‘Internet Explorer’ and ‘Corel Draw’).
Bob in action
Why did Bob fail?
Microsoft Bob offered an alternative to using the desktop metaphor, aimed at novice users, but its primary failure was that it didn’t offer any significant advantages. For a product that came over twenty years after the ethos of its competitor, this wasn’t a good sign…
Despite being based on the home, Bob still had a learning curve, and so missed it’s key objective of being intuitive. Clicking on a clock to open a calendar, or a pen and paper, still required just as much learning as a calendar or notepad icon in a traditional desktop environment. More complex tasks than just opening programs still require further learning. Also, the enforced ‘home’ layout is just plain inefficient – rifling through a cabinet to find a file offers no advantage to browsing a list of files in a folder.
By attempting to change the way people interacted with computers, Bob alienated itself from existing computer users, and prevented new users from being able to ask for help from power-users. By offering only ‘simple’ ways of interacting with computers, the user was unable to allow users to grow, and learn superior (and more efficient) ways of performing tasks.
It’s also apparent that the ‘cuteness’ of Bob didn’t sit well with users. The two elements of this operating system which outlived the OS itself are among the most hated villains of computing – Virtual assistants like Clippy, and the Comic Sans font. Obviously Microsoft failed to understand the needs of their target market.
The final nail in Bob’s coffin came within a year of its release. Microsoft released Windows 95. It sold… quite well, and offered a fully-powered alternative to Bob based on the traditional desktop metaphor. Bill Gates punished those responsible for the mess that was Bob. He married lead project manager Melinda French. Burn.
What will replace the desktop?
It’s obvious through Bob’s failure that the Desktop cannot be beaten by a simple re-skin or appropriation of another metaphor without offering significant advantages.
As I wrote about in my review of The Humane Interface, Raskin proposes a ‘Zoomworld’ which offers a non-windows environment with no gaps between the operating system and the files. However development by Archy has stalled and they seem to have fallen off the internet…
Or maybe the future will be more like Google, and involve typing queries or commands into a prompt to find answers and perform tasks? Although this does seem like a regression, and breaks several key usability best practises.
So what other systems are out there that offer a viable alternative? Or will it be desktops forever? As ever, I’d be interested in hearing your thoughts in the comments below!
Games Usability Testing is not QA!
As an emerging field, the aims and characteristics of usability and player experience testing can often be unknown or unrecognised within game companies, outside of a few key industry leaders, such as Valve and Bungie. This can often lead to comparisons being made with QA testing, or confusing usability testing as an element of QA. As an already established area of game development, it’s a common misconception that they are the similar, or the same fields. This is not the case
There are similar elements, as both involve considering the end user’s experience, and involve getting players to physically play the game. However, they have different goals, and this is what I will be covering today, by looking briefly at the aims of QA, the aims of usability testing and how they differ.
What is QA?
Quality Assurance (or Assessment) is an established field within game development. Often performed at both a developer and publisher level, it typically involves a room full of underpaid gamers endlessly playing a game in every conceivable way. They’re looking for bugs, which will be documented and then passed on to the coding or art department.
Typical bugs would be “the hitbox on that model is wrong”, or “when you shoot the tires on that Jaguar E Type, it makes a metal ricochet sound, not a rubber one”. So the task is largely monotonous, and will involve running into every wall in the game, and testing every dialogue choice, to find one which gives an unexpected result. Moving beyond bugs, QA often includes other areas such as localisation, compliance, and compatibility with a range of devices.
...and taste
Throughout the process, QA is trying to make sure the player can’t get themselves stuck, and no bugs in the game prevent them from completing, or enjoying the game.
What is Usability and Player Experience testing?
Similar to QA, usability and player experience testing involves playing the game. However, this typically wouldn’t be in a similar ‘farm’ setting. Instead, usability tests often attempt to recreate a typical playing environment, to emulate how a player would typically play the game (required: a sofa, and a 2 litre bottle of pepsi .)
Player experience is focused on whether a player enjoys what they do in the game, and whether they understand their goals. This can encompass many sub-categories of player experience, such as how is challenge and interest is maintained, why and where players give up, and how they understand the game. Essentially, the aim of player experience testing is to optimise the game so that people want to play the game.
Meanwhile usability testing blurs slightly more with QA, yet has some fundamental differences. Usability testing is focused on whether the game allows the player to achieve their goals, for example do they notice when something in game changes (such as picking up a new item), or do they understand where they have to go to complete the level. This is different to the ‘bugs’ that QA discovers, since these are game features, not ‘mistakes’.
Why do they differ?
Essentially this is the key difference between the two. QA focuses on the unintentional problems with a game, and aims to make the final product as close to the design features documented. A QA success would be a game with no bugs, and with an implementation that matches the designer’s vision.
My vision is money... lots of money
In contrast, usability and player experience testing aims to influence and guide the designer’s vision. By involving players, they gain an insight into how the game will be received by its audience, and help the designer create the reaction they’re after. As such, the focus is on the game’s intentional aspects, not its unintentional bugs.
This leads to different characteristics in the problems found. QA works largely in the edge cases, and tests every hit box, every wall, every enemy and every dialogue choice. Player experience and usability is much more interested in the average experience, to ensure that players ‘get’ it.
We can see a shift in the gaming industries perceptions of player experience and usability testing, having been championed by leaders such as Microsoft and Valve, more companies are starting to recruit user experience professionals. With the rise of free social games which require the player to ‘get’ games quickly, we can see that the field is becoming a key requirement for success, and is becoming a focal point of how games are made. But it’s not QA!
iPad Usability Study
Just a short update this week, sharing some thoughts on the recent Nielsen-Norman report on usability for the iPad. The recently published study was based research from a combination of both expert evaluation and user-testing, and aimed to discover how people interact with the iPad, and what issues typical users would encounter that prevent them from achieving their goals.
An iPad Nano
Jakob Nielsen admits that the study is not as thorough as a typical usability study. However has decided to share it anyway, due to the over-inflated impact that usability studies produced early in a devices’ lifecycle have been seen to have. As an aside, this is an interesting contrary viewpoint to the disadvantages of being first-to-market noted in ‘Inmates’, which argues that being first to market is irrelevant compared to being ‘best’.
The report has some interesting key findings, including that the apps seen on the iPad and the iPhone suffer from the re-emergence of a problem not seen since the early 90’s. Unlike web browsers and desktop software, which has established graphical conventions to highlight buttons and GUI elements, iPhone and iPad software has not implemented standard conventions, such as making a clickable button appear 3D. Hence there is no consistent manner of designating important aspects of the UI, and users just didn’t know what they could click on. Nielsen likens this to the first emergence of graphical interfaces of the early 90s, when anything and everything could be a button.
Its therefore clear that the main recommendation of the study is to standardize common elements, like navigation, among first and third party applications, such as “swipe to turn page”, or “press and hold to delete”. This also links with the studies’ findings that users were unsure what reaction their action would cause, as the apps have yet to find a consistent manner in which to work.
Not just a big iPhone
The study highlights how the iPad is not just a big iPhone, and different usability issues emerged on the larger device – most pertinent is that navigation elements on the bottom of the page, as seen in many iPhone applications, will not work on the iPad. The larger screen means that these elements are too far from the user’s field of vision to be noticed – and hence are not appropriate. What this means for people who make apps is that a custom iPad version is needed, not just relying on the ‘universal’ up scaling of iPhone apps.
The full report is linked below, and worth a look if you’re interested in the usability, the iPad, or designing an application!
Read the full report, here.
