Posts Tagged ‘quantitative’
multi.player Conference Review – Social Interaction, Bartle and Orcs!
I’ve just got back from a few days in Hohenheim, at the multi.player conference, and wanted to share my experiences. The conference was on “The social aspects of digital gaming”, and was a fantastic chance to meet people working in a similar field to myself.
The talk Pejman and I gave, on measuring social interaction in collocated gaming sessions, seemed to be really well received, and has received some insightful comments. I was a bit worried before about presenting an application of the Bartle Test to the man himself, however he didn’t seem to complain (much…). The research gave a better insight into how different player types react in collocated sessions, and hence was distinct from the other presentations – we also had a strong link to industry!
Amazon understands its customers
Twice this week, I’ve had to discuss the success of Amazon. In my opinion, it’s not due to their content, but rather their successful application of usability and user centred design. Today I’ll be looking at how Amazon has applied user experience principles to build upon their early success and stay competitive in the marketplace.
Click to continue…
Remote Research – Book Review
Remote Research is a new book by Nate Bolt and Tony Tulathimutte, who have worked with the UX agency Bolt | Peters on a wide range of studies, with clients such as Wikipedia and Electronic Arts (I recommend watching the funny out-takes of Spore user testing).
Their new book sums up their experiences with performing remote research (Tony has previously discussed this subject on this blog, in the comments here), and gives clear instructions on how others can perform a wide range of usability and user experience studies with people who are physically distant, by using the internet.
Don't judge it by it's cover...
Why would you consider remote research?
Written by advocates of remote research, the book highlights many of the potential advantages that remote research gives compared to a more traditional lab based study. These advantages are fleshed out throughout the book through testimonies of experts who have experience in this field, who offer real world examples to emphasise these points.
Some key advantages are:
- Access to a geographically diverse user base. Unlike traditional research, where a moderator would have to be in the same physical location as the subjects, remote research allows a study to be run with anyone who has a high speed internet connection, widely expanding the potential study-group.
- Easy to let stakeholders get involved. Because the research session is being broadcast over the internet, it’s possible to allow stakeholders (i.e. executives and designers) to view the session, and give (moderated) input. This of course increases their engagement with the process, and will be the ‘evidence’ for any conclusions derived from the research.
- Natural browsing environment. The validity of the research can be improved, not only because you are allowing the user to perform the task in a familiar environment (their own home computer), but also some recruitment methods allow you to capture a user performing a task they have selected. For example, recruiting a user who came to the site to buy trousers, for a task based on buying trousers, would provide more accurate results than asking someone to pretend to buy trousers…
- Cheaper (debatably). Not having to pay for travel can keep costs down, however other costs, such as incentives, will still be required, as well as paying for the software.
The remote research book doesn’t advocate killing off lab tests though – instead, it recognises that there are cases when the lab is still appropriate, such as when privacy is a concern. The book also features Andy Budd’s defence of the lab, which argues that remote research fails to pick up aspects of non-verbal behaviour, as well as arguing that remote research doesn’t just remove a selection bias (geography), since it also adds another (internet speed and technical ability). It’s brave of the book to include the case against remote research, and helps project a more trustworthy and reliable image for the book itself.
How to do remote research
The ‘meat’ of the book are the sections dedicated to how-to guides on the different forms of remote research. The book contains step by step instructions on performing moderated or un-moderated research, and includes key topics such as recruitment (and live recruiting), card sorts, and lots of handy hints – such as using IM clients as a chat room for multiple observers to automatically share and timestamp notes.
The book doesn’t just cover basic topics – it goes on to develop novel approaches to user research, such as using ‘reverse screen sharing’ to protect confidential software or data, and using mobile web to gain a new understanding of time-dependant information, outside of the traditional moderated setting.
It also extends the remits of remote research – it doesn’t have to just be websites, but can include doodles or sketches, as well as developing ideas for automatic research with analytics.
Another sort of remote research?
Conclusion
Remote Research is one of the easiest to read UX books I’ve reviewed. Like many Rosenfeld publications, it is laid out well, without appearing dense with text, and has a friendly tone throughout. The book can be likened to Krug’s writing in its style, and presentation.
The book is also practical and realistic, and deals with real world issues, like ‘fakers’ (who can be outed by using open ended questions to discover motives), legal issues, and common challenges such as reluctant stakeholders.
Most importantly for the practical UX practitioner, the book is not dogmatic. This is especially evident in the last chapter which admits that usability shouldn’t be the exclusive goal of product design, and needs to be coupled with initiative, and innovation to develop great things.
Overall this book is a great introduction, and how-to guide to the growing field of remote research, and will be an important tool for anyone trying to keep up to date with the latest research methods.
Understanding players through biometrics
Last week UXBrighton hosted an event focused on Biometrics, which featured an interesting presentation by Vertical Slice. Pejman Mirza-Babaei presented his PhD research on the application of biometrics to help understand a player’s experience when playing games. This was presented as a ‘guerrilla’ method, since it was a speedy and rough implementation, not a definitive and comprehensive methodology.
We’ll be looking at what biometric research is, how it can be applied to games research, and the problems that became apparent with this method.
What is biometric research?
Biometrics are traditionally an automated way of recognising, or recording, peoples physiological data, or characteristics. To apply this to video-games readings were taken by Vertical Slice by hooking players up to machines which record their heart rate, brainwaves, or galvanic skin response (…how sweaty their skin is, presumably). It’s proposed that there is some correlation between how their bodies react, and how the player is feeling – such as how a player’s heart will beat faster while fighting Gunther Hermann’s Skull Gun or scoring a tetris.
Pejman Mirza-Babaei has been investigating how this can be applied to games research. Working with Vertical Slice, he is interested in measuring the player experience – how to know when players are having fun, or becoming frustrated, and so has been performing studies to see the feasibility of measuring this with biometric data. By having players play either Haze, or Modern Warfare 2, while hooked up to this machine, maybe it’s possible to gain a greater insight into the player’s thoughts, and how they feel when playing.
And in a non obtrusive way...
What did biometric research show?
When playing the games, the player’s heart rate and GSR ratings (that sweatiness rating) were recorded along with a video of the player, and of their screen. What was found from the biometric readings, at the simplest level, was when the player’s heart rate went up. The researcher would then conduct an interview after the gaming session, and ask why the heart rate went up at those points, for the player to justify the measurements.
We saw examples of these spikes when the player enjoyed, or was frustrated by, a task (such as using a machine gun, or getting stuck looking for a vehicle), and were given the player’s justifications for feeling like this.
The most obvious advantage of this method is that it allows a more natural environment to be created for the player. Since biometrics doesn’t require distracting the player by asking them to perform a think-aloud, or by interrupting their game by asking questions and yet still having a degree of insight into how they are feeling, a more natural game play experience can be achieved, without stopping useful data from being gathered.
Problems with biometric research on games
However, some limitations on the application of this technology became obvious through the presentation. Biometric data (in its current form) doesn’t give any insight into why the player’s heart rate has spiked, just that it has. This problem is exacerbated by the single range of readings it can give – there is no way to distinguish between stress and happiness (or any other reasons a heart rate can spike).
such as exciting vehicles
Because of this, biometric readings have to be justified by another method, to give some understanding as to why the heart rate spikes at certain moments. Traditional UX methods, such as a post-test interview, are therefore needed in addition to biometric readings. However this reintroduces traditional UX problems. A player may not be able to accurately remember why they felt excited at a certain moment, and as pointed out by Sam Nixon, may simply justify their opinion by what they see on screen.
For example, the player may explain a heart rate spike caused by audio cues as being caused by the enemy visible on screen when the clip is shown later, missing the real reason for their excitement.
Conclusion
So biometric readings alone cannot tell us what a player is thinking. Hence they cannot (currently) be a replacement for traditional UX methods.
What biometric readings can do, is aid the application of current UX methodologies. When combined with tools such as think aloud, or interviews, they can add weight to the findings. For a think aloud, it can tell you which parts of the game particularly affected the player, and hence what comments to pay attention too. Similarly with interviews, biometric research can pinpoint the areas that the player should be asked about. When used in combination with typical UX tools, biometric research can be justified and have some understanding applied to its findings.
There is amazing potential in the application of biometric data to games. Currently, the ‘AI director’ in Left For Dead controls the game based on how the player is doing – giving less zombies to fight if the player is doing poorly, or making the game harder, and giving the player some nasty surprises, if they are doing well. Imagine if a system like this could take biometric data into account, and change the game experience based on how the player was feeling. Vertical Slice have begun to show us the potential of this technology, and I feel we’re at the start of an exciting journey.
Quantifying the unquantifiable – Expert Evaluations
At a recent UXBrighton talk, iCrossing presented an interesting idea about applying metrics to expert evaluation. This is a potentially controversial topic, yet has numerous benefits if it can successfully make qualitative data quantitative (and turn impressions and thoughts into numbers). I’ve outlined the method, and my thoughts on the issues around this.
The UXBrighton event was presented in a new format as a series of short talks, from Harry Brignull’s tips on time stamping notes, to Danny Hope’s templates for understanding user roles. Also interesting was a talk on using google analytics, although the length of the talk meant that topic could only be skimmed, dissapointing as I’m an analytics fan. The most interesting idea presented was iCrossing’s presentation on “The iCrossing Connected Brand index: how to measure a brand’s effectiveness online”, given by Ifraz Mughal.
Expert Evaluation
As I’ve mentioned before an expert evaluation is a useful tool for getting an insight into potential usability and user experience issues on a website, or game, with limited resources. Although it can never replace running tests with real users, it can provide a quick approximation, and help highlight the biggest issues.
The ‘method’ for an expert evaluation is simple. Get an expert to look at the site, or game, and tell the client what they think. Job done.
My expert eye tells me you need smarter users
However an expert evaluation can only ever be subjective, and this is it’s biggest weakness. A client can look at your page full of recommendations, and dismiss it as the opinion of one person. There’s no easy way to see progress with changes, and a comparison with other sites can only ever be abstract.
Quantifying an Expert Evaluation
iCrossing’s solution is to quantify their expert evaluation. As part of their ‘Connected Brand Index’ idea, they rate their clients sites (and competitors), on UX-centric areas such as “usefulness”, “usability” and “desirability”.
A traditional expert evaluation would give a qualitative rating, and give examples to back this up, i.e. “Poor – little emphasis, and diffused call to actions”. Instead iCrossing will give the site a score, on a scale of -2 to 2 (2 being very good). This of course can be backed up with examples in a more in depth report.
after the first few pages, the report can just be pictures of kittens. No-one reads that far.
The advantages:
There are numerous reasons why a client would prefer a scored ‘rating’, rather than comments.
- A ‘score’ makes it easy to benchmark, and compare your own scores against competitors. By dividing the expert evaluation into separate topics, and scoring each, a finely grained comparison can be made, and communicated
- Similarly, a score makes it easy for a client to see progress. If they scored -1 before hiring you, and 1 after, your work can be justified (as long as no-one questions who is doing the scoring!)
- Because this produces a concrete score, clients will be able to handle and communicate the data. Graphs can be made, which wouldn’t be possible for subjective comments. These can be invaluable for justifying and communicating with managers and project sponsors, who do not need to see the details, just get a high-level overview.
- This expert evaluation can be encompassed as one aspect of a larger ‘score’ given to websites, or games. This is the idea behind iCrossing’s connected brands index.
Conclusions:
There is an argument this can be seen as a bit of a scam. Giving arbitrary numbers to your opinions doesn’t make them any less subjective. This method of presenting the data could be misleading if presented incorrectly, and the client should be made aware of the method behind the score system. This could become an issue when running comparative studies before and after your work, since you’d be biased towards giving the site a better score after you’ve worked on it.
The point of this method is to aid communication with the client, and give them data in a format that is useful to them. As I discussed in the review of Selling Usability, management and non-technical people would typically much rather see pretty graphs, and statistics, than a list of comments. This method helps manage client expectations, and gives them what they want.
To make the method more valid, it would be useful to perform a study to ensure the method is sound. Perhaps get a wide range of experts to independently rate a wide range of websites on this scale, and note the correlations between the scores. It’d be first step in countering complaints that this method is still inherently subjective, and help make an art into a science.
