Comments on: HCI learning, a day analyzing user experience, and thoughts about remote usability testing http://www.stevebromley.com/blog/2009/10/19/hci-learning-a-day-analyzing-user-experience-and-thoughts-about-remote-usability-testing/ Usability, user experience and HCI for games and the web Thu, 02 Sep 2010 07:28:02 +0000 hourly 1 http://wordpress.org/?v=3.0.1 By: sad http://www.stevebromley.com/blog/2009/10/19/hci-learning-a-day-analyzing-user-experience-and-thoughts-about-remote-usability-testing/comment-page-1/#comment-1666 sad Thu, 02 Sep 2010 07:28:02 +0000 http://www.stevebromley.com/blog/?p=102#comment-1666 Nice post. I definitely enjoy your sense of humor. And Feralabs sounds interesting. Nice post. I definitely enjoy your sense of humor. And Feralabs sounds interesting.

]]> By: WP Themes http://www.stevebromley.com/blog/2009/10/19/hci-learning-a-day-analyzing-user-experience-and-thoughts-about-remote-usability-testing/comment-page-1/#comment-291 WP Themes Mon, 01 Feb 2010 00:26:50 +0000 http://www.stevebromley.com/blog/?p=102#comment-291 Amiable dispatch and this enter helped me alot in my college assignement. Gratefulness you on your information. Amiable dispatch and this enter helped me alot in my college assignement. Gratefulness you on your information.

]]> By: Amanda McNeill http://www.stevebromley.com/blog/2009/10/19/hci-learning-a-day-analyzing-user-experience-and-thoughts-about-remote-usability-testing/comment-page-1/#comment-21 Amanda McNeill Thu, 05 Nov 2009 04:11:12 +0000 http://www.stevebromley.com/blog/?p=102#comment-21 Hi Steve, Nice post. I definitely enjoy your sense of humor. And Feralabs sounds interesting. Here is an article from Website Magazine http://bit.ly/32mqlQ that reviews usability tools (remote and moderated) as well as usertesting.com, who I am affiliated with. Amanda Hi Steve,

Nice post. I definitely enjoy your sense of humor. And Feralabs sounds interesting.

Here is an article from Website Magazine http://bit.ly/32mqlQ that reviews usability tools (remote and moderated) as well as usertesting.com, who I am affiliated with.

Amanda

]]> By: Software http://www.stevebromley.com/blog/2009/10/19/hci-learning-a-day-analyzing-user-experience-and-thoughts-about-remote-usability-testing/comment-page-1/#comment-17 Software Sun, 01 Nov 2009 06:50:21 +0000 http://www.stevebromley.com/blog/?p=102#comment-17 Another great post. Thank you for the information, Its good to see such quality posts. Im subscribing to your blog. Keep them comming. <a href="http://www.rain-clouds.com" rel="nofollow">Classified Posting</a> Another great post.
Thank you for the information, Its good to see such quality posts.
Im subscribing to your blog.
Keep them comming.

Classified Posting

]]> By: Steve Bromley http://www.stevebromley.com/blog/2009/10/19/hci-learning-a-day-analyzing-user-experience-and-thoughts-about-remote-usability-testing/comment-page-1/#comment-13 Steve Bromley Fri, 23 Oct 2009 14:19:33 +0000 http://www.stevebromley.com/blog/?p=102#comment-13 Really interesting points Tony, and will definately check out the book. Good promotion! :-) Really interesting points Tony, and will definately check out the book. Good promotion! :-)

]]> By: Tony Tulathimutte http://www.stevebromley.com/blog/2009/10/19/hci-learning-a-day-analyzing-user-experience-and-thoughts-about-remote-usability-testing/comment-page-1/#comment-10 Tony Tulathimutte Wed, 21 Oct 2009 19:21:13 +0000 http://www.stevebromley.com/blog/?p=102#comment-10 Hey Steve—you're right, Webnographer (and other automated services like Loop11, Usabilla, Webeffective, UserZoom, etc) do provide a way to get around moderating expenses, but those studies address different issues: automated research is generally good for large-sample, quantitative answers to very specific and pre-defined tasks. Moderated research, on the other hand, is good for obtaining qualitative behavioral feedback in a rich usage context—you can see all the different tools and websites and workflows and even physical artifacts the participant is using to get his task done. It helps you understand the motivation of the task, not just the outcome. And on top of that, you can see completely unexpected behaviors come up, ones that you didn't plan for during testing, and that can't really happen in automated testing. That's just a brief overview of those issues, and of course there are ways to get quantitative data from moderated studies and some qualitative data from automated ones. We deal more with those issues in our book, which is coming out soon. Check it out if you're interested: http://www.rosenfeldmedia.com/books/remote-research/ Hey Steve—you’re right, Webnographer (and other automated services like Loop11, Usabilla, Webeffective, UserZoom, etc) do provide a way to get around moderating expenses, but those studies address different issues: automated research is generally good for large-sample, quantitative answers to very specific and pre-defined tasks.

Moderated research, on the other hand, is good for obtaining qualitative behavioral feedback in a rich usage context—you can see all the different tools and websites and workflows and even physical artifacts the participant is using to get his task done. It helps you understand the motivation of the task, not just the outcome. And on top of that, you can see completely unexpected behaviors come up, ones that you didn’t plan for during testing, and that can’t really happen in automated testing.

That’s just a brief overview of those issues, and of course there are ways to get quantitative data from moderated studies and some qualitative data from automated ones. We deal more with those issues in our book, which is coming out soon. Check it out if you’re interested: http://www.rosenfeldmedia.com/books/remote-research/

]]> By: UX Brighton Round-up | Remote Usability http://www.stevebromley.com/blog/2009/10/19/hci-learning-a-day-analyzing-user-experience-and-thoughts-about-remote-usability-testing/comment-page-1/#comment-9 UX Brighton Round-up | Remote Usability Wed, 21 Oct 2009 17:23:53 +0000 http://www.stevebromley.com/blog/?p=102#comment-9 [...] Steve Bromley: In the heated Q&A session after, it was discussed at length that this should be used in conjunction, and not instead of face to face interviews, for it was agreed that remote usability studies cannot log or reproduce every element of a close personal study, you fail to see the emotions and reactions of the participant involved, and it’s harder to adapt the test to study interesting emerging behaviour variants. [...] [...] Steve Bromley: In the heated Q&A session after, it was discussed at length that this should be used in conjunction, and not instead of face to face interviews, for it was agreed that remote usability studies cannot log or reproduce every element of a close personal study, you fail to see the emotions and reactions of the participant involved, and it’s harder to adapt the test to study interesting emerging behaviour variants. [...]

]]> By: Jack Josephy http://www.stevebromley.com/blog/2009/10/19/hci-learning-a-day-analyzing-user-experience-and-thoughts-about-remote-usability-testing/comment-page-1/#comment-8 Jack Josephy Wed, 21 Oct 2009 10:31:23 +0000 http://www.stevebromley.com/blog/?p=102#comment-8 Nice post Steve. I have actually been using Pidoco to help with a client project. The tool is pretty intuitive and has about a half hour learning curve, though it seemed to occasionally suffer from a bit of lag due to the fact it works online. But I'm all up for these tools that make UX/web-design easier and I do believe remote testing often captures the natural user-system interaction more precisely than live testing, particularly in terms of actual usage data. Keep up the good work Nice post Steve. I have actually been using Pidoco to help with a client project. The tool is pretty intuitive and has about a half hour learning curve, though it seemed to occasionally suffer from a bit of lag due to the fact it works online. But I’m all up for these tools that make UX/web-design easier and I do believe remote testing often captures the natural user-system interaction more precisely than live testing, particularly in terms of actual usage data.

Keep up the good work

]]> By: Steve Bromley http://www.stevebromley.com/blog/2009/10/19/hci-learning-a-day-analyzing-user-experience-and-thoughts-about-remote-usability-testing/comment-page-1/#comment-7 Steve Bromley Wed, 21 Oct 2009 10:08:30 +0000 http://www.stevebromley.com/blog/?p=102#comment-7 Hi Ofer, thanks for taking a look! My issue wasn't just with monitoring twitter, it was due to the unmonitored nature of the tasks - i think that asking people to log their impressions of using the camera after the event would produce a different range of responses to their impressions during the user experience. For example a bad experience downloading the pictues of a camera may cloud the users opinion of every aspect of the camera, if asked about it later when writing their web diary. I think you've addressed this though, as you said in your comment above, by developing things such as the WAP interface, and the iPhone app - these would allow the user to log their experience while using the camera, and avoid losing valuable user experience data. Let me know when the website is live, and i'll be sure to check it out in more detail and link to it! Hi Ofer, thanks for taking a look!

My issue wasn’t just with monitoring twitter, it was due to the unmonitored nature of the tasks – i think that asking people to log their impressions of using the camera after the event would produce a different range of responses to their impressions during the user experience. For example a bad experience downloading the pictues of a camera may cloud the users opinion of every aspect of the camera, if asked about it later when writing their web diary.

I think you’ve addressed this though, as you said in your comment above, by developing things such as the WAP interface, and the iPhone app – these would allow the user to log their experience while using the camera, and avoid losing valuable user experience data.

Let me know when the website is live, and i’ll be sure to check it out in more detail and link to it!

]]> By: Ofer Deshe http://www.stevebromley.com/blog/2009/10/19/hci-learning-a-day-analyzing-user-experience-and-thoughts-about-remote-usability-testing/comment-page-1/#comment-6 Ofer Deshe Wed, 21 Oct 2009 09:18:43 +0000 http://www.stevebromley.com/blog/?p=102#comment-6 Thank you for covering the event well. I do agree with you that by simply monitoring Tweeter without well-defined research questions, tasks and incentives it could be hard to achieve valid conclusions. However, I thought that our main example was showing how people were given specific tasks related to using a specific type of camera in different contexts. They then uploaded their impressions and satisfaction level into our tool answering specific questions. By the way they were also given an incentive. Our tool uses a dedicated iPhone application, a WAP/.mobi and a web-based diary. All of which are configurable and allow for specific interview questions to be asked and linked to structured tasks. The key is the capture of data at the point of experience, which could be whilst completing a task. The EthnoLabs app also simplifies data analysis by using qualitative analysis tools that implement a number of algorithms. I am surprised by the fact that you decided to only focus our API and on Twitter, which was an example for a feed that we have as an additional way to monitor trends and brand related conversations. Thank you for covering the event well. I do agree with you that by simply monitoring Tweeter without well-defined research questions, tasks and incentives it could be hard to achieve valid conclusions. However, I thought that our main example was showing how people were given specific tasks related to using a specific type of camera in different contexts. They then uploaded their impressions and satisfaction level into our tool answering specific questions. By the way they were also given an incentive. Our tool uses a dedicated iPhone application, a WAP/.mobi and a web-based diary. All of which are configurable and allow for specific interview questions to be asked and linked to structured tasks. The key is the capture of data at the point of experience, which could be whilst completing a task. The EthnoLabs app also simplifies data analysis by using qualitative analysis tools that implement a number of algorithms.

I am surprised by the fact that you decided to only focus our API and on Twitter, which was an example for a feed that we have as an additional way to monitor trends and brand related conversations.

]]>