The IBF (Intranet Benchmarking forum) go into orgs around the world an measure (amongst other things) the usability of their intranets.

Last year they asked me to help revise their model, which involves a brief expert evaluation against fixed criteria and a day of quantitative-based user observation.

I was super sceptical at first, even feeling a bit dirty about the idea of ‘usability-by-numbers’. So we built in lots of measures to
a) ensure that all evaluators could get close in their findings
b) there was opportunity to identify the subtlety in the scoring.

So the metrics are highly specified – with exact interpretations of what should score 0,1 or 2; the report is presented with lots of co-located expert commentary; the reports are all re-scored by another evaluator; scoring is monitored to see if individual evaluators are generally more generous etc.

The user session are split, with the task completed first so that users can be timed and tracked for comparison, and then encouraged to provide the kind of qualitative feedback that makes studies so powerful. Also, some of the orgs (I think BT included) use the results to drive performance related pay, so objectivity is essential.

So I agree with your comments. IBF provide a cunning blend of stats and scores – to grab the attention and interest of the management, and screenshot examples and comments – to point the way forward for change.

Also, the IBF is effectively the study you mention in your last paragraph – a group of independent usability evaluators applying a consistent framework across a big range of intranet sites. Check out the website for more info if your interested in the issues, http://www.ibforum.com or I’d be happy to chat.

Lou.