While working at Intel Security, working with colleagues we developed and tested an economical and expedient alternative method for testing usability.
Usability testing has long been considered a gold standard in evaluating the ease of use of software and websites-producing metrics to benchmark the experience and identifying areas for improvement. However, logistical complexities and costs can make frequent usability testing infeasible. Alternatives to usability testing include various forms of expert reviews that identify usability problems but fail to provide task performance metrics.
This case study describes a method by which multiple teams of trained evaluators generated task usability ratings and compared them to metrics collected from an independently run usability test on three software products. Although inter-rater reliability ranged from modest to strong and the correlation between actual and predicted metrics did establish fair concurrent validity, opportunities for improved reliability and validity were identified. By establishing clear guidelines, this method can provide a useful usability rating for a range of products across multiple platforms, without costing significant time or money.
We presented the findings at CHI EA in 2016. the abstract is located at: ACM Digital Library
Here is an article Christian Rohrer wrote with an update to the methodology based on his continuing work Quantifying and Comparing Ease of Use Without Breaking the Bank