Practical Usability Rating by Experts (PURE): A Pragmatic Approach for Scoring Product Usability

While working at Intel Security, working with colleagues we developed and tested an economical and expedient alternative method for testing usability.

Usability testing has long been considered a gold standard in evaluating the ease of use of software and websites-producing metrics to benchmark the experience and identifying areas for improvement. However, logistical complexities and costs can make frequent usability testing infeasible. Alternatives to usability testing include various forms of expert reviews that identify usability problems but fail to provide task performance metrics.

This case study describes a method by which multiple teams of trained evaluators generated task usability ratings and compared them to metrics collected from an independently run usability test on three software products. Although inter-rater reliability ranged from modest to strong and the correlation between actual and predicted metrics did establish fair concurrent validity, opportunities for improved reliability and validity were identified. By establishing clear guidelines, this method can provide a useful usability rating for a range of products across multiple platforms, without costing significant time or money.

Video Summary

Abstract
We presented the findings at CHI EA in 2016. the abstract is located at: ACM Digital Library

Further Innovation
Here is an article Christian Rohrer wrote with an update to the methodology based on his continuing work Quantifying and Comparing Ease of Use Without Breaking the Bank

Aardvark Shutting Down

According to an email I got this morning:

Dear Aardvarker, 

This mail is to remind you we will be shutting down vark.com shortly, and you will no longer be able to access Aardvark.

We’re sad to say goodbye to Aardvark now, but pleased that this project has inspired so much related activity in this area.
(See http://googleblog.blogspot.com/2011/09/fall-spring-clean.html for the broader story.)
Thank you for your support for Aardvark along the way!

All our best,

– The Aardvark Team

(p.s. You can visit blog.vark.com for information about data portability after Aardvark shuts down.)

I am really bummed. I enjoyed being a part of this community for the last year and a half.

My only solace is the hope that with the fall-spring-clean, that Google can finally deliver on the projects they keep promising (I’m looking at you, Google TV!)

Here is the text of the post on blog.vark:

Goodbye Aardvark

By Max, Zoo Director | Published: September 2, 2011

As part of the shutdown of GoogleLabs, Aardvark will be shutting down at the end of September.

Aardvark began as a small experiment in a new kind of social search, and over a few years blossomed into a service that made millions of connections between people to answer each other’s questions.  It was a great experience in seeking to combine a broad vision for the future of technology with a rigorous user-centered design process.  Over this time, we learned a lot about creating and maintaining online communities, and how to facilitate sharing of knowledge between people.

We’ve been excited to share these lessons within Google over the past year, especially as part of the effort behind Google+.  It has been gratifying to see how well this project is doing — even in these early stages, Google+ has already become a great place to share knowledge online, eclipsing the original vark.com! — and there is much more to come very soon.  In this and other projects at Google, the Aardvark team remains committed to developing powerful tools for connecting people and improving access to information.

Of course, we’re also sad to say goodbye to the original Aardvark here at vark.com.  Who would have thought that a digital version of a nocturnal burrowing mammal would have engendered so much affection!  We’re very grateful to the whole Aardvark community for your support along the way.

Onwards,

Max and Damon

p.s.  In the spirit of data liberation, we want to make sure that any of you Aardvark users who would like to retrieve your past data from the service can do so.  Right now of course you can log in and manually copy anything you’d like to keep.  Or if you prefer, you can email us at support@vark.com and we’ll send you a bulk dump of your data (please allow time for us to process these).  After September 30, there will be no way to retrieve your data, as shortly after that we will delete everything permanently.

Android Security Application

With the recent rash of viruses and malware in the Android Market, more attention is being paid to the need to protect mobile devices. A recent free application is Lookout Mobile Security reviewed on InformationWeek . I have to say I am still partial to WaveSecure, winner of Software of the Year 2010. It does everything that Lookout does, with additional features including tracking, buddies and better updated virus information.

Full Disclosure: I am employed at McAfee who now owns WaveSecure.

Whichever app you decide on, it is obvious that people can no longer wait to get protection on their mobile devices. Not to be “Fox-ish”, but you may already be infected and should at the very least, download a free trial to ensure your device is not compromised.