Practical Usability Rating by Experts (PURE): A Pragmatic Approach for Scoring Product Usability

While working at Intel Security, working with colleagues we developed and tested an economical and expedient alternative method for testing usability.

Usability testing has long been considered a gold standard in evaluating the ease of use of software and websites-producing metrics to benchmark the experience and identifying areas for improvement. However, logistical complexities and costs can make frequent usability testing infeasible. Alternatives to usability testing include various forms of expert reviews that identify usability problems but fail to provide task performance metrics.

This case study describes a method by which multiple teams of trained evaluators generated task usability ratings and compared them to metrics collected from an independently run usability test on three software products. Although inter-rater reliability ranged from modest to strong and the correlation between actual and predicted metrics did establish fair concurrent validity, opportunities for improved reliability and validity were identified. By establishing clear guidelines, this method can provide a useful usability rating for a range of products across multiple platforms, without costing significant time or money.

Video Summary

We presented the findings at CHI EA in 2016. the abstract is located at: ACM Digital Library

Further Innovation
Here is an article Christian Rohrer wrote with an update to the methodology based on his continuing work Quantifying and Comparing Ease of Use Without Breaking the Bank

Daily UX Stories

Check out the Daily UserZen!

I have been curating my UX Twitter feed for years. (a second feed is in the works) I feel that the stories, suggestions and news that come each day are relevant and interesting. Many years ago I also found the service which puts together the feed into whatever layout I like automatically each day.

Not only is it a nice way to view the content, but also highlights different Twitter feeds each day and does some linking back to each company or professional.

Beer & Wine Application UX Project

World Usability Day 2015 has come and gone. I really enjoyed introducing craft beer and wine to my colleagues.

The world of craft beer and wine has just gotten more mainstream. That means there are lots of people who are trying to learn about and try these wonderful concoctions.

I just wonder how good the apps are at helping people find and track their imbibing. There are quite a few options and considering my inquisitive nature, I thought I would put together a basic competitive analysis of some beer and wine applications.

It isn’t exactly comparing apples to apples since some of these apps serve different purposes but I still hope to see some kind of differentiation and information for each.

I will be updating here as the survey continues and I invite people to let me know if you have additional applications that should be added or other feedback.

June 29 Update:

After several months of feedback on these applications, we are now going to be changing the study to reflect what people have been saying as well as include new applications on the market.

This article references a study done by Xenopsi about Craft Beer purchases and mobile usage that is related to this work.

Try one

Try one of the applications

Wallpaper Beer UX slides.002

Now rate the app

Here are the Individual QR codes and links in case its easier:

Brewery Passport

qrcode (Brewery Passport)

Brewery Passport

Next Glass

qrcode (Next Glass)

Next Glass

Picky Pint

qrcode (Picky Pint)

Picky Pint


qrcode (Untappd)



qrcode (Vivino)


Winery Passport

qrcode (Winery Passport)

Winery Passport

Store Basket Usability Opportunity

The biggest tool I use for my job trying to figure out what people need and how they use products is my power of observation.
Sometimes it is hard to figure out the nuances of people’s behavior and what they say their wants are. Ideo tried to improve the shopping cart but those are, sadly, not being incorporated in our trips to the stores.
But sometimes, the truth and the need is right in front of your face.
These shoppers aren’t dumb, the user is never wrong! What I see here is that people find it easier to leave the baskets on the ground. It all makes sense to me. Think about how annoying it would be if you had to carry your belongings through the airport.
We have had roller cases for quite some time; why wouldn’t we apply this to our consumer brick & norther shopping?
Target Shoppers using their baskets “wrong”