Top Ten Usability Findings of 2010

I think these findings have been “found” before, but its good to reassure the researchers that our assumptions are still valid.

Keep these, posted by Jeff Sauro on measuring usability, in mind as you design your next research project.

  1. 5 Second Usability Tests: Ratings of website usability after only 5 seconds are the same as those after 10 minutes.
  2. Unmoderated Usability Data is Mostly Reliable: Data from remote usability test takers is rather similar to lab based studies except for task-times which differ more substantially.
  3. Cheaters: Around 10% of paid usability testers will cheat on your test by rushing through the questions just to receive the honorarium.
  4. The Geometric Mean works better than the median for reporting the best middle task time for sample sizes less than 25.
  5. Usability accounts for at least 30% of customer loyalty: Net Promoter Scores correlate highly with scores from the System Usability Scale (SUS).
  6. Users Self-Reporting Problems: Users are able to find and report around 50% of the problems usability professionals find. Just asking users to report what problems they encountered, how severe they are and potential fixes can be a cheap and effective complement to other usability activities.
  7. Survey respondents prefer the left-side of the rating scale. The way you order your response options matters. People generally lean toward responses that are on the left-side. If you have more favorable responses (e.g. Strongly Agree) on the left you’ll get a slightly inflated score.
  8. Asking users to rate task-ease during a task lowers ratings: If you give users only five seconds to complete a task they will rate the task as much more difficult than those who are given no time limit. Contrast this finding with the 5 seconds tests results which shows that user attitudes about usability are different at the task vs. whole website level.
  9. Making survey questions more extreme will generate more disagreement: Scores will be higher if questions are all extremely negatively worded and scores will be lower if all the questions are extremely positively worded.
  10. Usability problems are almost 10-times more common on business applications than on websites

Myths About Remote Usability Testing

Remote testing is becoming more prevalent in the industry and thanks to many industry people, the previous notions of why it either was or wasn’t wasn’t a good choice are being revisited.

Each research situation needs to be assessed to determine the best methodology, and the researcher should be aware of both the challenges and benefits involved with remote testing.

REMINDER: Nate Bolt is giving a Remote Usability Testing presentation through UIE this Thursday.

TechSmith put out a fairly good pdf about Remote Testing if you are using Morae in conjunction with remote software.

I wrote about many tools (including remote monitoring) that are available in a previous post

In an article Published: July 5, 2010 on UX Matters, Corrie Kwan, Jin Li, and May Wong wrote about some of the Myths of Remote Usability Studies

“Funding for user research travel is becoming more limited, and the availability of local users who meet the need for diversity is often insufficient. Therefore, UX professionals have started using remote usability testing methods to gather adequate user feedback.”

Success in a diverse global marketplace increasingly demands that companies engage customers from diverse global backgrounds in both discussions and usability studies. However, funding for user research travel is becoming more limited, and the availability of local users who meet the need for diversity is often insufficient. Therefore, UX professionals have started using remote usability testing methods to gather adequate user feedback.

The software development industry is relatively young, the UX professions within it are even younger, people working in user experience have different backgrounds, and their professional practice is still evolving. Remote usability activities have not yet been well studied. Consequently, a number of myths have arisen.

In this article, we’ll draw on our collective, first-hand experiences doing remote usability studies for numerous real-world projects to describe and debunk these myths. Our goal is to share knowledge and inspire action.

The Myths

We’ve identified six myths about remote usability studies.

Myth #1—Remote usability studies are more prone to user distractions and interruptions that could invalidate results.

Our Experience—Interruptions are part of a natural work environment and actually make the testing environment and results more realistic.

“Because users are working in their native work environments or at home, we can observe and capture more real-life feedback in these natural settings.”

When facilitating remote usability studies, a common frustration is our inability to control the users’ environment. During remote usability studies, users might be participating over the phone, from their offices or homes. Study participants frequently get interrupted by their coworkers—inviting them to go to lunch or asking them questions—or by kids playing in the house. Many facilitators feel these distractions are a challenge for remote usability studies, because they can sidetrack users from the tasks they’re working on. However, such interruptions and our apparent lack of control over them might be just what we need. Because users are working in their native work environments or at home, we can observe and capture more real-life feedback in these natural settings.

We have an interesting story to tell about interruptions. During one remote usability test session, a user was working on a task from her home office, with a connection to a source-code server. Like many other people on her team, she often works from home. The session got interrupted by her crying baby. She went away to calm down her child, then 15 minutes later, when she returned to the task, she was faced with a time-out error message! The software assumed users would always be working with the user interface, so if there was no interaction for more than 10 minutes, it terminated the server connection.

This is the type of feedback we could easily miss in a face-to-face usability test session in a lab, because it is usually difficult to mirror a user’s exact work environment, especially one that includes a crying baby.

Remote usability studies reflect the natural environments users typically work in. They enable us to observe unexpected factors that can affect a participant’s interactions with an application, such as interruptions by phone calls, a pop-up online message, a baby’s crying, or even a system crash. Our designs must not only consider users’ goals for performing tasks, but also reduce the impact of distractions that could happen in a user’s daily work.

Myth #2—Results from remote usability studies are as good as those from face-to-face usability studies.

Our Experience—Quantitative data from remote usability studies can be as good as that from face-to-face usability studies. However, qualitative data from remote usability studies is often inferior.

“Without a way to observe study participants during remote usability studies, facilitators cannot detect nonverbal signals.”

Initially, remote usability studies were for testing Web sites. UX professionals have had early success with collecting Web site usability test metrics such as the number of clicks finding a target page requires. Such quantitative data is as easy, if not easier, to collect through remote usability studies, in comparison to face-to-face usability studies.

However, without a way to observe study participants during remote usability studies, facilitators cannot detect nonverbal signals. In such cases, lacking the direct observation of participants’ nonverbal expressions and behaviors and basing results only on metrics such as task completion rates or user satisfaction scores can lead to false optimism.

In our experience, certain types of usability studies are more difficult to conduct remotely. For example, requirements brainstorming and design walkthroughs are less suitable for remote usability studies, because they rely on synergy between a facilitator and participants through both verbal and nonverbal communications. For types of usability studies such as these, which rely on empathy, understanding, and creating a connection with participants, the loss of nonverbal communication in remote usability test sessions is significant. This kind of empathy is nearly impossible to build using Web conferencing software. Since 80% of our communication is nonverbal and these types of usability studies depend on nonverbal communication, the results of such remote studies are poor.

Myth #3—Poor audio and slow screen sharing hinder user feedback.

Our Experience—Audio, video, and screen-sharing applications have improved dramatically and enable good collection of user data.

“Audio, video, and screen-sharing applications have improved dramatically.”

Just a few years ago, getting these technologies to work for remote usability studies was a headache. Now, however, audio, video, and screen-sharing applications have improved dramatically. Today, we are no longer plagued by static-filled, half-duplex audio conferences and jumpy mouse pointers during screen sharing.

In our experience, new VoIP software sometimes works better than plain old telephone service (POTS) for audio conferences. POTS-based audio conferences have poor quality for trans-Atlantic and trans-Pacific calls and issue annoying beeps when participants put their calls on hold. In a few of our remote usability studies, participants have commented that the screen-sharing application performs so well, it seems as though they were sitting next to the facilitator in person, watching the same screen. We find that, at a minimum, good audio-conferencing and screen-sharing applications are technologies that are essential for remote usability studies.

Even with audio and screen sharing, interpersonal interactions during remote usability studies are still a bit impersonal. We have tried using a Webcam to make testing sessions more personal and gather some nonverbal feedback. However, for privacy and security reasons, a Webcam might not be acceptable to some remote participants. Therefore, we have tried another approach to enhancing visual contact between participants and session facilitators.

Recently, we held a series of remote usability sessions for design exploration with customers. During the first session, even after a virtual roundtable of self-introductions and ice-breakers, we found customers were not fully engaged, so were not providing feedback on the design. So, before the second session began, we asked the same participants to send us their photos, and we showed their photos to everyone at the beginning of the session. We displayed the photos during the session, so participants could match a photo with the person talking. The addition of visual identity seemed to help participants to open up and talk more and become more engaged.

Myth #4—It is quicker to recruit participants for remote usability studies.

Our Experience—While there are potentially more users available for remote usability studies, rapport building and user screening are more difficult.

“Remote usability studies enable us to reach a broader range of users and increase our chances of finding qualified participants.”

Remote usability studies enable us to reach a broader range of users and increase our chances of finding qualified participants. However, if we use the same approach in screening participants as in conducting the actual sessions, we might not take full advantage of the larger user pool remote usability studies offer.

A common practice in participant screening is to evaluate candidates’ technical experience. However, we’ve found that the answers we get in response to questionnaires during screening usually don’t provide us with many clues about a candidate’s likely performance during a usability study. Participants who satisfy the screener perfectly might not be productive during a study if their personalities don’t suit the types of interaction a study requires or facilitators aren’t aware of their communication preferences.

Introverts are usually reserved in communicating with others until they know and trust them. In a local usability study, which permits face-to-face interaction, a skillful facilitator can create a connection with introverted participants, making them feel more comfortable and relaxed in expressing their thoughts. Facilitators can observe introverted participants’ nonverbal signals and prompt participants with a series of questions that helps them to understand what’s on their minds. However, the traits of introverts make it more difficult for facilitators to conduct remote usability sessions. This is especially true for studies that involve design brainstorming, where thoughts participants exchange among themselves can bring out the best results.

Based on our experience, it’s not always quicker to recruit participants for remote usability studies. We can offer the following recommendations to make the recruiting process more effective. First, add a few personality questions to the screener to determine whether a potential participant is an introvert. Second, try to recruit extroverts—or at least people who enjoy expressing their opinions in group sessions. Third, use an online polling system and private chats to elicit feedback from all participants. This lets you draw out the thoughts of introverts.

Myth #5—It is easier to ask for participants’ time for remote usability studies than for local usability studies.

Our Experience—We might get the participants’ time, but not necessarily their attention.

“It is harder to hold remote participants’ attention during a remote session….”

Because remote participants can participate in usability studies from their homes, cottages, or wherever is convenient for them, some believe it is easier to ask remote participants to give their time for a remote study. However, in our experience, it is harder to hold remote participants’ attention during a remote session, because they can put us on hold to take another phone call or start doing other things on their computer. Thus, we may not have their undivided attention throughout a remote session. Often, participants are not only physically absent, but are not fully present even in a virtual sense.

Myth #6—It is cheaper to conduct remote usability studies than local usability studies.

Our Experience—Considering the money and resources that are involved in each type of study, face-to-face usability testing is more cost effective.

“We also need to consider intangible costs to better compare the overall expenses of remote and face-to-face usability studies.”

For a face-to-face usability study, it costs money to recruit users, reimburse meal and travel expenses for participants, and visit a customer site to conduct field studies. For a remote usability study, it costs money to recruit users, pay for teleconferencing and the licensing fees for software tools. In terms of tangible costs, remote usability studies are much cheaper in general.

However, we also need to consider intangible costs to better compare the overall expenses of remote and face-to-face usability studies. The logistics, setup, and coordination of face-to-face sessions are simpler. For remote usability studies, we need to be careful about additional matters such as pairing up participants from different countries and cultures, ensuring participants’ software platforms and tools are not incompatible with ours, working effectively across different time zones, and the more frequent need to reschedule sessions.

For example, we recruited participants for one remote focus group session from both India and Europe, which posed a significant time-zone problem for us. To get all of these qualified participants together at the same time, some of them would need to stay up late at night to participate. They initially accepted the proposed session time, but in the end, they didn’t show up.

Another intangible cost occurs when our technology fails us. For example, we ran a remote usability session and recorded it. The session included some priceless moments when users were struggling to interact with the product. We were eager to show our recording to the product development team and influence changes. However, to our horror, the session recording failed to save. We lost the entire recording! After this episode, the entire team that was involved in the study felt completely stressed and demoralized. What dollar value should we put on a lost audio or video recording of such priceless moments? What dollar value should we put on the devastating psychological effect on the morale of the team members? We cannot assign a dollar value to such intangible costs, but their impact goes well beyond money.


“Remote usability studies are not necessarily a cure for all of the problems that global usability testing presents.”

Drawing on our collective, first-hand experiences of remote usability studies from numerous real-world projects, this article has debunked some of the myths that exist about remote usability studies. We have found that remote usability studies are not necessarily a cure for all of the problems that global usability testing presents. There are many considerations other than travel expenses.

New Mac Products for 2009

Macworld is where Apple likes to introduce their products just a couple days earlier than CES…just to, you know, set themselves apart.

There are new accessories like mice, chargers, notebook cases and hands free bluetooth accessories for the car.

New software was introduced including all new iLife 09 with photo, movie, garageband and web applications that include alot more interactivity and control. The iWork suite has some new features, most notable is the new application for iPods or iPhones that allows for remote control of Keynote presentations that is similar to the Remote application for iTunes and AppleTV. The app will be available for $1 at the iTunes Store.

Real Software announced Tuesday at Macworld Expo that REAL Server 2009 will ship in February. REAL Server is a cross-platform, multi-user database server. There are several major upgrades for the update. The Real Server Developer Edition is free for developers to use to develop a server application.

iTunes is going DRM free in March, they are also changing their pricing on the products (songs, shows and movies) from a straight $.99 to $.69, $.99 or $1.29. Additionally, users will be able to make purchases over 3G as well as Wi-Fi, which would make it possible to buy items as long as you have a cell connection.

iPhone 2.0 Free Applications Review Part 1

iPhone Application Store

It seems that each night well after I should be in bed, I end up downloading and playing around with all the new applications available for the iPhone.

I realize that it may not be the most mature thing I am doing with my time… My only defense is that I have yet to pay for any (I cruise the free apps section) and try out those rather than paying the fees for apps that probably don’t work well yet, anyway. We are, after all, still in a kind of Beta grey area.

If you haven’t already looked, the new iTunes 7.7 is available and then you can see the application store. You will need an iPhone or iPod Touch with the 2.0 firmware upgrade to try these out.

The iTunes download page.

People don’t really know yet quite how all this will shake out. Remember that July 11th when everybody went to get their new iPhone that the servers were overloaded all day. There are still some bugs to be sure.

Regardless, I have already tried out:

Career Builder– I should have known from the fact that their site does not really provide quality jobs or networking that this app would not be any better, but like many others, I was exited to try. Don’t waste your time.

Holy Bible– While I would not call myself a religious person necessarily… I was intrigued by the description that it was the whole King James Bible with annotated notes as well as a dynamically updated comments on each entry. I was surprised and delighted by the interface and the content. I recommend it for those who wish to study and discuss the Bible

I’ve tried out the Jirbo Arcade, Break and Avatar – while the Break game has a cool interface and I like the idea of uploading my scores to compare with others, the interfaces seem simple and unimaginative. The icons are cutesy so maybe its not geared for my demographic, but I don’t see highly stimulated teens getting into memory games and a measly 15 levels of game play (I finished them all the first time I played). Forget it.

iMaze – This is one of the games that takes advantage of the accelerometer feature of the iPhone. Arguably the aspect that differentiates the games on the device from those available on other phones. Here, you attempt to move a ball through a maze by tilting the iPod touch or iPhone to get the ball to roll. As the levels increased in difficulty, I found myself getting fairly frustrated (a good sign) and one that shows that the game will provide distraction on those long flights or BART rides. Get it.

Moonlight Mahjong – This game is worth downloading if only for the interface. Gameplay is the same as Mahjong in that ‘free’ tiles must be matched in twos to remove them from the board, but the player can rotate and zoom in and out of the board in three dimensions. The graphics really take advantage of the iPhones abilities and when I get frustrated, I sometimes just zoom around the board to see the pretty blocks…Get it.

Twitterriffic – This application has too many letters in its name and there are too many applications accessing Twitter, considering that the interface for Twitter worked great on the previous iPhone firmware versions. I will admit that it is nice that now the tweets are ‘pushed’ to your device and the interface is a little cleaner and links are nicely dealt with. The application is free, so there is nothing wrong with downloading it if you already have a Twitter account (and the Fail Whale is submerged). Just don’t expect a paradigm shift to occur with the application. Take it or leave it.

PageOnce – I love it! One great compact interface that the user can customize to connect to bills, email, social sites, wireless accounts. Each link gives updates and hints as too when there is new information. A must have for those on the go who want to stay connected in business. Must Get it.

Remote – I have used this application nearly everyday. It allows me to control my music, podcasts, movies and iTunes on my network from my iPod. Now the iPod is a remote that I can use anywhere where I can access my home network. An absolute must have.

I will review more applications later and invite those of you who have already tried these or others to chime in with your opinions.

Read Part 2 of the Review