Skip to page content

United States (English) Change

Usability testing at OCLC


Methodology

Introduction

The HCI team is probably best known for doing usability testing in the OCLC Usability Lab (Ulab). The Ulab was established in July 1990 and since that time, has usability tested over 120 products with more than 500 users.

photograph of OCLC Usability Lab

But usability testing really does not require a lab. It does require eyes, ears, and a recording mechanism (pencil and paper work great!). Also needed are something to be evaluated and a user to evaluate it, of course!

With this 'equipment', the four basic components of usability testing can be implemented:

  1. Identify, watch and listen to users
  2. Record user behavior
  3. Analyze the data
  4. Report the results

1. Identify, watch, and listen

The first step is to identify users who typify your expected audience. Are they high school students or professors? Professional researchers or casual users? In usability testing, 3-5 users will usually provide the most information for your investment (Nielsen[2]).

Next, you need to watch and listen to users as they use your products. You could do this passively, quietly observing users as they use your product. This provides some very valuable information, especially about what it is users want to do with the product.

However, this passive technique does not allow comparisons across users, because each user may do something different. And the user may not use the component you are interested in. To do this, you may need a more controlled method:

  • Identify those parts of your product you would like to usability test.
  • Develop tasks that require the user to use those parts.
  • Establish usability questions and quantitative goals. A usability question is a statement of what you are trying to determine, such as "Does the user easily find the Search button?" There is usually one question for each task.

The quantitative goal is something measurable that defines the 'correct' answer for the usability question. For example, "User must find Search button within 1 minute of displaying the page."

The drawback of this controlled method is that users may not be doing what they usually would do when using the system. Hence, it is recommended that identifying the parts of your product to test use the passive viewing method, to see what parts of the product users use most. Second, develop tasks that require use of those parts, to allow you to compare results across users, and use the controlled method.

2. Record user behavior

We use video and audio recording equipment for archiving test sessions, in case later review is needed. However, our primary tool is taking notes with paper and pencil! This is still the fastest way we have found to get information quickly to developers and designers.

There are some things to keep in mind while taking notes:

  • Keep track of what task the user is working on
  • Note any user comments or actions you feel reflect on the usability goal of the task
  • Take the measurements needed to determine if the quantitative measure was achieved
  • Note any ideas you or others may have on how to improve the product or service's usability

3. Analyze the data

You have your data. You now need to tabulate and analyze it.

We create two tables. Both tables contain the following basic information:

  • The task number
  • A brief description of the task
  • The task's usability question
  • The task's quantitative goal

In one table, this additional information is presented:

  • Whether the quantitative goal was achieved for each user
  • Some summary statistic that represents whether the goal was achieved across users. Because of the small sample size (3-5 users), this is often a mode or median.

In the second table, we add to the basic information a "Suggested Solution's" column. These "Suggested Solutions" describe the problem and suggest a solution. The solutions are based on published research and what has been observed in previous users and tests.

4. Report the results

How do you report your results, so they (1) Are most useful to those who can actually change the product, and (2) Actually results in improvements to the usability of the product?

We have found that the data analysis is not the most important component of a test(s) report. Rather, it is how the findings are presented.

For example, you could limit your analysis to:

  • What problems occurred
  • What goals and measures were and were not achieved
  • Ways you think the product could be improved

Certainly this information should be in every report. However, reporting only this information did not work for us!

We quickly found that such a "just the facts" approach alienated our audience, because of an attachment of that audience (usually designers and developers) to their product. We were telling them their baby was ugly, an underachiever, and only we could help!

To correct this, we changed our reports to also include:

  • Congratulations to the designers and developers for subjecting their product to criticism to improve it. (This is in the very first paragraph of every report.)
  • A "kudos" section, also at the beginning, citing the good things seen during testing
  • Changing the "Solutions" section to a "Suggested Solutions" section

The tone of the reports changed from "Here are the facts. Deal with them" to "Good job. Here are some things to consider." After this change, we had more repeat 'customers' and more of our suggested solutions were used.

For more information about usability testing in the OCLC Ulab, see How we do it: usability testing.

Remote usability testing procedure

Remote usability testing is a variation of the standard usability testing procedure, with one major difference: the test user can be anywhere in the world that has a PC, and either a Internet connection capable of sending audio or a phone on a separate line.

Web remote observation software allows you to view the user's screen and mouse movements on your computer. The audio connection lets you hear the user's comments as they use the software. About the only thing missing is actually seeing the user use the product. We use Microsoft NetMeeting® as our remote observation software, but you may find that other packages (e.g., Netopia's Timbuktu® or CuseeMe Network's CU-SeeMe® Web) work better for you. NetMeeting is free Microsoft software and often comes bundled with a PC's operating system.

Compared with testing in the usability lab, remote usability testing can gather usability data from a wider range of users and is more comfortable for most users, who may get nervous in a usability lab with its cameras, microphones, and less-than-cozy feeling. It does however require more pretest preparation.