Methodology

How we do it: usability testing

Purpose

An OCLC product should be easy-to-use and easy-to-learn (i.e., usable) by the user of the product. This procedure outlines OCLC's implementation of an industry standard method for developing this usable product. The method, usability testing, identifies problems in the ease-of-use and ease-of-learning of a product and provides solutions to those problems early in the product's development.

When to do a usability test

Usability testing can be applied at every stage of product development. It is particularly effective in the initial stages of development (e.g., requirements gathering, product design) when changing a product's user interface or design is less expensive.

What to test

When the user interface component of a product is available and can be easily and inexpensively modified, this component can be usability tested. However, it is often difficult and very expensive to change the user interface of the actual product. Hence, a more cost-effective approach is to test, in the initial stages of development, a prototype of the user interface. The prototype may range from a sophisticated pseudo-product, created with third-party software, to an unsophisticated series of screen drawings (research indicates that the sophistication of the prototype has little impact on the validity of the usability data).

Procedure

There are six steps in using the OCLC Usability Lab.

1. Complete Request for Usability Evaluation (RUE) form

The OCLC client, working in conjunction with an HCI team member(s), completes the RUE form. The completed RUE provides information about hardware, software, and help facilities needed for the test. On the form, the client indicates whether supplemental methods of evaluation will be required (e.g., a user questionnaire) and supplies a user description, list of test tasks, and a Usability Question(s) and Quantitative Measure(s)for each task. This form is completed once for a product, unless the information requested on the form changes substantially.

2. Meet with the HCI team

After the Request for Usability Evaluation is completed, the client and HCI team plans the usability test. Test procedure responsibilities are assigned and test details are finalized, including who will contact and schedule the users, and when the test rehearsal will be conducted.

A second meeting is often necessary to review the test tasks. This includes reviewing the wording of the task, the Usability Questions each task is to answer, and the Quantitative Measures used to determine how the user "answered" the Usability Question. An example Usability Question might be Can the user easily find the Search button? Its corresponding Quantitative Measure might be User completes task in one minute or less.

3. Conduct test rehearsal

A rehearsal of the test is conducted in the Usability Lab. The rehearsal is identical to an actual test except (1) user characteristic requirements are relaxed, and (2) user behavior is not recorded or analyzed. Any test equipment or procedural problems encountered during this rehearsal are resolved before the actual usability test. This rehearsal is conducted once for a product unless there is substantial change in the product or testing procedures.

4. Conduct a usability test session

During a typical test session, a user attempts to complete a set of tasks using the product or procedure being tested. The users' comments and behavior are recorded on Videotape.

After completing the tasks (or if the test is stopped for other reasons), the user typically completes a questionnaire and is interviewed by a product team member, an HCI team member or by an HCI team member together with a product team member. Typically, three Usability Lab staff members are present to conduct the test. The product development team members should be present during the test.

After the test session, the client and HCI team members often discuss potential problem areas and possible solutions. The videotape record of the session is cataloged and may be checked out by the client for review (allowances can be made for a more rapid review if necessary). Several test sessions may be conducted on a version of a product, forming a test iteration. A minimum of three tests per iteration is recommended.

5. Create and distribute analyses

For each test, an HCI team member transcribes the questionnaire and interview comments and saves them in the Ulab archives. He/she also writes a brief analysis of the usability test. This "Quick Test Summary" records the task during which a usability problem was observed, a brief description of the problem, and one or more suggested solutions. The analysis also contains Web links to the transcribed questionnaire and interview comments.

In addition, an Executive Summary of all the tests for each iteration or all test iterations is provided. It tabulates the Quick Test Summary results. In addition, summary statistics for the Quantitative Measures are compiled.

It is recommended that copies of the written analysis and/or executive summaries be distributed to all relevant staff. The client is responsible for this distribution. The client and the Director of Quality Assurance also receive copies of the written analysis and/or executive summaries. The HCI team is responsible for this distribution.

6. Complete Usability Lab Effectiveness Questionnaire (ULEQ)

After all testing iterations of the product or procedure are completed, the client completes the Usability Lab Effectiveness Questionnaire (ULEQ). The ULEQ asks the client to evaluate the procedures used in the usability tests and suggest areas for improvement. Most importantly, the ULEQ asks the client to estimate the value of the usability tests to improving the quality of the product or procedure. In essence, it is a way for the HCI team to evaluate the usability of the Usability Lab.

Glossary

Client
The person(s) requesting the usability test.

HCI team
The Human-Computer Interaction team. This is OCLC staff who are responsible for running the Usability Lab. HCI team members have knowledge and skills in the areas of prototyping and user interface design.

Iteration
The usability evaluation of one version of the product or procedure. An iteration typically consists of several test sessions.

Prototype
A version of the product with user interface attributes but very little (if any) coding. Paper screen captures are an example of a prototype. Prototypes are often usability tested because they contain the essential user-apparent features (and thus provide accurate usability data), are available early in the design process, can be easily modified, and are inexpensive.

Test Rehearsal
A run-through of a test session conducted before the sessions with the users. This rehearsal is conducted with an HCI team member acting as a user, and allows the correction and fine-tuning of test procedures before an actual test session.

Test Session
The attempt by a user to complete pre-assigned tasks with the product or procedure to be usability tested.

Usability (ISO Definition)
The effectiveness, efficiency, and satisfaction with which specified users achieve specified goals in particular environments.

Usability Lab
A facility specifically designed for evaluating the usability of a product. The OCLC Usability Lab contains video, audio, and computer equipment for the archiving and analysis of a user's behavior while using OCLC products.

User
The person(s) who attempt to complete a set of pre-assigned tasks using the product or procedure. The user matches as closely as possible a set of user characteristics developed by the client and the HCI team.

References

Dumas, J. and Redish, J.(1994) A Practical Guide to Usability Testing. Norwood, NJ: Ablex Publishing Co.

Winograd, Terry (1995, June). From Programming Environments to Environments for Designing. Communications of the ACM, 18(6), pp. 65 - 74.