Skip to page content

How we do it: heuristic evaluation

Fourteen heuristics  |  Scales of severity and extent

1. Create something to evaluate

We first create a prototype, using paper, HTML, etc. We use usability and design guidelines, and involve at least one informal reviewer.

2. Determine the method of evaluation

We select from three methods when deciding how an evaluator should do her/his evaluation:

  1. Develop a set of tasks for evaluators to attempt. This is our most commonly used method. The tasks should reflect the most critical elements to evaluate, such as those elements expected to cause the most difficulty, elements expected to be used most often, elements that can be fixed within the schedule, etc.
  2. Provide evaluators with the goals of the system and allow them to develop their own tasks. An example goal might be users should be able to find information about FirstSearch online. This method allows greater coverage of the system and is more similar to what real users might do, However, it does not provide the systematic feature comparisons that a single set of tasks done by all evaluators may allow.
  3. Ask evaluators to go through the interface several times and inspect the various dialogue elements. This is most useful for evaluators familiar with the domain of the product. It may provide (across evaluators) the greatest coverage of system elements, but also decreases the level of system feature comparisons.

3. Select the evaluators

We select at least 3-5 evaluators. The more evaluators, the more problems are discovered, but the benefit/cost ratio decreases at about five evaluators [3]. Evaluators should not be associated with the project. Those with user interface or domain expertise find a greater percentage of actual problems (65%) and suggest a greater percentage of improvements (88%) than do developers (24% and 4%, respectively) or nonexperts (12% and 8%, respectively) [3].

4. Do the evaluation

Using one of the methods from section 2, we ask the evaluators to review the UI. It is very important that the evaluators not discuss among themselves the problems they found during the evaluation. They should look for points in the prototype where they are confused or feel the user would be confused. These points should be described, evaluated for severity, extent, and the heuristic that was violated noted. It is also important to suggest a solution, if your evaluators are comfortable making a suggestion.

The evaluator or an observer of the evaluator may record this information. We use a Microsoft Access database and form to record and compile the results, but a paper and pencil work too!

5. Compile the results

We compile the results in a post-evaluation meeting of the evaluators. Every problem identified by an evaluator is tacked to a bulletin board, usually sorted by severity. As each of the most severe problems is presented, the same problem found by a different evaluator is grouped with it, regardless of its severity rating. This is done for each of the remaining severity levels.

Same-problem groups are then discussed and reduced to a single problem description, a single severity rating, and one or more suggested solutions.

The final step is to re-group each of the problems by severity, and/or by the same problem type or evaluator task. This choice is usually made based on how the development team wishes to approach correcting the problem. For example, do they want to first correct (1) the worst problems (severity), or (2) all 'Navigation' problems (problem type), or (3) all 'Doing a Search' problems (evaluator task) ? We always include a 'severity' group, followed most often by a 'problem type' group

6. Deliver a summary report

In our summary reports, the Introduction presents a brief description of the product and how the heuristic evaluation was conducted.

In the Results section, we begin with a discussion of the good features of the product. We also commend the product team for allowing others to criticize their work. This is followed by a table that presents summary results and suggested solution, such as:

Forty-five reports presented by the evaluators were categorized into four Problem Types: Search, Navigation, Terminology, and Graphics. The number of reports by Severity and Problem Type are listed below:

  Problem Type








Severity
 
Search
Navigation
Terminology
Graphics
Total
1
 
1
 
1
2
2
2
4
 
7
13
3
1
6
4
11
22
4
 
 
3
2
5
5
 
 
3
 
3
Total
3
11
10
21
45

These 45 problem reports were grouped into 11 distinctly different problems. Below are descriptions and suggested solutions for each of these 11 problems.

However, use in your report whatever works best to accurately convey the results of the evaluation to your particular audience.

7. Provide a reference section

Lastly, we list references for our statements, methodology, and suggested solutions. This gives clients greater confidence in our findings, and also supplies the sources they need to do their own evaluations in the future.

References

  1. Jeffries, R., Miller, J. R., Wharton, C., and Uyeda, K. M. 1991. User interface evaluation in the real world: A comparison of four techniques. In Proceedings ACM CHI'91 Conference (New Orleans, LA, April 28-May 2): 119-124.
  2. Karat, C., Campbell, R. L., and Fiegel, T. 1992. Comparison of empirical testing and walkthrough methods in user interface evaluation. In Proceedings ACM CHI'92 Conference (Monterey, CA, May 3-7): 397-404.
  3. Nielsen, J. (1994). Heuristic Evaluation. In Usability Inspection Methods. Nielsen, J., and Mack, R. L. (Eds.). Wiley and Sons, New York, 1994, pp. 25 – 62.