Using the Cognitive Walkthrough for Operating Procedures

David G. Novick

EURISCO

Your system's interface is great. The users love it. Except that if they don't use it correctly, bad things could happen. Maybe people's lives are at stake. So you add a set of operating procedures for users to follow in order to ensure that the outcomes are good. Generating plants, hospitals and aircraft cockpits are examples of settings where procedures are used to reduce risks to extremely low levels. For instance, air-traffic controllers have many procedures, including standard time intervals and a spatial pattern for visually scanning their screens.

Or maybe you've developed a kiosk interface for the general public, perhaps to charge fees at a parking garage. The kiosk has, stenciled on its front, step-by-step instructions for use. Of course, even if things go badly wrong in following the instructions, no one will get hurt. But the garage could lose customers if they keep getting frustrated or lose money, and if people tend to get confused while using the system, an increasingly long line of waiting patrons may show their impatience. The step-by-step instructions consitute an operating procedure, and you hope you've got a good one.

So how, absent extensive late-process usability testing, can improve your system's operating procedures so that safety and effectiveness are increased?

Jonathan Grudin observed [3] that the user's interface to a computer is not just the hardware and software that compose the computer. From the user's point of view, the interface includes an array of associated elements in the context of use, including documentation, training, and advice from colleagues. From this perspective, one can view the interface of a computer as including the usual screens, buttons and knobs, plus other users present, training received, documentation provided and operating procedures prescribed. Indeed, it turns out that the guiding of users through their interactions in a computer interface can largely be allocated either to the interface itself or to a set of associated procedures. This is what Guy Boy [1] called the interface-procedure duality. To take a simple example, if the stenciled instructions on the garage kiosk were displayed on a computer screen, it would ordinarily be considered part of the "computer" interface.

Consider designing procedures and documentation for a new, text-based air-traffic control communications system for aircraft to complement the current voice-based system. Figure 1 presents an example of a typical draft procedure that I developed for this kind of "datalink" interface. How could I assess, at design time, whether this draft procedure and the many others associated with the datalink interface would be effective?

Figure 1. Example of a procedure
[Figure 1]

The correspondence between interfaces and operating procedures suggests a solution: adapt known evaluation techniques (as applied to interfaces) to the evaluation of procedures (considered part of the interface in the wider sense). Accordingly, this article shows how to use an adaptation of the cognitive walkthrough [7], as revised and extended specifically for the evaluation of operating procedures. I will.discuss how I adapted the regular cognitive walkthrough into a cognitive walkthrough for procedures (CW-OP) and show an example of CW-OP in use.

Adapting the Cognitive Walkthrough

The cognitive walkthrough, as it has evolved [4, 7, 8], is a usability inspection method for interfaces that originally focused on evaluating a design for ease of learning and that has been extended to other phases of interaction. The cognitive walkthrough has been adapted to the evaluation of complex interfaces such as a CAD tool [13]. The method leads the designer to consider factors such as usersŐ backgrounds and mental effort. It is based on a model of interaction like that of NormanŐs stages of user activities [5]. In brief, a task is decomposed into steps in the interface, which are analyzed individually with respect to connections between goals, artifacts, actions, and results. Each step is described as a "success story" or a "failure story," depending on the outcome of the analysis. Walkthroughs can be conducted either as a group or individually. Designs evaluated in this way should, among other things, reduce the need for training. An extensive literature, summarized in [7], has examined the relative effectiveness of the cognitive walkthrough in identifying usability problems.

From the most practical version of cognitive walkthrough [8], I developed, with help from my colleagues at EURISCO and Aérospatiale, a new version adapted to operating procedures. This involved an iterative process of revision and use of forms and instructions adapted to procedures and their documentation. The adapation of the walkthrough involved taking into account information and constraints that are present in the interface's context of prescribed and actual use. While a physical interface (that is, an interface in the hardware/software sense) is designed to specifications that account for things like the nature of the users and the task, an operating procedure includes these as specific elements. A procedure exists to specify, unambiguously,

Accordingly, the adaptation to procedures included five key changes:

How to Use the CW-OP

The CW-OP, like the cognitive walkthrough for the physical interface, is supported by the use of two forms: (1) a cover sheet and (2) a form that presents the success or failure story of each step analyzed. Figure 2 presents the contents of the cover sheet, which in practice is a normal A4 or letter-sized page. Figure 3 presents the content of the form for each step; in use this form is also a normal-sized page. The interface needs to be well-specified for the the evaluation of its associated operating procedures to succeed. Note that this may limit the method's usefulness during early phases of development.

Figure 2. Contents of the Cover Sheet
[Figure 2]

Figure 3. Contents of the procedure-step form
[Figure 3]

The CW-OP is normally conducted using a group of evaluators. A study validating the CW-OP [6] found that the CW-OP was comparable to the regular cognitive walkthrough in terms of burden on evaluators, and that the CW-OP process clearly identified issues involving the procedural as well as the physical interface. In most cases, evaluators agreed in their assessments. However, as is the case for the cognitive walkthrough for the physical interface, it is considered sound practice to use multiple evaluators. Evaluations can be performed together as team, or in parallel individually. However, experience suggests that when training evaluators, it is important to have trainees perform some evaluations individually. The idea is to make sure that each person has experience in answering all questions in the CW-OP forms. For example, one trainee evaluator reported difficulty in making the success/failure distinction . For both group and individual walkthroughs, I found useful the recommendation in [6] that the assumed characteristics of users be written in a place where everyone can see them and, if necessary, update them in common.

The team of evaluators should consist of people who have been trained in human-computer interaction and who have knowledge of or access to the specifications or characteristics of the interface being evaluated.The group could benefit from the inclusion of users, particularly if the evaluation is conducted as a team. I want to stress that multiple evaluators bring a variety of experiences with interfaces and procedures, different kinds of knowledge about human factors, and different levels of expertise about the interface being evaluated. The range of evaluations for operating procedures at which we have looked suggests that multiple evaluators can provide complementary perspectives on the usefulness and usability of operating procedures.

You may want to ask evaluators to mark sections of the documentation that they read or used during the evaluation process. Otherwise there may be no record of whether the evaluators used any particular part of the documentation as an aid to understanding the procedures. For instance, the "example" section provided in the draft procedure in Figure 1 would usually not be considered steps subject to analysis, but it would be useful to know if the evaluators had referred to the examples to help them understand the procedure.

Example of Evaluation

The CW-OP was developed as part of a project sponsored by Aérospatiale, the French part of Airbus Industrie. The project involved design and evaluation of a datalink interface. One of the datalink draft operating procedures I developed, called simply "Respond" but too long and detailed to be usefully presented here, consisted of a general techique for creating and sending a response to a message from air-traffic control. This draft procedure had four main parts: build the message, send the message, confirm receipt, and store the message.

Figure 4 shows a cover page filled out for an evaluation that included the "Respond" operating procedure. The cover page indicates that this was one of three procedures comprising a task. For the particular steps, the cover page refers to a specification; it might be better practice to list the specific steps directly in the form.

Figure 4. Example of cover sheet
[Figure 4]

Figure 5. Example of story for a procedure step.
[Figure 5]

Figure 5 shows a form for the first step ("Build the message") in the "Respond" procedure, as produced by an individual evaluator. Walkthrough Questions 1-4 address the step's potential points of failure. In Question 2, the procedure's affordance is analyzed in terms of its representation in the documentation. The evaluator's annotation "experience" indicated the areas supporting the conclusion later in the form that experience or training would be required. The walkthrough observation items 1-3 address concerns specific to procedures, particularly in safety-critical systems. These items are useful even for less-complex systems, too. So if the procedure for a walk-up-and-use kiosk is judged to require training, then the procedure and presumably the interface should be reworked. In this example, the evaluator did not address the "errors" item because it was already evident by that point in the analysis that this step would have to be changed. Observation items 4 and 5 are intended to elicit design alternati ves and any other comments while the step and the overall procedure are still fresh in the evaluator's mind.

In the case of the "Respond" procedure, it was eliminated in the next version of the documentation. The elements of the procedure were incorporated into other, more specific procedures such as "Respond to a message with multiple clearances."

CONCLUSION

Aside from the direct assessment of usability, the cognitive walkthrough for operating procedures provides insight into usefulness and safety beyond that associated with the cognitive walkthrough for physical interfaces. In particular, the CW-OP's requirement of a reason for linking goals to actions leads evaluators to determine whether training or experience are required for correct use of the procedure. In the aviation industry, for example, reductions in training required to operate an aircraft would both increase reliability and lower operating costs.

When using the CW-OP, evaluators should be alert to a couple of areas not explicitly handled in the process. First, evaluators tend not to evaluate the the names of the procedures, presumably because they are not steps as such. These names, though, are usually users' main means of access to procedures, and evaluators should consider treating the name a step in order to make sure the name's affordance is examined. Second, overall instructions tend not to be evaluated. For example, the draft manual for the datalink interface contained an instruction on when members of the crew should make announcements to each other during procedures. The CW-OP evaluation of the draft datalink operating procedures produced only one comment on this instruction in the session. How to obtain analyses of such overall instructions remains an open question. Similarly, it is not clear how to use the CW-OP technique for evaluation of non- procedural parts of operating documentation. Nevertheless, the CW-OP is being used to evaluate draft operating procedures, and has resulted in specific, useful changes in the operating procedures developed as part of the datalink interface program.

ACKNOWLEDGMENTS

This research was supported by a research contract from Aérospatiale Aéronautique. Thanks to Florence Buratto, Meriem Chater, Laurent Moussault, Florence Reuzeau and Stéphane Sikorski for their participation in and contributions to the research.

References

  1. Boy, G. Cognitive function analysis for human-centered automation of safety-critical systems. Proceedings of CHI 98 (Los Angeles, CA, April, 1998), 265-272.
  2. Degani, A., and Wiener, E. Procedures in complex systems: The airline cockpit. IEEE Transactions on Systems, Man, and Cybernetics, 27, 3 (1997), 302-312.
  3. Grudin, J. interface. Proceedings of CSCW 90 (Los Angeles, CA, October, 1990), 269-278.
  4. Lewis, C., Polson, P., Wharton, C., and Rieman, J. Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces. Proceedings of CHI 91 (Seattle, WA, April, 1991), 235-242.
  5. Norman, D. Cognitive engineering. In Norman, D., and Draper, S. (Eds.), User centered system design. Lawrence Erlbaum Associates, Inc., Hillsdale, NJ, 1986.
  6. Novick, D., and Chater, M. (in press). Evaluating the design of human-machine cooperation: The cognitive walkthrough for operating procedures. Proceedings of the Conference on Cognitive Science Approaches to Process Control (CSAPC 99), Lille, FR, September, 1999.
  7. Wharton, C., Bradford, J., Jeffries, R., and Franzke, M. Applying cognitive walkthroughs to more complex user interfaces: Experiences, issues and recommendations. Proceedings of CHI 92 (Monterey, CA, May, 1992), 381-388.
  8. Wharton, C., Rieman, J., Lewis, C., and Polson, P. The cognitive walkthrough method: A practitioner's guide. In Nielsen, J., and Mack, R. (Eds.), Usability inspection methods. John Wiley & Sons, Inc., New York, NY, 1994.

This paper is in press as: Novick, D. (in press). Using the cognitive walkthrough for operating procedures, Interactions.
© Copyright 1999 by ACM, Inc.

__________
DGN, March 31, 1999
Valid HTML 4.0!