The Investigation Process Research Resource Site A Pro Bono site with hundreds of resources for Investigation Investigators | |||||
Home Page | Site Guidance | FAQs | Old News | Site inputs | Forums |
INVESTIGATING
INVESTIGATIONS to advance the State-of-the-Art of investigations, through investigation process research. Research Resources: Search site for:: Launched Aug 26 1996. |
More typically, the NTSB discusses safety problems in terms of "issues" in its outputs. For example, in its 1988 Annual Report it describes issues in each of the modal discussion sections. Procedures and justifications for selecting the issues from among all the safety problems it viewed since the last similar list are not described in the sense suggested by the questions in the Analysis and Justification form above. The report gives no indication of the risk levels associated with each issue. The issues may be based on the predicted possibility of recurrence of a similar accident described in the NTSB's recommendation follow-up requirements, but the basis for their selection (and predicted safety value if implemented) is not provided. Recommendations Produced
through July 25, 1990
During the investigation process, any staff member can propose any safety recommendation, if is "justified and adequately supported by the findings of accident investigations or conclusions or reached as a result of safety studies." Each proposal must be assigned a "follow-up priority" before it is launched into the development or Review Board steps on its way to the NTSB Member-level docket item approval process. When the NTSB Members adopt a recommendation, one of three follow-up classification is assigned to the recommendation, as shown in Table 2.
While not explicitly stated in NTSB documents, the Order indicates that the main purpose of the follow-up system is to indicate follow-up urgency by the NTSB for each recommendation, based on criteria for assigning a follow-up class. Table 3 on the following page shows the follow-up classes assigned to recommendations approved by the NTSB since 1975.
Note: Excludes 1853 recommendations issued prior to mid 1975 to which no follow-up class was assigned.
Evaluating the Recommendation ProcessIs this kind of process effective? The answer depends on criteria used to judge the process effectiveness. The criteria "fully justified" and "adequately supported," used by the NTSB for this recommendation process, are too abstract to be considered useful criteria for this purpose. However, such ambiguous or abstract kinds of criteria are typical for accident investigation programs observed by this author[10]. How then can the process be evaluated? Results-based Effectiveness Evaluation Nobody knows. First the problems are not defined in a manner that lends itself to measurement of the correction. Secondly, with few exceptions,[11] the safety results of the recommended actions - predicted over the remaining life cycle - are not documented in observable terms, and thus remain unknown. Determining the recommendation effectiveness record would take an extraordinary amount of research because the generation of relevant safety performance data has not been a part of the NTSB or related processes.[12] The uncertainties shown by these questions can be observed in most investigation programs that produce safety recommendations. In the case of the NTSB, careful examination of the NTSB's recommendation follow-up classification process suggests the reasons. That process is geared to the possibility of recurrence of similar accidents.[13] While question 12 calls completion of action to be measured, the measure of effectiveness is suggested to be recurrence of similar accidents over time. Effectiveness is not predicted in terms of the exposures, and the combination of the probability of recurrence and the probable consequences, or the risk. The NTSB recommendation process does not require or report, in terms managers or the public can understand, the observed risk[14] associated with a safety problem it finds. Further, the predicted reduction in risk that will be achieved if the recipient of a recommendation does what the NTSB proposes is not reported either. By not specifying the risks attached to either the safety problem or proposed action, the safety value of the recommendations can not be judged, and priorities for its implementation must be inferred. What then is the basis for determining if recommendations should be implemented, and if so, in what order? This raises questions about a national policy perspective. If the relative priority among 8115 recommendations can not be determined, how is the competition for resources to implement those actions supposed to be resolved except on an ad hoc basis? Pity the recipients! This deficiency escalates the probability of misdirecting energies and resources to "the squeaking wheel of the day," which may be a low-risk problem, rather than toward serious, properly defined, long term high-risk problems. Similarly, for an industry or regulatory agency perspective where competition for or concern about resources existed, the relative priorities among the 2714 aviation safety recommendations, for example, can not be determined. Are marine risks really as low compared to pipeline risks as suggested by the percentage (1% vs 19%) of Class I Urgent priority recommendations in these modes? What actions should be funded - marine or pipeline, and when, if the basic desire is to be responsive to the most important safety needs with available resources? The absence of safety risk data ultimately leads to poorly informed or misdirected judgment calls, because judgments are based on differing experiences and unclear safety improvement values. Alternatively, it leads to avoidance efforts based on such judgments.[15] Without an indication of the changes in risks, what technical basis remains to resolve uncertainties that arise in such circumstances, in any organization? System Safety-based Effectiveness Evaluation System Safety Evaluation Criteria 1. Life cycle risks. 2. Working Group or Team effort. 3. Predictive analytical technology. 4. Tailored, timely analyses. 5. Package for decision making. 6. Use proven control strategies. 7. Monitor your hazard predictions. 8. Track problems until resolved. System safety technology, applied predictively, offers a structured, disciplined management and technical process for defining safety problems and developing predictive safety action recommendations. Safety problem discovery and definition are one of the goals of the predictive analytical technology used in the system safety process. In system safety documentation, such as a Preliminary Hazard Analysis, each safety hazard or problem is listed. Each problem on the list is typically followed by an estimate of the risk level it poses, using a Risk Assessment Coding scheme, and by one or more action options that would eliminate or control the risk (e.g., a safety recommendation) along with the new RAC if the option is implemented. That process can be used as a yardstick for assessing the recommendation development process.
NTSB Recommendation Process EvaluationA review of the NTSB recommendation process as described in its Order 82 using these criteria suggests the following observations: Evaluation of NTSB Recommendation Process
Overall, Table 4 suggests that the process is not very effective, by most system safety criteria. Each criterion raises an area of concern about the process. However, basic deficiencies, from which others flow, seem to be:
Problem and Risk Level Documentation This is not to say that risks are ignored in the recommendation process. For example, an analysis of the contents of NTSB's Order 82 and its investigation reports suggests that the NTSB considers risk involved in accidents it investigates in several ways. It could be argued that the risk posed by the problem the recommendation addressed had an influence on the follow-up class NTSB assigned to its recommendations. Based on this premise, the risks addressed by the 6262 recommendations assigned a follow-up class under this process are summarized in Table 5. However, in the absence of specific instructions relating the risks to the classification, the implicit risk levels in Table 5 should be viewed as speculative. Inconsistencies among modes and Table 5 suggest that no risk-based criteria are intended for recommendations.
The NTSB also considers risks when it makes interim recommendations to address any urgent problem it finds in an investigation, by the assignment of urgent processing priorities, as described above. It could also be argued that when the NTSB issues an interim recommendation, it considers the risk involved so unacceptable that Members want to act quickly to control it, although the Order uses different terms. The time limits for preparation of an NTSB decision package for its consideration of an "urgent" safety recommendation proposal is 15 days after assignment of a proposal to one of its internal Bureaus, 75 days for a "priority" proposal, and 120 working days for a "longer term" proposal. However, the follow-up period is substantially longer, as shown in Table 2. These time limits seem to be predicated on the organization's desire to get action from the recipient rather than getting the problem corrected, perhaps on the assumption that its recommendation will solve the problem. The 90 day response period allowed under the law further diminishes the urgency of a response even to an urgent problem. Because the risk level posed by the problem is not made explicit the impetus for action is driven by the focus on closing the recommendation file, rather than the solved problem. An "Urgent" recommendation follow-up classification conveys a different message to recipients than a risk level rating denoting a catastrophic risk level for a hazard or safety problem. Focusing on action taken on recommendations rather than the safety problems or hazards and the reduction in risk needed would introduce a strong tendency to gloss over problems if you have no recommendation to offer. Many field accident investigations produce no recommendations[17]. Recommendation Selection and Priorities The NTSB Order provides no formalized process for reporting corrective action choices, their derivation, and the relative effectiveness of each option it considered, or the other factors by which it arrives at its recommendations. Thus, the NTSB circumvents the trade-offs and hard choices among different ways to fix a problem faced by recipients of its recommendations who must act on them. The large number of recommendations it makes suggests that rather than dealing with options and making its choices on the relative merits, it recommends everything it can think of, including recommendations that might be characterized with a "nice to do" label[18]. Additionally, the conventional approach of recommending a single action implies that the recommender of an action, like the NTSB, is capable of determining the correct solutions to the problems it defines in its investigations. If the options considered and new risks are not reported, and the tradeoffs in the selection of the action recommended are not presented, operators wanting to solve the problem may have little assurance that the proposed action is truly the "best" remedy. Such uncertainties breed hesitation, duplication of analyses, delay and ultimately controversy.[19] Again, pity the recipients! When given a choice, the "best" corrective actions to reach a specified risk level are often best identified, selected and implemented by line personnel closest to the operations or resources - the regulatory agencies or operators who must identify and then balance all the factors that must be weighted and weighed in arriving at the "best" action, and live with the results. But the NTSB, like many other recommending organizations, does not consider economics, for example, in selecting the "best" action to recommend. By shifting its focus from its recommendation to the problem definition and risk levels involved, the NTSB could trigger the creativity of a lot more people who have a direct interest in fixing the problem in the "best" way. From a corporate or national policy perspective, focusing on the problem and presenting safety effectiveness-ranked options with indicated risk reductions would seem far superior to the present system of focusing on a recommendation. Misdirection of Follow-up Efforts It could be argued that the NTSB's recommendations will prevent all similar accidents if implemented, e.g., achieve 100% safety for that kind of accident. Continuing accidents suggest otherwise. Unfortunately, verification of that argument reflects a retrospective approach to safety, and with the advances in safety analysis technology over the past 20 years, has little merit since accidents have to happen before the NTSB gets involved again. Further, they can continue to happen after it closes its recommendations, if either the recommendation was wrong, or if the monitoring action taken over the remaining life of the system is misdirected or ineffective. This deficiency can be exacerbated by the way a safety problem is stated, as when an judgment of cause is stated by the investigator. If a recipient's action is not readily seen to correct the "cause" the manager is likely to encounter trouble - or possibly even bad press if "the public interest" is involved- for not implementing a recommended action. The recipient's choice of the "best" overall action must weigh ALL risks, including financial, social, schedule and political risks. The result may be to select the investigator's second or third ranked preference. However, if the recipient is offered no options, the investigator places the manager in position where a third-rated action may have to be taken because of the cause problem. Accountability for Recommendations Made Most investigators will argue the position that the trade-offs are for the recipient to analyze, especially the economics of the action. This paper does not argue that point. What is unacceptable for the recommending entity - whether at the local plant level or at a body that views itself as the national overseer of transportation safety - is for it to shirk the routine prediction and publication of at least the safety effects of its recommendations.[21] The potential for arbitrary and capricious making of recommendations without such assessments demands that the choices and trade-offs considered be documented, preferably including the safety level before and the predicted safety level after a recommendation is implemented. If the risk levels and options are being properly considered, documentation should be no problem. Alternatively if they are not being considered, they certainly should be before a particular option to recommend is chosen. Any other basis for judging performance effectiveness is suspect, at best.
Correction of these problems requires two steps. The first is a change in an organization's approach to its recommendation process. The second is implementation of the change with a practical system of assigning relative risk estimates to the problems it identifies and recommended action it proposes. Both steps are readily achievable by the application of system safety concepts, principles and tools. Changing the Approach To illustrate the approach, any organization -including the NTSB- could define the operation and document the specific safety problems it finds during an accident investigation. It would define the problem in terms of the current estimated risk level if no action is taken (which it considers unacceptable as evidenced by its recommendation). It would then report the options it has identified to demonstrate that the risk it has defined is controllable, showing how much each option is expected to reduce the safety risk. Then it could simply recommend that the operator or regulator act to eliminate or reduce the risk to an acceptable level. The estimated current risk level would indicate the priority for action (and, concurrently, follow-up action by NTSB and the operator or regulator.) However the selection of the "best" option, would be left to others who must consider more than the safety consequences of the actions.[22] Then investigating organizations such as the NTSB would be free to establish a specific follow-up process and monitoring plan for each safety problem, rather than its recommendation, to see that the corrections introduced really fixed the problem as it was defined. Occasionally, the recipient might argue about the estimated risk level. The dialogue flowing from these differences could be used to refine either the estimating process or the levels that are acceptable. At other times, the recipient organization might argue that it is willing to accept the risk. There isn't enough money in the world to fix ALL safety problems by tomorrow morning! Resolution of this kind of difference could be aided by another system safety concept - the system safety working group. With this approach, representatives from major interest groups participating in the recommendation development process, and airing differences early in a process, would help the recommending individual reach better decisions. Implementing Changed Approach A second change is to institute a requirement for investigators and others who propose recommendations to analyze and assign risk levels to problems or hazards and effects of corrective actions. A practical method for the investigators and recommendation developers is immediately available. It is a Risk Assessment Code assignment process widely used in the Department of Defense and other agencies and their contractors. It is described in greatest detail in MILSTD 882B, the Department of Defense's standard for System Safety analysis. The risk assessment code (RAC) assignment process has been used widely enough that it has proven to be a viable way to distinguish relative risk levels, to aid in the determination of the acceptability of risks, and to assess the relative safety value of safety actions to control unacceptable risks. By adopting the RAC system, organizations can establish problem risk level ratings for the problems they discover and the corrective action options they define in their investigations. Use of RACs could be expected to have several effects. The first effect would be to provide an indication of the relative significance of each problem the investigation has defined as a result of its investigations. For example, the NTSB could state a need and assign a RAC number to the risk level represented by that need. This would enable everyone with an interest in the problem to see and to use the risk level for determining the urgency of their own response and long term monitoring actions. A second effect would be that the "before and after" RACs would provide a basis for comparing the relative safety effectiveness of actions proposed by the recommendation developer or other options offered by other parties. This would overcome the stifling effect of single options as now practiced, or the proliferation of recommendations that sound nice, but would have little discernible effect on the risk levels. A third effect would be to provide a reasoned, relatively consistent basis for recommending staff follow-up actions to spur action on problems or solutions, if required. The urgency of follow-up action, and possibly the design of the follow-up plan itself[24], are far different for a RAC 1 (very high) risk than for a very low RAC 5 risk. This would help shift the emphasis from just answering the recommendation to solving the safety problem. It would also change the nature and approach to data gathering for the purpose of monitoring the achievement of the reduction in risk. A fourth effect would be to provide a basis for evaluating the effectiveness of the recommendation development organization itself. This effectiveness should be judged by the support it provides to line managers in controlling the risks, rather than the count of recommendations with a positive response by the recipient. By comparing predicted with actual results, the effectiveness of both the recommendation and the recommending organization become visible. For example, as an investigating and recommendation development organization, under the new approach the NTSB would not be judged on how many recommendations it made or had accepted. Rather, it would be judged in the context of a knowledge-building organization, on the importance of the safety risks it discovers and defines, and the risk reduction effeectiveness of its recommendations. The sting of such accountability could have a dramatic effects on any recommendation-producing group in any organization. It can be argued, with justification, that the RAC method is less than a perfect solution to estimating risks. However, RAC use in decision making is already so widespread that this argument should not be allowed to interfere with its adoption to overcome the deficiencies described. Implementation of the RAC assignment process may result in frequent negotiations about the RAC that should be applied in particular instances. In the example, since the NTSB is a deliberative body of Congressionally-approved Presidential appointees, the Members' negotiations among themselves or with others could produce valuable criteria for the "right" RACs over time.
Conclusion
Processes for the investigation of accidents and development and implementation of safety recommendations, as presently practiced, have numerous deficiencies when judged by safety performance and by system safety criteria. The evaluation disclosed four serious deficiencies:
These deficiencies deprive decision makers of a basis for judging the urgency for action, the validity of a recommended action, or risk acceptance thresholds; delay or misdirect action; create trade-off analysis problems; compel duplication of analyses; deny needed information to support performance monitoring efforts; misdirect assessments of recommendation effectiveness; create problems for managers; and contribute to unnecessary controversy, among other consequences. By applying selected system safety concepts, principles and tools, the accident investigation recommendation process can be significantly improved in any organization, and can help overcome the deficiencies cited. These actions include
In summary, to improve the investigation and recommendation process, focus on safety problem definition, recommendation options, risk level documentation, and safety performance monitoring follow-up efforts, and relax the emphasis on "cause" and recommendations accepted.
About the author: Ludwig Benner is currently engaged in safety risk analyses and investigations, with experience in managing advanced system safety analyses and accident investigation projects, developing and instructing risk assessment and accident investigation courses for the Army, GSA, U.S.DoT, Navy and private sector, and investigation and decision research. He is the co-author of INVESTIGATING ACCIDENTS WITH STEP, a system safety-based investigation system, and has over 60 publications in the safety field. He career includes positions with the University of Southern California, the NTSB, and Air Products and Chemicals. [1] For the purposes of this paper, a safety recommendation is viewed as a proposed action to address a safety problem discovered by the investigation of accidents., either after or before they happen. [2] For a fuller discussion of this difference, see Hendrick, K.M. and Benner,, L. INVESTIGATING ACCIDENTS WITH STEP, 1987, Marcel Dekker, NY p 197. [3] Wood, , R.H., HOW DOES THE INVESTIGATOR DEVELOP RECOMMENDATIONS?, Proceedings of the 10th International ISASI Seminar, September 24-27, Quebec, Canada. [4] Bruggink, G.M. and Fritsch, O., THE SAFETY RECOMMENDATION PROCESS, ISASI forum, November 1989, V22:3, p 1. [5] Hammer, W. HANDBOOK OF SYSTEM AND PRODUCT SAFETY, 1972 Prentice Hall, Englewood Cliffs, NJ, p 251-296. [6] This does not suggest that system safety principles can not be applied to the process, but only that the process has not been documented in that standard. [7] From POLICY section of NTSB Order 82, 6/11/87, p 1. [8] See NTSB News Digest, Volume 9, No. 10, October 10, 1990. [9] The law allows a 90 day response period from the U. S. Department of Transportation. [10] See Benner, L., RATING ACCIDENT MODELS AND INVESTIGATION METHODOLOGIES, JSR 16:, pp 105-126, 1985. During that research, terms such as "clear" or "concise" were observed frequently. [11] See for example the prediction of 95% effectiveness in the reduction of accidents in Department of Transportation Docket HM 44; subsequent accident observations and statistical results indicated that the predicted safety effectiveness of the recommendations reflected in this rulemaking were achieved. [12] The NTSB 1988 Annual Report to Congress is cast more in terms of function than safety achievement, and does not contain information to permit an assessment of safety achievement of its recommendations, either individually or in the aggregate. In this respect, the NTSB program is typical of most safety programs. Few recommendations claim safety results - except in the most general way. Aggregate accident counts or rates can rarely be related to specific actions. For example, the NTSB publishes annual statistics purporting to describe transportation safety performance, but does not claim that any of its specific actions produced that record. [13] The same thought is also expressed as "accident prevention" in other investigation programs. [14] Risk is used in the sense of a predicted chance of loss associated with the predicted chance of the gain from a proposed action. In the context of a safety recommendation, a decision to act on a safety improvement recommendation must consider the safety and related gains likely to follow the decision, and balance that gain against the loss or costs likely to follow the decision to act or not to act. This definition should not be interpreted to infer a "cost-benefit" analysis in economic terms, since other social values tradeoffs are usually involved. [15] For a discussion of this phenomenon, see Hendrick, K.M. and Benner,, L. INVESTIGATING ACCIDENTS WITH STEP, 1987, Marcel Dekker, NY p 364-365. [16] See, for example, MILSTD 882B, Hammer's book and others available through the System Safety Society for a description of system safety management and engineering concepts, principles and procedures. [17] At the rate of 2700 investigations per year, one recommendation per accident would have resulted in 40,500 recommendations, rather than 8000. [18] For example, in one report, the probable cause was the failure of a train crew to properly secure their train by placing the train brakes in emergency, and applying hand brakes when it was left standing on a mountain grade. Contributing factors included decisions by a Helper locomotive engineer, effects of cold on air brake system and crew and crew members, management assessment of employee qualifications and employee training and hazardous materials. Ten months after the accident, well into the next winter season, the NTSB adopted 25 new recommendations and reiterated 3 previous recommendations. All 28 recommendations carried the NTSB's Class II Priority Action designation. (NTSB /RAR-89/05) [19] The NTSB provides for acceptance of alternate action, but unless relative safety effectiveness is predicted and disclosed, it can not be monitored and managed properly. [20] The FAA, for example, by law has 90 days to respond to an NTSB recommendation, regardless of the urgency of the problem. If the problem, rather than the recommendation were given priority ratings, the response time could be tailored to the problem by whoever is in a position to correct it. [21] The NTSB has done this in a few instances in the past, with significant effect. For example, the NTSB's recommendations and activities which led to DOT docket HM-44 requiring head shelf couplers and head shields predicted the degree of improvement over time. [22] The NTSB has a precedent for this approach when it closes a recommendation file with its "Closed,-Acceptable Alternative Action" designation. [23] For example, the US Department of Energy specifies that a section on "Judgment of needs" be included in its accident reports. This kind of a section could be adapted to NTSB reports. See p 129, U. S. Department of Energy Accident Investigation Manual, DOE/SSDC 27, 2nd Edition. [24] An operational safety performance monitoring plan for predicted hazards is an integral part of system safety practices, but not often addressed in NTSB recommendations. |