The Investigation Process Research Resource Site
A Pro Bono site with hundreds of resources for Investigation Investigators
Home Page Site Guidance FAQs Old News Site inputs Forums
. . . .. . . . . . . . last updated 4/28/07

INVESTIGATING
INVESTIGATIONS

to advance the
State-of-the-Art of
investigations, through
investigation process
research.



Research Resources:

Search site for::


Launched Aug 26 1996.

 
This is a paper I learned about as a result of Mr. Munson's messages about the IPRR web site. Mr. Munson has granted me permission to post this copy of his paper at this site. It is one of a very few investigations into relative merits and application of alternative investigation methods of which I am aware. Information about other such research, and comments on this paper would be welcomed.

go to Table of Contents
Ludwig Benner
Your IPRR host.

ASSESSMENT OF ACCIDENT

INVESTIGATION METHODS

FOR WILDLAND FIREFIGHTING INCIDENTS
BY CASE STUDY METHOD


BY

STEVE MUNSON



PRESENTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE IN FORESTRY

UNIVERSITY OF MONTANA
1999



APPROVED BY:


------------------------------------------
CHAIRPERSON

------------------------------------------
DEAN, GRADUATE SCHOOL

------------------------------------------
DATE
ABSTRACT



Munson, Steve. M.S. January 2000 Forestry



Application of Accident Investigation Methods to Wildland Firefighting by Case Study Method (112 p.)


Director: Ronald H. Wakimoto



Wildland firefighting is an inherently risky occupation. Firefighter entrapments, burnovers, and fire related fatalities continue to occur on an annual basis throughout the United States. The purpose of this thesis was to identify, from USDA Forest Service recommendations, an accident investigation method that would be most applicable to wildland firefighting. The most applicable method(s) would best be able to pinpoint causal factors and identify areas where future occurrences could be reduced.

This thesis examined a single event as a case study, the 1994 South Canyon Fire. Due to the volume of published material and its position as an extreme case, this fire was determined to be a suitable study. The South Canyon Fire was reinvestigated utilizing each of the U S Forest Service proposed methods. Wildland fire experts evaluated each method according to six criteria. This determined an overall ranking used to determine the applicable method(s).

Results suggest that two methods, The Sequential Timing and Events Process (STEP) and Fault Tree Analysis were acceptable accident investigation techniques. Each method had strengths (and weaknesses) in distinct areas. The third evaluated method, Controls/Barriers Analysis was determined to be not as applicable to wildland firefighter entrapments. Used individually or as a composite/cross reference application, these two methods would be valuable tools in investigating wildland firefighter entrapments.

Research indicates that a composite model that utilizes the strengths of each method would be the most valuable in determining the accident causal factors that once identified, would lead to reduced incidents/accidents in the future. It is recommended the future accident investigations should use both methods separately, in conjunction, and as a third composite method in order to revalidate thesis findings. In addition, this thesis identified the need to utilize a reliable, established accident investigation method for wildland firefighting entrapments. An applicable method would determine causal factors for near misses and accidents in order to track, mitigate, and identify areas in need of revision. These areas must be identified if future firefighter entrapments are to be reduced.



ACKNOWLEDGEMENTS


The author would like to gratefully acknowledge all those people that gave their time and energy to assist in this thesis. First, to Professor Ron Wakimoto, for all his guidance, ideas, and insight into the project. And to the other committee members, Professors Mike Patterson and William Borrie, and Dr. Ted Putnam. Their support and assistance were invaluable and stretched my thoughts and education across other fields. And to my friends and family who offered constructive criticism, proof reading skills, and asked some tough questions, I greatly appreciate the help. Also the grad school partners, Lisa and Kristen, both of which where there on the long road. And finally to Bernie, who has given me more than I could ever repay.




TABLE OF CONTENTS

    CHAPTER 1 INTRODUCTION

    CHAPTER 2 GOALS AND OBJECTIVES

    CHAPTER 3 PREVIOUS WORK AND PRESENT OUTLOOK
ADDITIONAL APPROACHES
USDA FOREST SERVICE PROPOSED METHODS

CHAPTER 4 METHODS
CHAPTER 5 RESULTS

CHAPTER 6 CONCLUSIONS
CHAPTER 7 DISCUSSION
LITERATURE CITED

Appendix A EVALUATION PACKAGE

Appendix B USDA FOREST SERVICE DOCUMENTS LIST OF FIGURES

  1. Accident generation, analysis and prevention process
  2. Five accident perceptions
  3. “Home accidents” diagram of accident process
  4. Events and Causal Factors Chart illustrating analysis process
  5. Events and Causal Factors Charting firefighting accident example
  6. Multilinear Events Sequencing diagram illustrating process
  7. Multilinear Events Sequencing firefighter accident example
  8. Change Analysis diagram
  9. DOD “3W’s “ accident analysis process
  10. Barrier Analysis diagram
  11. Fault Tree Analysis diagram
  12. Fault Tree firefighter accident example
  13. STEP accident information building block
  14. Building Blocks incorporated into the STEP Worksheet
  15. Map of South Canyon fire area
Figure A.1 GENERAL OVERVIEW FOR EVALUATORS
Figure A.2 FAULT TREE ANALYSIS
Figure A.3 STEP WORKSHEETS
Figure A.3 FINAL STEP LAYOUT

LIST OF TABLES

1. Evaluator’s ratings of criteria
A.1. DOE Barriers Table
B.1. 10 Standard Firefighting Orders, 18 Watch-Out Situations, Downhill Line Construction Guidelines
B.2. Wildland Firefighter Entrapment Matrix



CHAPTER 1  -  INTRODUCTION

GENERAL OVERVIEW

Wildland firefighting is an often risky occupation. The buildup of wildland fuels from decades of fire suppression and disease/insect outbreaks has compounded the risk firefighters have traditionally encountered. The accumulation of dead timber has increased the available fuels and fires are becoming larger and more difficult to control. Increasing wildland/urban interface complexities and public demands for rural firefighter protection of property have added to the inherent risks. Since 1976, a study of published accident reports showed 1,589 firefighters have been entrapped by fire behavior related, life-threatening situations (Munson 1998). The National Wildfire Coordinating Group (in USDA 1995), a collaboration of the five U.S. federal firefighting agencies and States’ representatives, defined an entrapment as: “A situation where personnel are unexpectedly caught in a fire behavior related life-threatening position where escape routes or safety zones are absent, inadequate, or have been compromised. An entrapment may or may not include deployment of a fire shelter”. During the period 1976 to 1998, Munson’s (1998) study showed 104 fatalities have occurred and a yearly average of 42 protective fire shelters deployed. The traditional risks to wildland firefighters have become more varied and complex as the incidents and accidents associated with those risks have increased.

As the risks have increased and mishaps continued, the need to identify these risks and reduce firefighter exposure to those risks is paramount. But as Briscoe (1990) warns, “ if absolute safety is literally accepted as having priority, with no acceptance of risk, operation is not possible, for all activity involves some risk.” This thesis suggests it is critical to improve accident investigations to better identify firefighter risks that can be accounted for and managed. An appropriate accident investigation system would identify the hazards; illuminate the areas needing revision, and track progress of preventative measures. From insights such as these management of acceptable, known risks can be aided that can lead to increased safety and effectiveness of firefighting resources.

On July 6, 1994, 14 fatalities occurred on Storm King Mountain (The South Canyon Fire) in Colorado. This event was a primary catalyst for wildland firefighting agencies to reassess safety protocol throughout their respective organizations. It was determined that “fire agencies are not routinely collecting and analyzing data...(particularly) crucial near-miss information on the wide variety of risks inherent in firefighting” (USDA 1995). Wilson (1989) stated that the federal firefighting agencies have made substantial progress in areas of aviation, equipment technology, fire prevention, and suppression tactics. The less tangible, more difficult to determine factors have not had the same focus. Braun and Latapie (1995) mentioned that though we have made progress in these more tangible areas little has been researched in the firefighter environment of human and organizational factors. Federal firefighting agencies have looked at these more physically tangible, identifiable, and correctable areas for solutions to entrapments yet “fail to deal with a major cause of the fatalities (human error)” (Putnam 1995). It is clear that it is time for the federal agencies to introduce operational procedures that better elucidate those human contributing factors. This would be a critical first step in the documenting and investigation of causes.

In order to reduce or potentially eliminate the causes of accidents, investigations must identify the causal factors so that preventative measures can be instituted (Mansdorf 1993). The National Safety Council (1984) defined an accident as “that occurrence in a sequence of events, which usually produces unintended injury, death, or property damage”. The key elements in the definition were unintended results and effects. Perrow (1984) said that this distinguished accidents from willful harm, violations, and incidents (near misses or near serious).* As Reason suggests, violations are of particular interest, as they become an increasing factor in the involvement of accidents (Reason 1995). He adds that they are a deliberate deviation from safe operating procedures and “occur in a social context and involve motivational as well as cognitive factors”. Figure 1 illustrates the relationship between accident generation, the ensuing investigative process, and the resulting prevention measures (Diehl 1991). The figure suggests the fundamental importance of documenting, investigating, and understanding near misses, hazards, and incidents, as well as accidents. It graphically depicts the process of learning from incidents and accidents by thorough investigations that identify the causal factors. Once these factors are identified, preventative measures can be instituted to reduce the chance of future occurrences. Kenney (1993) illustrated this fundamental concept by suggesting that for every fatal accident there were 30 major accidents and 300 recordable incidents and some 30,000 unsafe actions or conditions.

* Though violations can sometimes be considered as purposeful, their outcome was generally unintended. Thus recent research has included purposeful violations (with unintended results) in a working definition of accidents (Reason 1995).

The application of this hierarchical concept to Munson’s (1998) findings on firefighter accidents/incidents showed that for every firefighter fatality there were over 15 major accidents (entrapments). Munson further suggests that over 240 recordable incidents may have occurred, and that there might be over 24,000 unsafe actions and/or conditions that so unreported. The accident reports showed that these hazardous incidents were not routinely investigated or reported (Munson 1998). The closed loop concept of the accident cycle illustrated the need to investigate all incidents in order to circumvent the transition from near-miss incidents to major accidents and fatalities. The closed loop illustrates the circular pathway that proceeds from an accident to the investigative process to determine the causes. Preventative measures are then instituted from the investigation findings until another “leak” in the safety program identifies another hazard, then incident or an accident occurs. The cycle repeats itself as loopholes found in the safety programs are identified, then mitigated, until another loophole is breeched and an unwanted, harmful accident occurs. The safety program is therefore a continuous ongoing process to reduce the harmful outcomes. Therefore in order to be proactive in accident/incident prevention an effective and comprehensive reporting and investigation program would have to be established and maintained.


image
Figure 1. Diagram illustrating the “closed loop” accident process of accident generation to investigation to preventative measures (Diehl 1991).


JUSTIFICATION

Accidents have been described as a process in which a perturbation “transforms a dynamically stable activity into unintended interacting changes of state with a harmful outcome” (Hendrick and Benner 1987). Thus a homeostatic process was interrupted by a disruption that when unimpeded by barriers/controls or subsequent recovery efforts resulted in unintended harm to people or property. There are many reasons to investigate accidents such as faultfinding, fact-finding, the need to evaluate current safety programs and to monitor prospective changes (Ferry 1988). However, the major focus of all accident investigations is to identify where safeguards failed so that recurrence of the accident can be prevented.

“The wildfire suppression organization is inherently complex, both through its internal structure and the dynamic nature of its configuration as it changes to meet the variable complexities of the environment in which it functions”(O’Brien 1997). This inherent complexity, the dynamic environment of wildland firefighting, and low-level worker autonomy, exemplified the need for a specific accident investigation method that incorporated the variable components unique to the profession. The variability of the wildland firefighting environment and the complexities that are routine make this profession dynamic and challenging. The structure of the front line firefighting community is one of responsibility and accountability at the bottom where the fire suppression work is accomplished. Thus the low-level firefighters perform autonomously in an environment they sometimes may not be equipped to handle safely.

As a direct result of the 1994 South Canyon Fire, the five federal wildland firefighting agencies commissioned a study by TriData Corporation (1998). This study was to identify firefighter safety culture and concerns by conducting a survey of over 1,000 wildland firefighters. The study’s goal was to investigate the underlying organizational culture that negatively affected firefighters’ safety. The questions that were developed were used to generate insights into how firefighters did their jobs and how their beliefs influenced how they safely performed. It did not attempt to investigate the five national firefighting organizations themselves but focused on the underlying firefighter safety culture. Of the 86 recommendations listed in the study, the fourth and fifth highest priorities were directly related to wildland firefighter accident investigations. These priorities were determined by responses from firefighters to the survey. The fourth priority recommendation was that “the five agencies should strive to obtain a clear, quantitative picture of the pattern of safety incidents, their causes, trends, and the lessons learned; and to identify potential problems at the earliest time possible.” The fifth priority was to “define interagency protocols for the process and substance of investigations.”

The final report of the Interagency Management Review Team (IMRT 1995) on the South Canyon fire recommended “the USDA and DOI develop improved, coordinated accident investigation procedures” (IMRT 1995).

Ted Putnam (1995), a member of the South Canyon Fire investigative team, cautioned that the more tangible aspects, such as fire behavior, weather forecasting, and fuels inventories received the primary focus by the team and that psychological and sociological perspectives were dismissed as unimportant as possible causative factors. Team members had expertise in fire environment factors and not in issues pertaining to the firefighters themselves. He added that the review team’s focus “fails to deal with a major cause of the fatalities (human error)...and calls into question the very process and structure by which we investigate fatalities and communicate the results to the fire community.” Putnam stressed that human and organizational error in recent wildland firefighter accidents and fatalities were common denominators and their study was overdue.

Previously the National Wildfire Coordinating Group (1980) formed an ad hoc committee to look at firefighter entrapments and fatalities and concluded that since the 1950’s many of the same factors that were in place then as causes were still applicable. Factors such as fire behavior, firefighter qualifications, and communications were cited as recurring problems. Therefore these factors were featured highly in reports. But it was noted that these factors were continually the major focus of committee’s inquiry. More in depth and difficult to quantify issues such as organizational, sociological, and psychological factors were not identified and investigated as causal factors. Lucas (1991) offered insight into firefighting organization’s search for accident causes when she wrote,
“The search for patterns of causes is dependent to a great extent on the underlying perception of the causes of accidents and human failures held by an organization. This model of accidents and errors is a key element of an organization’s “collective memory” and of its prevailing safety culture.”

Thus there is a need for investigative procedures that circumvent this circular, self-protecting logic. Hendrick and Benner (1987) reported that repetitive accidents should be recognized as an indication of an inadequate investigation process that has failed to identify, evaluate, or act upon the relevant underlying causal factors.

Ferry (1988), in reviewing a variety of industrial accidents, mentioned 20 major accident investigation methods but found no one method universally accepted. The specific application and inherent deficiencies of each model have resulted in no nationally accepted method for accident investigation (Hendrick and Benner 1987). Recent national scale investigations such as the Challenger mishap have failed to identify environmental/organizational factors using the traditional predictive analysis (Vaughan 1996a). This may have resulted from agencies having embraced accident analytical techniques on the basis of perceived potency, current popularity, and agency investigator preferences, rather “than on the basis of their worth at dealing meaningfully with the real technical issues at hand” (Clemens 1993). The inability to have obtained a universally accepted model for all accident investigations, in addition to determining a wildland fire investigative model that identifies possible human and organizational error, has drawn attention to the current need.

Although many risks and hazards are associated with wildland firefighting (falling trees, rolling rocks, sprained ankles, etc...), this study focused exclusively on each method’s applicability to wildland fire entrapment investigations. They were not only the most publicly visible accidents but by definition, the cause of entrapments, burn fatalities, burns injuries, and fire shelter deployments (NWCG 1997).

Currently the analytical techniques used in wildland firefighting accident investigations have deficiencies that have not been completely addressed. This study has undertaken the inquiry into an accident investigation method that can be applied to wildland firefighter incidents/accidents and is best in identifying the major physical causal factors, vectors for human error, and latent organizational causative factors within the system.

CHAPTER 2 - GOALS AND OBJECTIVES

The purpose of this thesis is to identify a comprehensive, easily utilized, and systematic accident investigation method derived from US Forest Service (1998) recommendations that could determine most causal factors that have led to wildland firefighter fire burnover accidents (entrapments). The goal was to define a method that identified causal factors. Once identified, these factors would be addressed to prevent future accidents, reduce risk and hazard, and monitor safety programs. This method should be; 1) directly applicable in the field environment with a minimum of formal instruction, 2) required to be objective, proceduralized, and systematic to reduce or eliminate investigator bias and subjective analysis, 3) discipline the investigator and promote logical interpretation by others, and 4) reliable (testable) and document the accident process and identify any gaps in knowledge discovered in the investigation. The hypothesis of this thesis is that one of the three accident investigation methods derived from current USDA Forest Service recommendations would be the most applicable to wildland firefighting incidents/accidents using the proposed evaluation methodology. The investigation methods would each be used to evaluate the 1994 South Canyon Fire and the twelve “West Flank Group” fatalities and assigned ratings as to their overall ability to

meet the proposed criteria. This will be the initial test of the selected accident method proposed for future wildland firefighting entrapments. Subsequent direct application to ongoing accidents/incidents would increase the validity and reliability of the proposed method.

CHAPTER 3 - PREVIOUS WORK AND PRESENT OUTLOOK

ACCIDENT METHODS OVERVIEW

Accident investigation methods have developed significantly throughout the industrial age into the “age of the organizational accident” (Reason, 1990). As technologies advanced and systems became more complex, many disciplines have researched the sources of human, machine, and organizational failures that have led to accidents. Engineers, economists, psychologists, sociologists, attorneys, insurance companies, and industry managers were among the most prominent disciplines to actively seek out causes of accidents. The realm of accident investigation research currently encompasses risk management, problem solving, decision-making, human error, organizational safety culture, safety systems, and other human, machine, and environmental interactions. As Kjellen (1987) remarked, “The development of the necessary means to reduce risk of accidents involves a multidisciplinary approach and a close cooperation between theory and practice.” The integration of disciplines has facilitated a broader perspective and the ability to examine causative factors more accurately. This collaboration will remain essential to the ongoing search for the understanding and insights into the causes of accidents.

The perceptions of accident causes have been categorized into five main areas according to their history, limitations, and applications. Figure 2 illustrates the various approaches to accident investigations. Three additional approaches to accident investigations will be presented following the five perceptions summary. They do not meet the five previous categories criteria due to their investigative nature. These three approaches are Change Analysis, Managerial Failures approaches, and Multi-faceted/proactive approaches. General overviews of the various methods will be included.


image
Figure 2. Diagram showing the dominate five perceptions of accident causation (Benner 1975).


SINGLE EVENT CONCEPT

History

The first perception of accident causation is the single event concept. This concept focuses on the premise that accidents are caused by a single event. This simple model exemplifies the quest for the “cause” of what occurred. The search for a scapegoat and taking care of the scapegoat would solve the problem. This concept is the most widely perceived and least complex. The public and media typically utilize this concept when they ask “what caused the accident?”

Limitations

The single events concept is limited in its ability to see the accident as a process or sequence of events in time. The factors that may contribute to the accident are not identified or pursued due to the fact that the “real” cause is obvious and visible. Causes that may underline human behavior are rarely determined.


Application

Current applications are primarily apparent in how the public and media view accidents. This viewpoint is reinforced by findings such as when an airline accident was caused by “pilot error”. Police citations are another example of the perception.


CHAIN OF EVENTS CONCEPT

History

The chain of events concept or domino theory was originally developed by Heinrich (1941). The basic concept implied that accidents resulted from a sequence of events that led to an accident. Like a row of dominos, once the sequence began each event led to the next until an accident occurred. Intervention at any point along the events sequence could halt the accident process and eliminate the unwanted results. An unsafe act starts the chain of events that began with an unsafe condition.


Limitations

This concept is limited by the linear progression characteristic of the model. Interactions among events, contributing causes, and the duration and timing of each event limit the identification of all causal factors.

Applications

The current use of this concept is prevalent in the legal field that attempts to reconstruct the sequence of events that led to the accident.


STOCHASTIC EVENTS CONCEPT

History

The prevailing idea behind this concept is the gathering of data and facts in order to isolate the factors not due to chance. The model searches for variables common to all accidents. This approach utilizes statistical comparisons to search for causal factors present in accidents.

Limitations

The Stochastic Events approach is limited by its dependency on the data reported by the accident investigators. The gathering of the facts supercedes any attempts at analysis. The validity is lacking in this procedure because of investigators assumptions about the cause bias the reporting of the facts. The procedure is undisciplined and unstructured.

Applications

This concept is practiced to a large degree by the USDA Forest Service. A form is completed that has pre-identified contributing and causal factors to consider and record (see Appendix Table B.2). The “just the facts” approach is a commonly accepted way of investigating accidents in industry, law enforcement, and the medical profession.


BRANCHED TREE PERCEPTION

History

The development of the logic tree perception is illustrated by the following various accident investigation methods. The Management Oversight and Risk Tree approach encompassed several analytical techniques in a logic tree format as integral aspects of the investigative process. These techniques include Fault Tree analysis, The Haddon Matrix, Barriers Analysis, and Events and Causal Factors Charting.


Management Oversight And Risk Tree

History

Traditional accident investigations focused on the active response to a mishap and the identification of procedures to prevent future occurrences. The degree and intensity of the accident dictated the intensity of the investigation response and subsequent preventative action (Brown 1993). But as technology advanced and systems became more complex, the consequences of accidents became increasingly unacceptable to society and industry, particularly in the nuclear power industry. The nuclear industry and similar high-risk technologies have determined that learning from accidents and even near misses was not an option. The consequences of accidents precluded the traditional trial by error approach where as accidents occurred the problem was fixed subsequent to the next mishap (termed the fly-fix-fly approach). A new approach was undertaken to become proactive as well as reactive in accident analysis techniques to determine possible failure points prior to occurrence. Johnson (1973a) working for the National Safety Council and under a contract from the US Atomic Energy Commission focused on a systems approach to accident analysis. This approach focused on the entire system in which accidents occurred and the interaction of events within that system. Johnson merged two basic views to focus on management responsibility in planning the context in which accidents occur. These views, understanding the energy release process and focusing management of that hazard on the route of its release, led Johnson to develop the concept of “less than adequate” management decisions. This progressed to the Management Oversight and Risk Tree (MORT) accident analysis tool. He said MORT was “an analytical procedure that provides a disciplined approach for finding the causes and contributing factors of mishaps”. It entailed a very broad and detailed checklist that facilitated the search for safety problems. It incorporated 1500 possible causes and 98 generic problems and was the initial methodology to embody management oversight into accident causation. The Department of Energy currently employs this method as one of its most comprehensive analytical techniques (DOE 1992). It is more generally used as a proactive method in safety system evaluations than as an accident investigation method. This is primarily due to the fact that it can be time consuming and intensive and due to the nature of the nuclear industry, identifying possible loopholes in the safety system to eliminate hazards is more cost effective and publicly expedient than after the accident occurs.

This concept was highly visible, easily reviewed and updated as new relevant facts warrant, and provided structure to help reduce overlooked factors and bias. Within the MORT system incidents were defined as inadequate barrier/controls or as failures without consequence. Accidents resulted in adverse consequences. The MORT system incorporated the concept of the unwanted transfer of energy that can cause mishaps due to inadequate barriers/controls. These barriers and controls may be physical (protective clothing, concrete walls, etc...) or administrative (codes, standards and regulations). The MORT system is based on two main sources of accidental losses: 1) specific job oversights and omissions and 2) the management system factors that control the job (Johnson 1973a). A third source he mentioned was “assumed risk”. Johnson noted that once this source was properly evaluated it could not be considered accidental in nature since we have consciously decided to accept the risk. Integral aspects of the MORT process are Fault Tree Analysis, Barriers Analysis and Event and Causal Factors Charting. Each of these approaches will be subsequently explained.

Limitations

Limitations of MORT are that it can be insufficient in finding specific causes as it designed to identify general causal areas (Gertman and Blackman 1994). These authors do recognize its strengths in identifying more specific control and managerial factors. Moreover, this systematic process is advantageous when system experts are not available.

Application

Its current use as a proactive safety system analysis tool for the Department of Energy has long standing (Briscoe 1990). It has been used exclusively as both a proactive technique and an accident investigation method for the Nuclear Regulatory Commission.


Fault Tree Analysis

Fault Tree Analysis is a “branched tree” approach that uses Boolean logic to work backward from the accident event to identify causal factors. This technique is elaborated on in more depth in the Methods section as it is one of the USDA’s proposed methods for accident investigations (USDA 1998).


Haddon Matrix

History

Haddon (1968) was a medical doctor who introduced an epidemiological approach that is still currently used. It was a matrix of accident phases (pre-event, event, post event) and components (agents in the accident sequence) used to describe the accident sequence. Some investigators currently use the Haddon matrix to identify where effective interventions may be implemented in the accident sequence.

Limitations

This approach is limited by its inability to discover more deeply rooted causal factors. It is not as an intensive approach or as systematic as other methods.

Application

The most notable proponent of this approach is the National Institute for Occupational Safety and Health. They use a Fatality Assessment and Control Evaluation (FACE) protocol that is based on the Haddon Matrix (Casini 1998 personal communication).


Barriers Analysis

Barriers Analysis is an accident investigation method that is an additional component of the MORT process. The method identifies barriers/controls that are in place to prevent accidents. These barriers may be physical and/or administrative and must be absent, inadequate, or bypassed in order for the accident to occur. A more detailed account of this approach will be undertaken in the methods section as this method is one of the USDA proposed investigative tools (USDA 1998).


Events and Causal Factors Charting

History

Events and Causal Factors Charting (ECFC) graphically depicted a mishap from beginning to end and showed the relationship between related causal factors and conditions that influenced the accident sequence (Buys and Clark 1995). In the mid 1950’s the National Safety Council developed “Dynamics of Home Accidents” in an attempt to illustrate the multifactoral aspects of accidents (Johnson, 1973b). Figure 3 shows the suggested sequence of factors involved, including human factors, environmental factors, and mitigating factors.

Limitations

Johnson recognized the limitations of this model in that its simplicity failed to recognize organizational and industrial situations. He added that these situations were more complex and had many additional factors that were inherent in those systems. Though relatively simple as an accident process model, the diagram was a significant contribution to accident cause analysis and became a prototype for events chain. It provided a systematic, standardized approach that disciplined and organized the investigator and allowed for logical, critical review of the investigation process. Events and Causal Factors Charting extended Johnson’s work by illustrating the proceed/follow approach that characterizes the technique in addition to the systemic and contributing factors (Figure 4). Figure 5 illustrates how this concept could be applied to wildland firefighting and the example of a firefighter receiving burns. Systemic and contributing factors could be incorporated so that organizational and managerial control factors could be linked and assessed. This addition was a major contribution to previous linear

image
Figure 3. National Safety Council (1984) diagram of the “home accident ”

sequence and possible causal factors that can lead to an accident.


investigation methods. Interactions and relationships between factors could be easily depicted and questioned if information was absent or questions arose. Multifaceted problems with long or complex causal factor chains could be better analyzed by this method (Gertman and Blackman 1994). This is because the accident sequence can be visibly outlined and worked backward from the accident to reveal causal factors as they lead to and interact with, each other. Benner and his associates (in Ferry 1988), while working for the National Transportation Safety Board were innovators in the development of sequence diagrams and the charting processes such as ECFC. The charting process visually allowed for evaluation of factors that sequentially led to an accident.

Limitations

Limitations of this method include the amount of time required to conduct the analysis and the need for investigator familiarity with the process in which the accident occurred (Gertman and Blackman 1994). The absence of a time scale to relate simultaneous events to each other is another limitation of the method.


image
Figure 4. Event and Causal Factors Charting (ECFC) diagram showing the integration of systematic factors with contributing factors leading to direct causal factors (Ferry 1988).


image
Figure 5. Event and Causal Factors Charting diagram of an example of the accident sequence where a firefighter gets burned.

MULTILINEAR METHODOLOGIES

Multilinear Events Sequence (Mes)

History

Hendrick and Benner (1987) developed a systems based multilinear sequence method (Figure 6) to accident investigations that sought to overcome the deficiencies that were inherent in earlier methods. Multilinear Events Sequencing (MES) was an analytical technique initially developed by Benner (1975) while working with the National Transportation Safety Board and a further development of Events and Causal Factors Charting. Figure 7 illustrates the MES method using a firefighter receiving burn injuries. This approach incorporated a temporal consideration that recognized and accounted for multiple events by multiple actors (or agents) that previous methods failed to take into account. In addition, some of these events may have occurred simultaneously, this method provided a chronological validation and event comparison format. Thus this process provides the opportunity to discover possible unknown linking events, causes, and contributing factors. Benner (1977) remarked that this approach provided a “method for proving the hypothesis that differs from traditional, statistical, or experimental approaches of the scientific method” by illuminating areas that may not be directly linked in the causal sequence. There were two distinct differences of the MES technique that has built upon the work of Benner and associates’ (in Ferry 1988). The first was the identification of the beginning and end of the accident sequence. The accident sequence began when a perturbation disturbed the homeostasis (therefore this method has been called the P-theory in reference to a perturbation). When this stable flow of events was interrupted by external influences the possibility of a harmful outcome increased. Identification of the flow deviation from the normal harm-free process was necessary to accurately pinpoint the start of the accident sequence. Identifying the end of the sequence (the final damaging event) would allow the accident process boundaries to become established so that the entire flow of events could be framed. The full sequence could then be subdivided into individual events and causes.

image
Figure 6. Multilinear Events Sequence (MES) diagram showing the analysis process in reconstructing the accident sequence. Note the time scale at the bottom and the incorporation of simultaneous conditions and/or events (Benner 1975).


image
Figure 7. Multilinear Events Sequence example illustrating accident process of firefighter receiving burns.


The second major contribution the MES process has embodied is a more

distinct time frame than was present in antecedent linear models. The timeline has aided investigators by structuring the search for relevant factors and events. Newly discovered conditions or events could be easily tested and then inconsistencies and gaps in knowledge could be more readily determined. The Civil Aeronautics Board (1962) in the early 60’s incorporated a time line when flight data recorders came into use.

Limitations

This method may be limited by its perceived complexity in developing the framework to process all the information gathered. Underlying human factors may also be more difficult to identify if experience in the relevant work tasks is limited.

Application

Currently the National Transportation Safety Board utilizes a similar concept as part of a hybrid approach. Their approach involves a quantitative assessment of engineering structures, the environment, and the time line analysis (Gertman and Blackman 1994).


Additional Approaches

There are various other approaches to accident investigations that deserve mention. These methods seek to determine causal factors in ways that preclude categorizing into the previous sections. A list and short explanation follows.


Change Analysis

History


The Rand Corporation (Ferry 1988) developed the concept of change analysis for the Air Force. Their concept was to identify change in a system that would normally operate without mishap. Something had to have changed to make the mishap possible. That is, a disturbance to a homeostatic process was the catalyst initiating the accident sequence. By comparing what changes occurred which resulted in a mishap to the normal accident free task, causal factors might be identified. Such change could be directional and exponential. It would be directional in that once change is initiated it would continue to proceed until another change occurred. It could be exponential in that once it was initiated the changes interact to compound the effects of mishaps. Figure 8 illustrates the basic concept central to Change Analysis. It is considered to be a relatively quick process for detecting obscure causes.

Limitations

An expert knowledge of normal systems operation was essential to the determination of changes that ultimately resulted in injury or loss. This method could become very involved when applied to complex processes (Ferry 1988).


Application

Though this approach is limited, it still is used by various private accident investigators as well as with the US Air Force.

image
Figure 8. Change analysis diagram depicting the concept and process that compares the pre-accident situation to the post-accident consequence. The process aids in determining the changes to the system that had to occur for an accident to be initiated. (Ammerman 1998).

Managerial Failures Approaches


History and General Overview


Many prominent accident investigators have stated the position that accidents have their roots in managerial and organizational failures (Fine 1976, Weaver 1973,

Grimaldi and Simonds 1984, Petersen 1975, Vaughan 1996a, 1996b). Fine (1976), for instance, summarized this concept when he stated, “all accidents and hazards are indicators of management failure.” Vaughan (1996b) directly related that concept to the USDA Forest Service firefighting community when she said that they are politically vulnerable and the policy decisions that they make directly affect how operations are done on the ground and how lower level employees make decisions. She concluded by saying “top decision makers are thus irrevocably responsible for safety.” Just as with the Challenger disaster, the USDA Forest Service has had warning signs of potential danger latent within the organization prior to the South Canyon Fire. These latent conditions brought about the transition of seemingly small, minor decisions towards what was described as an “incremental descent into poor judgement” (Turner 1978). Reason (1991) used the medical term “resident pathogens” to describe latent conditions in an organization that may have laid dormant for years until a triggering mechanism broke through the system defenses and barriers to cause an accident. He emphasized that these resident pathogens could be identified with “adequate access and system knowledge.”

One of several investigation techniques that looked more deeply into management failures and their contribution to accidents was TOR, the Technic of Operations Review (Weaver 1973). TOR was developed for the Wausau Insurance Companies to identify management oversight and omissions. Findings from accident investigations were analyzed using a four-step process. The process led investigators through a work sheet of eight general categories. The investigative team was to identify a direct cause to initiate the process. They then followed the factors that contributed to the direct cause that the worksheet proposed. This identified possible contributing factors to the accident and investigators eliminated factors that did not apply. The sequential process was used to locate the potential problem areas within the organization. Weaver recognized that though simple to use, TOR required an objective mid-level management team to be effective at exposing organizational deficiencies.

Another systems approach to accident investigations that has directly implicated management failure was Fine’s method (1976). While working for the Naval Surface Weapons Center, he had developed an approach based on the premise that for each causal factor identified in an investigation, the question needed to be asked, “Where did management fail?” His technique proposed fifteen possible management failures linked to each causal factor found in any mishap. Fine stipulated that expertise and sound judgement by the investigators was required in order to trace all the direct and indirect factors attributed to higher level management.


Multi-Faceted/Proactive Approaches

Root Cause Analysis


Root Cause Analysis (Ammerman 1998) was a method that incorporated a process for determining a single cause. The process involved a step-by-step sequence of previously known investigation methods. The step-by-step process was provided to systematically direct the investigator through a series of analysis tools so that the strengths of each were utilized toward finding the root cause. These analysis methods were; 1) Task Analysis, 2) Change Analysis, and 3) Control Barrier Analysis, 4) Event and Causal Factors Charting, 5) Interview Techniques, and 6) Root Cause Analysis. Ammerman (1998) added that even though the goal was to find the root cause, this process also identified contributing causes. He defined root cause as a causal factor that, when eliminated, would prevent recurrence of that problem. A contributing cause may not have directly caused the mishap but was identified as needing corrective action. The Root Cause Analysis process built upon the sequence of analysis tools as a means to document, systematically organize, and logically proceed through an investigation. The goal was to not only identify what happened, but why. Ammerman (1998) stated that any undesirable event, including those involving equipment failures and human error could be evaluated in this manner.


Human Reliability Assessment

In the 1960’s and 70’s human factors specialists, while looking for the role and causes of accidents in the work process, advanced the concept of human error in accident causation theory. Scientists that have worked in the nuclear weapons production industry such as Altman (1970), Chapanis (1965) Christensen (1972), Rigby (1970), Rook (1962) and Swain (1963) focused their research on human reliability and the description of human behavior in terms of errors. They recognized the major role human error had in potential mishaps and worked toward identifying possible areas that could compromise the traditional “defense-in-depth” safety backup systems. Defense in depth is the multiple layered barriers in place for the protection of workers from hazards.

Many proactive risk assessment techniques were developed and are still being updated and evaluated as to their relative effectiveness. These techniques were focused on the human-machine interface and identification and quantification of human actions on systems risk. Under the general heading of Probabilistic Risk Assessment (PRA), analysis techniques such as Human Reliability Assessment (HRA) and Human Error Identification (HEI) were major evaluation methods to assess potential risks to systems and the possible human contribution to that risk. Techniques such as Technique for Human ERror Prediction (THERP)(Swain and Gultman 1983), HAZard and OPerability study (HAZOP)(Kletz 1974), Generic Error Modeling System (GEMS)(Reason 1987), Systematic Human Error Reduction and Prediction Approach (SHERPA)(Embry 1986) were just a few of the many prospective accident investigation methods to reduce risks and accident rates in complex technological industries.

One method of accident analysis that was developed for the Department of Energy to identify human interactions within complex systems used a hierarchical tree format similar to MORT. Human SYStem interactions (HSYS)(Hill and others 1990) was a linear process based on input-action models. The process followed a sequential path to examine human performance factors in incident/accident occurrences. Errors could be classified according to these five steps; 1) input detection, 2) input understanding, 3) action selection 4) action planning, and 5) action execution. These five steps formed branches of the hierarchical tree and have aided in both prospective and retrospective analysis. Hill and the other investigators stated that incorporation of intra-group, inter-group, and organizational aspects were still being developed using this approach. Analytical techniques such as HSYS that attempt to categorize human error types offer the opportunity to identify, track, and reduce mishaps rooted in human error.

The following section details the USDA Forest Service proposed accident investigation methods that where highlighted in the previous sections.


USDA FOREST SERVICE PROPOSED METHODS

Prior to 1998 the National Wildfire Coordinating Group (NWCG) developed accident investigation policy for the five federal firefighting agencies. A significant change in investigative techniques has occurred since the 1994 South Canyon Fire investigation. The previous method involved a multi-methodological approach based on traditional methods, logical deduction, common sense, and expert judgement. The traditional analysis methods incorporated accident reconstruction, identification of unsafe acts and conditions, trial and error fixes, statistical inference (finding variables derived from data to determine probabilities of future occurrences), and trend forecasting (using historical data to predict trends). The South Canyon Fire investigation (USDA,USDI, AND USDC 1994) used a matrix approach (see Appendix Table B.2.) where predetermined criteria were categorized as to whether they were significant contributors to the accident, influenced the outcome, or were non-contributing (IMRT 1995). The criteria ranged from fire behavior factors, and equipment condition to personal factors such as training and fatigue. Every significant contributor is to have written documentation. These criteria were effective in recognizing possible causal and secondary factors but lacked the means to identify underlying human error (Putnam 1995). Because this checklist approach only accounted for those items on the list, no possible human, cultural, or organizational factors were available for evaluation.

In response to the South Canyon Fire, the USDA Forest Service (1998) has drafted new guidelines for the investigation of accidents. Based on US Army procedures (DA-PAM-385-40 and AR 385-40 1998), the process used a “3W” approach (Ricketson et al 1980). The “3W’s” are what happened, why did it happen and what to do about it. Figure 9 illustrates the approach. Investigations focus on assessment of elements that revealed human, materiel, and environmental factors that caused or contributed to accidents. The premise behind the concept is that by finding the reasons why people make errors, materiel fails, and environmental conditions contribute to accidents, then similar deficiencies can be identified and reduced. Subsequently, the Forest Service recommended four analysis techniques but none were specifically identified or suggested for wildland firefighter incidents/accidents. The Safety Management Mishap Investigation and Reports Guide (USDA Forest Service 1998) said that the basic premise was to examine “why the sequence of events happened in terms of task errors, materiel failures/malfunctions, and environmental factors.” The four methods cited were Fault Tree Analysis, Failure Mode and Effects Analysis (FMEA), Energy Trace Hazard Identification (ETHI), and Sequentially Timing and Events Plotting (STEP). No explanation was given as to why these methods were selected as analysis techniques.

image
Figure 9. The US Army and Department of Defense “3W’s” approach to accident investigation, analysis, and prevention (PAM 385-40 1998).




ENERGY AND TRACE HAZARD ANALYSIS


Energy and Trace Hazard Analysis is an integral aspect of the Management Oversight and Risk Tree process previously discussed. Gibson (1961) introduced the concept of energy flow and barriers in the classification of accident process. This concept focused on various vectors of potentially harmful energy sources (chemical, kinetic, electrical, and thermal) and the barriers provided to protect from their harmful effects (Figure 10). Identification of these barriers that have been compromised aided development of improved or additional defenses. Gibson’s search into safety analysis looked for a more behavioral approach in that these barriers can be supervisory, managerial, or organizational/cultural as well as physical. He stressed that these barriers may have worker behavioral implications in that these non-physical barriers are less visible and easier to violate without immediate adverse consequences. Administrative barriers such as rules and regulations are much easier to transgress than physical barriers such as containment walls or wire insulation. Examples of administrative barriers present in the wildland firefighting profession are the 10 Standard Firefighting Orders and 18 Watch Out Situations (see Appendix Table B.2). Examples of physical barriers would be fire shelters and personal protective equipment, such as fire resistant clothing, hard hats, gloves, neck shrouds, and leather boots. But a physical barrier would include any boundary of thermal protection between the firefighter and the fire itself.


image
Figure 10. Barrier Analysis conceptual framework where barriers/controls prevent the unwanted transfer of energy from a hazard to a valued target. These barriers may be physical (protective clothing) or administrative(safety rules)(EG&G Idaho 1985).


Haddon (1973) further developed the unwanted transfer of energy concept and its control by various measures or barriers. Again, sources of energy were derived from chemical, kinetic, electrical, and thermal vectors. He specified ten types of barriers to the accidental transfer of energy. These barriers are intended to:
  1. Prevent the marshaling of potential energy--do not produce or manufacture the energy (e.g. Prevent probabilities of fire ignitions).
  2. Reduce the amount of potential energy--voltages, fuel storage (e.g. Reduce fuels).
  3. Prevent the release of potential energy--strength of energy containment (e.g. Reduce fire probability under adverse weather conditions or increase separation distance of fire personnel).
  4. Modify the rate of release of potential energy--slow down burning rate, speed (e.g. Cool fire with water, dirt).
  5. Separate in space and time the source of potential energy--electric lines out of reach, (e.g. Escape route to safety zone or indirect attack).
  6. Interpose material barriers from the potential energy-Insulation, guards, (e.g. Personal protective clothing).
  7. Modify shock concentration surfaces of the energy--Round off and make soft (Probably not applicable to burnover incidents).
  8. Strengthen the target of the potential energy—Earthquake-proof structures (e.g. Fire shelters).
  9. Limit the damage of potential energy--Prompt signals and action, sprinklers (e.g. Lookouts).
  10. Rehabilitate persons and objects that may come in contact with the potential energy. (e.g. Discipline and/or retrain)
An analysis of an accident sequence can be initiated by investigating a) the energy source(s) and their paths, b) the people or objects that are vulnerable to the unwanted energy flow, c) the barriers and controls that were designed to protect vulnerable people and objects, and finally, d) the precursor events of energy transfers and barrier failures that lead to the accident. The ten types of barriers outlined above show examples of their applicability to firefighting operations. Barriers Analysis also allows safety personnel or investigators to examine the sequence of events/causes that may have led up to the accident. Ammerman (1998) provided a worksheet to document and track accident consequences, barriers in place and the reasons for barrier failure in any accident where there was loss of property or injury. The Department of Energy (1992) expanded on the barriers concept by implementing a six-step process that identified the barriers, found the ones that failed, identified how they failed, then why, where barriers may have prevented the accident and finally validated the findings from the information learned. This process was incorporated into this thesis and documented using DOE’s recommended worksheet (see Appendix Table A.1).

The Barrier Analysis method is currently one aspect of the accident investigation process (and MORT process) utilized by the Department of Energy (Trost and Nertney 1985, Buys and Clark 1995) and proposed by the USDA Forest Service (USDA Forest Service 1998). Though recommended, this method has not been utilized as of this date by the Forest Service. Though the concepts and processes are identical, Barriers Analysis is also called Energy Trace Hazard Identification, Control Barriers Analysis, and similar variations of those names.

Barrier Analysis is limited by requiring investigators to have a good working knowledge of the task process in order to properly identify and evaluate barriers/controls and possible avenues of barrier penetration (Gertman and Blackman 1994). Since barriers may be administrative, managerial, and supervisory, as well as physical, a competent overall knowledge of the work process is essential.


FAULT TREE ANALYSIS


Heinrich (1941) developed the methodology that preceded and formed the basis for Fault Tree Analysis. He illustrated the linear sequence of factors in accident causation by using a domino theory. The theory stated that a disturbance that caused any one of the five identified components of the sequence to fail would set off a chain-of-events that led to an accident. The five in the sequence were 1) ancestry and social environment, 2) conditions and fault of person, 3) unsafe act, 4) unsafe condition and 5) injury. He showed that by intervention at any point along the sequence an accident/injury could be prevented. This theory has been modified and updated (Baker 1953, Marcum 1978, Heinrich et al 1980), and has wide applicability in current automobile accident and law enforcement investigations.

Similar linear sequence models such as Critical Path Analysis (CPA), Gantt Charts, and Program Evaluation Research Task (PERT), were initially used in the 1950’s and 60’s as planning tools (Lockyer 1964). Though many names were given to their process they were very similar in their goals and methods. They provided a graphical display of activities linked to events by arrows in order to plan complex projects. The process illustrated a flow (path) from one task sequence to the next and incorporated time frames and interrelationships between tasks. Projects could then be analyzed by task, the amount of time needed for each segment and the relationship a task may have with another task. These methods offered an effective means of project planning, costs analysis, and time frame considerations by visually outlining the task process (Lockyer 1964). These processes also provided the means to better understand the interrelationships between and among tasks. This logical depiction of process flow related directly to analyzing an accident sequence and the precursor events.

In the 1960’s Bell Laboratories expanded upon the linear chain of events concept through missile system safety. They arranged events in a flow chart that used a proceed/follow logic pattern. Their concept, Fault Tree Analysis (Figure 11), is generally credited to Watson (1971). Figure 12 illustrates the fault tree concept as applied to a hypothetical accident where a wildland firefighter was burned. This analysis concept helped provide a sense of management by objectives by identifying unwanted events (the top event) and then systematically and sequentially determining the precursor events. The objective is the top event and the identification of the preceding causal factors aid in the management achievement of that objective. Watson’s Fault Tree Analysis investigation methodology provided a visible, easily understood and defendable format (1971). The methodology extended the linear chain of events into a “branched events chains” concept through the use of “and/or” logic gates. It uses basic Boolean logic in a hierarchical tree format. Other Boolean terms such as “not” are not used in Fault Tree Analysis. For example, “C” can only occur when both “A” and “B” occur. If two or more events are required for a cause to happen then an “and” symbol is used. Another possibility is when only one of the factors need be present. For “C” to occur, then “A” or “B” occurred. If only one event of two or more are necessary then an “or” gate is used. The “top event” is the unwanted result of the accident and causal factors branch out below leading to it. The downward sequence is continued until the root causes are found or the tree cannot be further developed. This technique, according to Benner (1975), “contributed a powerful tool for the investigation of accidents – both historical and postulated.” Accidents could be investigated or reinvestigated in the search for causal factors utilizing this method. It assisted in illuminating areas that may have previously been overlooked by other means. Numerous approaches to determining accident causal factor using “branched events chains” reflected the discipline of the investigations employing it; thus medical doctors

used an epidemiological approach (agent/host/environment), while psychologists focused on human factors.

image
Figure 11. Fault Tree diagram illustrating a typical failure process, symbols used,

and the logic sequence leading to an undesired event, a dark room (in Ferry 1988).


image
Figure 12. Fault Tree diagram illustrating the deductive process using an example of a sequence of events in which a firefighter receives burns.


One key limitation of Fault Tree Analysis is the inability to model time sequences that are concurrent and interactive (Hendrick and Benner 1987). Brown (1993) added that only one event could be analyzed at a time and thus primarily applicable to catastrophic events. Benner (1975) cited similar deficiencies, most notably that charting analysis methods focus on a single undesired event and provided no means to indicate the chronological relationships (and the subsequent concurrent interrelationships) of events. Another limitation is the restriction inherent in the method whereby causes must be either successes or failures and degrees of each are not accounted for (Tulsiani and others 1990).


SEQUENTIAL TIMING AND EVENTS PLOTTING

Evaluations of the multilinear systems safety approach have led to a procedure developed by Hendrick and Benner (1987). Multilinear systems approaches view accidents as multiple avenues of causal factors that react to previous factors and may interact with others throughout the system to ultimately lead to an accident. The Sequentially Timed and Events Plotting (STEP) procedure was a comprehensive approach to reconstructing an accident. It was based primarily on Events and Causal Factors Charting and Multilinear Events Sequencing previously cited. The key component of the accident reconstruction process was the STEP worksheet (see Appendix Figures A.3). The worksheet was the documentation that provided structure, visibility, and organization to data gathering and analysis. It illustrated the beginning and end of the accident sequence along columns that represented time. The rows of the worksheet listed the actors, either people or things, which acted to produce the harmful outcome. Each actor performed one action, termed an event, that when displayed along a timeline visually showed the interactions among actors and events. The process subsequently accommodated events that occurred at the same time. Each event was represented by a block diagram that displayed the time the event occurred, the information source, the actor and the action (Figure 13). These event building blocks allowed investigators to visually recreate the mental motion picture and determine gaps.

By performing three tests, the accuracy and validity of the entire worksheet could be assessed. Test number one is the column test to make sure the events sequence is accurate. Test two is the row test for completeness. The third test is the necessary and sufficient test to validate what events were necessary and sufficient to have caused the next event(s). These tests also helped investigators look for knowledge that may be lacking. This extended the cause and effect linear model into one that took into account contributing causes and conditions that occurred simultaneously. Interruptions and questionable cause and effect relationships could be more readily recognized and investigated than previous logic diagram techniques.

The STEP concept was substantially based on the development of a “mental motion picture” of the accident sequence as a reconstructive tool. The building blocks of actors and their actions were the “frames” in which to recreate the “motion picture”. Figure 14 illustrates the incorporation of building blocks onto the STEP worksheet in order to visualize the accident “motion picture”. Hendrick and others (1987) proposed an additional benefit to the STEP methodology. They added that the identification and utilization of an applicable decision-making model along with concrete terminology to specifically classify human error would expand the capabilities of STEP. The decision making model could provide the basis for the development of a data base to track and analyze human error that was unique to an occupation, task, and industry. In this thesis, the STEP method is applied without the human error classification since it is currently unrefined.

image
Figure 13. STEP CARD used to consolidate information used to reconstruct an accident sequence. (Hendrick and Benner 1987).


image
Figure 14. STEP Worksheet illustrating the placement of individual building blocks (of actors and events) their influence and interaction with one another, and relative sequence in time (Hendrick and Benner 1987).


CHAPTER 4 - METHODS

STUDY APPROACH

The application of an accident investigation method to wildland firefighting was evaluated by examining a single case, a portion of the South Canyon Fire. Case studies have provided an established, valuable method of study. Yin (1989) stated that the case study is a “frequent mode of thesis and dissertation research in... psychology, sociology, political science, anthropology, history and economics”. The frequent use of case studies is due to the fact that they allow close in-depth analysis and understanding of specific cases, aid in understanding unique realms of inquiry, and provide insight into cases that could not be duplicated experimentally. Reason (1990) has stated that when sufficient evidence regarding a single case is available, “we are able to study the interaction of the various causal factors over an extended time scale in a way that would be difficult to achieve by other means.” This case study allows for an evaluation of an extreme incident that could not be replicated by experimental means. Reason added that case studies have taught us “disasters are very rarely the product of a single monumental blunder.” He further states that human-made disasters are generally the result of accumulating, apparently negligible consequences that compound to contribute to the undesired result. In reference to this specific case study, Prineville Hot Shot Superintendent Tom Shepard (South Canyon Fire survivor) echoed Reason when he said, “There was a whole series of events and circumstances, a change in any one of those would have produced a different outcome”(Long and Hoover 1994).

Case studies, as applied to this thesis, involve the investigation into a single social phenomenon, in this study, the 1994 South Canyon fatality fire. Case studies can generate innovative new interpretations and concepts by selectively analyzing a single case (Feagin and others 1991). As laboratory experiments lend themselves to low-level generalizations, case studies can provide the means to generalize to theory (Yin 1989). This approach allows for a broader explanation of how cases that are deviant can provide insight into accident causation and investigative techniques.

Yin (1989) mentioned four procedures in order to construct case studies. The first is asking the right question you would like answered, what is the theory you are attempting to clarify and investigate? Data collection is the second phase and must attempt to include as many sources as possible in order to triangulate (come at the important data for various sides). This provides construct validity to the research. Data analysis is the third phase and techniques such as pattern matching, explanation building, and a logic model are techniques used to establish a chain of evidence that provides validity to the researcher’s conclusions. This phase provides internal validity. External validity is accomplished by entertaining rival explanations of the proposed hypothesis throughout the study. The last phase is the reporting of the findings and conclusions in which data in the form of tables, spreadsheets, statistical outputs, interviews, coded worksheets, etc. provide the evidence necessary to defend the conclusions. Reliability is established through the case study protocol developed in the research design phase.

Three accident investigation methods were selected for evaluation. These methods are the “units of analysis”. The method of investigation is under examination in this thesis, not the fire itself. As previously mentioned, three methods, Energy and Trace Hazard Identification (also called Energy Trace Hazard Analysis, Control/Barrier Analysis or Barrier Analysis), Fault Tree Analysis and STEP were recommended by the US Forest Service as their newly established preferred methods (USDA Forest Service 1998). Failure Modes and Effects Analysis (FMEA) was a fourth recommended method but was not integrated into the thesis. This was due to its primary quantitative application to hardware component failure rates in systems (DOE 1992, USDA Forest Service 1998). That particular method utilizes experimentally derived rates of failure in various components to obtain an overall failure rate. Benner (1985) rated these three methods highest among 14 methods he evaluated from 17 federal agencies. Benner utilized 10 criteria derived from OSHA statutes and policy to rate these methods. Five of these criteria were utilized in this thesis as Benner’s other criteria were directly applicable to satisfying OSHA’s mission and not directly to the accident methodology. Benner’s additional criteria were, a) satisfying, b) functional, c) direct, d) noncausal, and e) definitive. Criteria utilized in this thesis were selected because they satisfied standard assessments of reliability and validation (Benner 1985, Feagin and others 1991). Benner’s reasoning behind the high ratings was that they were focused on accident causation as a process where events occurred in a logical sequence. These “events process” methods showed the interactions between actors and events and the influence of contributing factors on the accident sequence.



LIMITATIONS AND GENERALIZABILITY


One caveat Reason (1990) mentioned is the limited information that is available from past accident investigations and the tendency of documentation to be “digitized” as opposed to an original, more complex and continuous nature of “analog” events. Past accident reports lack the information that was potentially available. The broader, richer, and more complex possibilities of the original account can be compromised in the written form. Though this thesis is constrained by this limitation, the systematic process of each method can not only identify contributing and causal factors but gaps in knowledge that need further inquiry. The identification of these gaps is a positive tool for improvement of future investigative procedures and could identify areas that a particular method takes into account or, conversely, fails to recognize. Thus the advantages and disadvantages of each method can be determined. One method may be more applicable to specific causal areas whereas another may be stronger in another. Determining the strengths and weaknesses of each method as applied to wildland firefighting entrapments could provide the theoretical framework for applications to subsequent accidents/incident investigations. The method determined most applicable would also provide possibilities for additional research to overcome any inadequacies inherent in that method.

The reliability and validation procedures of this case study approach, as applied to wildland firefighter entrapments, was an important factor in selecting this method of analysis. Each method used has been previously utilized in various industries and was found to be valuable tools in accident cause determination (Benner 1985). Therefore they have been found valid and reliable in other high risk occupations. This thesis utilized inductive reasoning where specific observations lead to theory generalization. Yin (1989) stated that “case studies, like experiments, are generalizable to theoretical propositions, and not to populations or universes and... the investigator’s goal is to expand and generalize theories (analytic generalization) and not to enumerate frequencies (statistical generalization)”. Just as statistics from experiments that were derived from population samples are generalized to a larger population, case studies can similarly generalize to a larger theory of the investigated process itself. As in any scientific experiment, replication of the results in other case studies can offer additional information and validation into the phenomenon studied. By examining an extreme case, generalizing to less complex, less extreme cases could be applicable, reliable, and valid.

In addition, case studies can provide invaluable modes of understanding (Yin 1989). Insights gained from their analysis can be incorporated into theories of error production. Case studies can expand on principles that “can reasonably be expected to reduce either the occurrence of errors or their damaging consequences” (Reason 1990). The study of individual cases can provide understanding into the breadth and scope of human performance capabilities that laboratory environments could not emulate. It would be impractical (and unethical) to attempt to replicate extreme circumstances that model the real world in a laboratory environment. The ability to investigate and learn from these extremes in human capabilities, high risk decision making processes, and problem solving under life threatening situations can only be studied in their complete context from case studies.


CASE STUDY BACKGROUND


Background knowledge of the South Canyon Fire studied in this thesis can provide insight into the work processes, complex interactions, and the accident sequence itself. This is a general overview from the fire’s inception to the time of the accident. A more complete account can be found in the published literature previously cited.

This fatality fire provided the detailed, published documentation to compare and contrast the proposed methods of analysis. It was inarguably the most documented wildland firefighter fatality fire investigated up until that time. On July 2, 1994 seven miles west of Glenwood Springs, Colorado, lightning started the South Canyon Fire. Due to the large number of fires that burned on BLM’s Grand Junction District at that time, the fire was monitored until July 4 when increased public concern and resource availability lead to the decision to begin suppression of the fire (IMRT 1994). Local resources made up of a seven-person BLM/ Forest Service crew arrived and began suppression activities early on July 5. This group was supervised by Butch Blanco who was designated the Incident Commander for the fire. Eight smokejumpers with Don Mackey as “jumper–in-charge” reinforced the local crew later that evening. When mechanical problems disabled their chainsaws the BLM/Forest Service crew hiked back down to Interstate 70 to do repairs and return the following morning. The smokejumper crew worked on the fire till early morning on the 6 th when the rolling of burning logs and pine cones made line construction too hazardous in the dark. They continued line construction on the southeast flank after dawn (Figure 15). Later that morning, eight additional smokejumpers parachuted to the fire with jumper–in-charge Eric Hipke. Hipke turned over the jumper-in-charge responsibility to Dale Longanecker who then became a line scout. A helispot was cleared near the fire and transport by helicopter of the twenty members of the Prineville Interagency Hotshot Crew began. Superintendent Tom Shepard led the Hotshots. After a 0930 reconnaissance helicopter flight by Blanco and Mackey was made to determine suppression tactics, Mackey instructed the smokejumpers to begin line construction down the west flank from the north. They refused the assignment due to the fire activity at that time, but when the 8 additional jumpers arrived, line construction began. They were reinforced by nine of the Prineville Hotshots upon their arrival at 12:30. After eating at the “Lunch Spot” at approximately 14:00, some smokejumpers worked to the south while the Prineville group of nine and several smokejumpers worked the West Flank. This group has been designated the “West Flank Group” (Butler et al 1998). The remaining Prineville Hotshots and the BLM/FS crew worked on the Main Ridge improving fireline and monitoring for spot fires. At approximately 15:20 that afternoon, a dry cold front passed the fire area producing increased winds and fire spread and intensity escalated. At 16:00 the fire had crossed the bottom of the west drainage and spread up the west side. The firefighters on the west flank were ordered “to get out of there” by Shepard (IMRT 1994, Butler and others 1998). The fire then spotted back across to the east side beneath retreating firefighters on the west flank fireline. The fire moved uphill in dense Gambel oak vegetation and overran firefighters attempting to escape up to the Main Ridge on the west flank. Of the forty-nine firefighters assigned to the fire, twelve perished on the west flank and two helitack personnel perished when they were overran northwest of the fire. A third group, called the Lunch Spot Ridge group (Butler et al 1998), deployed fire shelters and survived. A fourth group, called the Main Ridge group, escaped down the east drainage. An initial investigative report was published by the USDA, USDI, and USDC (1994) and followed up by two reports by the Incident Management Review Team (IMRT 1994,1995). The Occupational Safety and Health Administration also investigated the South Canyon Fire and published a report (OSHA 1995).

photo
Figure 15. Photograph of South Canyon Fire area and selected points where major events occurred. Photograph by Jim Kautz USDA Forest Service. Top of photo is Southeast.


CASE STUDY APPLICATION

This case study has investigated a major subunit of the South Canyon Fire, the twelve fatalities that occurred on the West Flank of the fire. It was selected due to its distinct environment separate from the three other subunits identified and the severity of the consequences (Butler et al 1998). The other three subunits were the “Main Ridge Group”, the “Lunch Spot Group”, and the “Helitack Group”. Accident investigators and safety professionals have agreed that the difference between a near miss (or near serious) incident and an accident is mainly a matter of luck or adequate recovery efforts (whether intentional or not)(van der Schaff et al. 1991). Therefore by examination of the accident subunit the applicability of the resulting accident method to wildland firefighting incidents/accidents ranging from relatively minor fire burnovers to severe consequences may be transferable. When safety barriers are breeched in occupations where the risk may be high, numerous controls must be circumvented. When the consequences are extreme and complex (e.g. fatalities) then the most stringent barriers must be eluded in order for an accident to occur. Therefore less serious accidents (near misses, etc) which have violated less stringent controls can be investigated with corresponding success. Inferences drawn from an accident investigation method could be applied to the range of failures from near misses to fatality accidents. This thesis would be an “instance of a broader phenomenon, as part of a larger set of parallel instances.” (Feagin et al 1991).


ANALYSIS

The following section outlines the operations used in evaluating the three investigation methods. This section illustrates the three methods of validation, construct, internal, and external, and the case study protocol as a means of reliability. This section includes the criteria selected to aid in determining the most applicable method and the operational procedures utilized. In addition, techniques were included to measure the consistency and reliability of the results.


Criteria and Procedures



Criteria have been established in order to determine each method’s applicability to wildland fire entrapment situations. These criteria were; is the method 1) realistic, 2) comprehensive, 3) systematic, 4) consistent, 5) visible, 6) simple and easy to learn? Criteria were derived from the only known source of accident investigation methods’ evaluations (Benner 1985) and were adapted to this case study. The criteria will be used by independent judges to evaluate the three methods. The judges were given a copy of the methods section in this thesis in order to understand the procedures and criteria. These ideal criteria are defined as follows:
1) Realistic – this method should produce a model that represents the sequential, concurrent, and interactive nature of the events flow. The model must also represent the events interaction with time. The model must also allow for the real-world representation of the events and the inherent risks involved in the work process. Does the method represent real people operating in the real world? In other words, does it have ecological validity? Does this method reconstruct the accident sequence as it would have had to occur, both in time and in space?

2) Comprehensive – the method must provide for the identification of the beginning and end of the accident sequence. It must describe the entire accident sequence so that no gaps in understanding exist. Does this particular method appear to miss contributing and/or root causes to the accident or does it seem to allow for all factors (organizational, managerial, supervisory, environmental, human factors) to be incorporated and analyzed? In other words does it have content validity?

3) Systematic – the method must provide a logical, disciplined approach that allows for mutual support of all investigation members. As information becomes available to the accident investigation, does this method allow for easy incorporation into the ongoing inquiry? Is it an easy to follow, step-by-step approach, that forces investigators to maintain that approach and not deviate as facts become uncovered? Does it exhibit face validity? Does the method’s approach reduce bias that may be introduced by the investigators?

4) Consistent – the method must be consistent and testable based on all available information. If someone else were to utilize this method would they be able to produce the same results? Is it reliable?

5) Visible – the method must discover and present events and interactions throughout the accident sequence that would be easy to comprehend for others and provide documentation as evidence. Gaps in the knowledge or understanding and any assumptions must be identified. The investigation process must be relevant and credible. When someone who knows nothing about the analysis method was to look over the analysis, they would find it easy to see how the results were obtained. Interpretation would be minimal for an uniformed person to understand the accident process.

6) Easy to learn – Due to the fact that most investigators may not have had extensive formal accident investigation training, the method must be relatively easy to learn, understand, and implement without extensive formal training or qualifications. Forest Service investigators are not primarily investigators but occupy other positions. The accident investigation method should be able to be learned in a training seminar of less than 1 week duration. This criterion may not be as critical to the evaluation of the “best” method as the other criteria because an overall best method may require more lengthy training. This criterion will be evaluated on equal standing with the other five and then eliminated from evaluation so as to better understand the implications.

The judges utilized a ranking system of these six criteria and assigned a ranking of 0,1,or 2 to each criterion. They applied each set of criteria to evaluate each of the three methods. A “0” meant that they did not meet this criterion. A “1” meant that they addressed the criteria but not completely and improvement would be required. A “2” meant that they fully met the criteria. This rating is a summated scale. The narrow span of the scale was used to reduce indecisiveness on the part of the subject matter experts, since they were not accident investigation experts. The 3-point scale allowed them to be more exact in their determinations of how each method satisfied that particular criterion. The rating scale also follows Benner’s (1985) approach in that until a more comprehensive scale is developed to better differentiate levels of compliance to the criterion, a more simple direct measurement scale is appropriate. This rating process limited the possible more precise evaluation of each method but provided a basis for determining accident investigation methods that would require further testing in field applications. Since each criterion is independent of the others no weighing factor was applied. Again this follows Benner’s (1985) format he derived from government statute and “did not conflict with one another.” Each criterion was evaluated as having no more importance to determining an appropriate accident investigation method than another. Each was a valuable determinant to the overall assessment.

Five subject matter experts evaluated the three analysis techniques by rating how well they satisfied each criterion. They examined the three methods utilized in the author’s reinvestigation of the S. Canyon Fire. This was a purposeful sampling of occupational experts. Since no known list of these experts exists, they were chosen from the first available. These experts were not accident investigation experts but wildland firefighting experts. This was utilized to most accurately emulate real world situations where investigators may have some investigative experience but their primary occupation and training is not in these techniques. Each expert had at least fifteen years of wildland fire suppression experience and is at a minimum qualified at the Strike Team Leader level. The Forest Service considers experts as those who teach a particular subject, so the author’s classification can be considered conservative. This definition of expert was developed by the author and is a source of author bias but reduced by the conservative definition. To reduce evaluator bias, none of the subject matter experts consulted or coordinated with each other in rating the methods. Preconceived impressions about the South Canyon Fire and its possible causes may have introduced evaluator bias into the ratings. Since they were evaluating the investigation methods and not the reinvestigation of the fire, bias should have been reduced.

The author (also qualified as an expert) reinvestigated the South Canyon Fire using the three methods so that the Subject Matter Experts could see how the methods could be used. They then rated the each method using the criteria previously mentioned and rating how well each method met those criteria. It was stressed that they were to evaluate the methods themselves and not how well I did used each method. The subject matter expert’s evaluation package included a summary explanation of each proposed method compiled from the standard reference literature (DOE 1992, Ferry 1988, Hendrick and Benner 1987). The package also included the author’s output worksheets of his reinvestigation of the South Canyon Fire (see Appendix Figures A.1, A.2, A.3, and Table A.1). A copy of the study approach along with the author’s results of the reinvestigation using the three methods was included. The subject matter expert’s familiarity and knowledge of the basics of the S. Canyon Fire was assumed.

The overall totals for each accident investigation method were assessed to determine the best method. In addition, the degree of agreement among evaluators (interrater reliability) was utilized to assist in the determination of the most applicable method and each method’s individual strengths according to each criterion. Comments by subject matter experts were solicited to obtain individual impressions and evaluations that may have not been covered in the assessment process. As no additional literature was found on the importance of the selected evaluation criteria towards accident investigation methods (and methodologies), the study used a variation of Benner’s (1985) criteria, methods and equal-weight approach. No evaluator or criterion was given more weight than another. This aided in strengthening the internal validity.

As a measure of the reliability between evaluators, percentage agreement was calculated. This was done to access the degree of reliability among scores assigned to each criterion. The higher the agreement the more valid the ratings are. Also an additional measure was computed using the index of Perreault and Leigh (1989). Their index is computed using the following formula:

Ir = {[(F/N) – 1/k)][k/k – 1)]} 0.5 , where F is the frequency of agreements between the evaluators, N is the total number of judgements, and k is the number of categories.

Multivariate techniques to interrelater reliability were not conducted as the number of evaluators and criteria were considered too small and would not constitute any meaningful insight.

The case study analysis used the pattern matching logic to strengthen and validate the results (Yin 1989). By comparing patterns predicted for each method, that is, that they meet selected criteria and are applicable to wildland firefighting, and matching with the predicted patterns (that they fully meet, or not meet the criterion), evidence is accumulated in determination of a most applicable method. Yin stated that though empirically based, the comparison of patterns and their ability to coincide with established criteria can “strengthen its internal validity”. The development of empirical, logically tested evidence would provide internal validation to the case study analysis process.

As mentioned previously, this system of evaluating accidental investigation methods followed an earlier attempt by Benner (1985). He acknowledged the assumptions and bias inherent in such an evaluation. The systematic accident method process, documentation of each event sequence, and independent consultation of subject matter experts reduced the bias. The five evaluators had no prior experience with any accident investigation method so there was minimal pre-study bias as to which method may be more applicable. Although the author used each method to reinvestigate the South Canyon fire, each evaluator was instructed to use the reinvestigation as a means to evaluate the methods and not to evaluate how well the author performed the analysis. The author is not an accident investigator, so the way the method can be used to investigate an accident is critical not the way the author used it. The experts were presented the tools to understand the process and application of each method, and the author’s working example illustrating how it can be used. They were instructed to analyze how well each method did, and possibility could, fulfill each criterion.


CHAPTER - 5 RESULTS

The results of the subject matter experts’ evaluations are presented in Table 1. The STEP accident investigation method received the highest overall rating with a score of 52 total out of a possible 60 (87%). The Fault Tree Analysis method received a rating of 51 (85%). The Control/Barriers Analysis method received a rating of 42 (70%).

For each criterion evaluated by the experts, the STEP analysis method was rated highest along with Control/Barriers as the most realistic with a score of 9 out of a possible 10 (90%). It was rated as the most comprehensive (100%), most consistent (100%), and tied with Fault Tree Analysis as the easiest to use (90%). Fault Tree Analysis was rated as the most systematic method with a score of 12 (100%), the most visible (90%), and tied as the easiest to use (90%). Two of the evaluators rated Fault Tree Analysis highest in overall applicability across all criteria to wildland firefighter entrapments, two rated STEP highest, and one rated both STEP and Fault Tree Analysis as equal. No evaluator rated Control/Barrier Analysis highest.

The majority of reliability indexes and percentage agreements calculated were acceptable within the limits prescribed by Perreault and Leigh (1989) and Kassarjian(1977). They reported that indexes above .85 were very good and below .80 may require reevaluation. The total agreement among evaluators for STEP and the six criterion used to evaluate this method was 80% (0.84, Perreault and Leigh’s (1989) reliability index). The Fault Tree Analysis method was 83% (0.86 reliability index). For the Control/Barriers method agreement was 75% (0.79 reliability index). The overall reliability of evaluator agreement for all three methods was 80% as a percentage and 0.84 when computed using the reliability index. These agreement indexes showed that consensus among evaluators as to their ratings was acceptable.

In reference to each criterion evaluated, the Control/Barriers and STEP method were the highest rated and most agreed upon methods for being realistic. Fault Tree Analysis did not achieve an acceptable agreement index (.63). In evaluating comprehensiveness, the STEP method rated highest for both overall score (10) and agreement (1.0, Perreault and Leigh’s reliability index) and the other two methods had high agreement on a value of “1” (addressed the criterion but needed improvement). In rating each method as systematic, Fault Tree Analysis rated highest in both score and agreement. Agreement was high that Control/Barriers rated a “1” in inadequately meeting that criterion. The STEP method did not receive an acceptable overall agreement index (.63). Consistency was highest using STEP with Fault Tree Analysis and Control/Barriers second. Agreement on the scores for all three methods was acceptable at above the .80 level. Visibility was highest using Fault Tree Analysis and of acceptable agreement. Agreement was high that STEP and Control/Barriers were not adequately visible in their application. Both the STEP method and Fault Tree Analysis were rated equally high as to ease of use and evaluator agreement was high. Although it received a lower score than the other methods, there was not acceptable agreement that C/B was easy (or not easy) to use. When the criterion “easy to use” was eliminated from evaluation, overall percentage agreement scores were Fault Tree Analysis (84%), STEP (80%), and Control/Barriers (70%) and therefore did not alter the findings.

Evaluator’s comments were solicited as to the applicability of each method beyond what each criterion addressed. One evaluator liked the way Fault Tree Analysis visually presented complex events and the way it showed accidents as a chain-of-events as opposed to a single random occurrence. Another commented on the way Fault Tree Analysis led backwards from the accident itself to logically uncover causes or reveal questions that may have been otherwise overlooked. They thought that this method might be better at uncovering managerial/administrative latent factors contributing to the incident than the other two methods. In contrast, one evaluator responded that the STEP method appeared more stringent in revealing underlying human causal factors. They commented that STEP (and Control/Barriers Analysis) provided an approach that was more likely to distinguish more abstract human factors from hard factual data considerations and therefore be better at raising questions into human error causes. The STEP method was cited by one evaluator as the approach that most visually displayed the actor/action sequence of events and identified knowledge gaps in the sequence. All evaluators expressed concern that Control/Barriers Analysis was inadequate in determining causal factors when applied to wildland firefighting. It had strengths in identifying needed and/or compromised barriers at an administrative level but the dynamic and highly variable aspect of the firefighting environment made its application to investigations inadequate. They commented that it did not appear to be an adequate tool to probe deeper into possible human error (and administrative/managerial oversights). The method was good at defining what control or barrier failed but not why it failed.


Table 1. Results of subject matter expert’s evaluation of the three investigation methods. A score of “0” meant that the criterion was not met, a score of “1” meant that this accident method had the ability to meet that criterion with some improvement. A score of “2” meant that the criterion was fully met. Consult text for definitions


.## table 1


CHAPTER 6 - CONCLUSIONS


The results of this inquiry into an accident investigation method applicable to wildland firefighter entrapments showed the Sequential Timing and Events Plotting (STEP) method to be the most desirable method, followed very closely by Fault Tree Analysis. Both methods together met the majority of the goals and objectives of the thesis. The total overall scores obtained for both the STEP and Fault Tree Analysis methods showed the two methods are not likely significantly different. Both were rated higher overall than Control/Barriers Analysis. Each accident investigation method had its strengths and weaknesses as verified by the resulting evaluations of each criterion.

The STEP method received the highest score and highest number of selected criteria that evaluators rated highest and in which they concurred. The criteria that the STEP method rated highest on (realism, comprehensiveness, consistency, and ease of use) showed this method to be the best investigation process in these areas. Therefore, overall, the most applicable method would be STEP. Fault Tree Analysis would be the most desirable method when accident investigators required a systematic process that was highly visible and easy to implement.


APPLICATIONS

Possible application could involve utilizing each method’s strengths in combination to overcome the inadequacies found with each method. A possible co-method approach where STEP is initially used to develop the timeline with actors/actions and events, identify gaps in knowledge, and provide consistency. Fault Tree Analysis would then be incorporated to provide the systematic framework to logically sequence the causal factors, identify any knowledge gaps not uncovered by STEP, and allow for additional multi-investigator input to be utilized. The resulting analysis obtained from STEP and Fault Tree Analysis could then be visually displayed using the Fault Tree diagram to provide an easy to see accident event sequence that would be understandable and informative. Since both methods were evaluated as easy to use, a co-method approach could be relatively easy to implement. In addition, this approach could produce a valuable cross-check and verification approach for each method.

An alternative approach to determining the most appropriate method would be to conduct investigations using both methods individually under a variety of entrapment circumstances to actively assess each method’s capabilities in causal factor determination. This would further validate each method’s applicability to wildland firefighting.


FUTURE RESEARCH

It may be desirable to research and develop a new integrated method that incorporated the strengths of STEP and FTA into a third more comprehensive method. Thus the weaknesses of each model could be accounted for (and the inherent biases of the author and subject matter experts) and eliminated to produce a method better suited to the unique, dynamic work environment of wildland firefighting. Current research into organizational/managerial and human factors involved in accident causation would need to be incorporated into the new model. In order to identify those factors that lie at the root causes of accidents, an updated model should integrate research that focused on high reliability organizations (e.g. air traffic control or aircraft carrier operations), decision making models, risk management concepts (e.g. risk homeostasis), intentional standards violations, and human error mechanisms (See Rasmussen 1997 for research into these converging fields). These fields of research could combine various academic and safety professional disciplines into a singular, encompassing causation model that would more accurately and effectively reflect more deeply rooted failure mechanisms.

Another alternative would be to investigate more thoroughly the possible application of Accident Fault Trees (AFT diagrams) to wildland firefighter entrapments. Love (in press) has extended the capabilities of traditional Fault Tree diagrams to include temporal properties, accident severity considerations, and possible interactions during the course of the accident. These enhancements account for the most significant shortfalls inherent in traditional Fault Tree Analysis.

In hindsight it may be more effective to survey additional evaluators in order to accumulate more evidence as to the best method. It would be more insightful to have investigators who are experts with each method investigate entrapment fires and subsequently compare results.

An additional criterion would also prove more informative in the evaluation. This criterion could be the practical utility of the method to wildland firefighter entrapments. This would allow the evaluators to provide input into the overall applicability.



CHAPTER 7 - DISCUSSION

Results of this thesis have shown the applicability of specific accident investigation methods to wildland firefighter entrapments. Subject matter experts have rated the Fault Tree Analysis and STEP method as the most applicable to this specific high risk work environment. STEP received the highest overall rating, the largest number of high rated criteria, and had acceptable evaluator consensus on all criteria except being systematic(face validity). Fault Tree Analysis was rated nearly as high and with the exception of being realistic (ecological validity), evaluators reached consensus. When “ease of use” was eliminated from the evaluation process (to determine whether that criterion affected the rating), results remained unchanged. Both Fault Tree Analysis and STEP scored a “9” for ease of use and evaluators acceptably agreed on the score. Fault Tree Analysis has limitations in representing sequential and simultaneous events along a time line. It also was limited in only portraying success/failure modes and not the varying degrees that are often the norm in human interactions. But its ability to systematically and logically solicit the cause of a particular event was its overall strength. The STEP process was not deemed as sound in that respect. The STEP method was determined to be a method that was valuable in organizing and collecting data at the onset of the investigation. It was valuable in its ability to follow actors and events along the causal chain and illuminate breaks in the sequence. It also illustrated possible interactions among actors and graphically depicted the accident process in an easy to follow and updateable format. The STEP method would be a powerful tool in the data collection/developmental stage of an investigation. When followed up by a Fault Tree Analysis, any subsequent questions uncovered may be investigated. This second stage analysis could also provide validation or invalidation of the STEP procedure. Upon completion of the accident investigation, the visual display of the root causes, contributing factors, and chain-of-events would be in the hierarchical tree format used by Fault Tree Analysis. Any interested party could then see graphically the sequence of events that have led to the accident. The appropriate and responsible administrators could then have accurate focus points on which to mitigate hazards and institute corrective measures. By the application of both methods in concert, latent agency inadequacies, managerial omissions and oversights, and human factors issues would more likely be identified than by utilizing a single approach.


INVESTIGATING NEAR MISSES

The premise has been raised as to the critical need to investigate incidents that have not led to disaster, the near-misses (Lucas 1991, Reason 1991, Vaughan 1996b). This approach would identify the sequence of events that could have led to a disastrous result were it not for luck and/or extraordinary recovery efforts. Identification of adaptive processes, modes of recovery, and conditions at the boundary of near-miss versus harmful accident situations could be invaluable in proactive measures to prevent future occurrences. Rasmussen (1997) discussed this point when he said that individual workers navigate freely within a work system shaped by objectives and constraints (administrative, functional, safety related). He stated that a worker searches freely within those boundaries “guided by process criteria such as work loads, cost effectiveness, risk of failure, joy of exploration, etc...” Managers supply the “cost gradient” in which a worker searches to identify an “effort gradient”. Therefore, in their search, a worker will systematically migrate toward the boundary of functionally acceptable performance, and when crossing the boundary is irreversible, an accident may result. In a dynamic work environment such as firefighting, many degrees of freedom exist where firefighters must continually validate boundaries and adapt to changes that may be frequent, rapid, and life threatening. Rasumssen (1990) said that removal of human errors cannot and should not be the goal of safety programs. He stated that “ the ability to explore degrees of freedom should be supported and means of recovery from the effects of errors should be found.” Through training (such as simulators), firefighters could subsequently learn better coping skills at critical boundaries and a more effective array of tools in which to successfully identify, adapt, and successfully recover from potentially hazardous situations. Agencies continue to add more rules to cover situations where disasters occur in an environment where all the conditions can never be known (Vaughan 1996b). Skills to avoid entrapments and assess risk at critical boundaries could provide alternatives to the addition of more rules that make completion of the job more difficult (Rasmussen 1997). At the managerial/administrative level, near-miss investigations would aid in locating those latent factors that may have lain dormant at various levels awaiting triggering actions that may result in an accident. Defenses, barriers, and safeguards within an organization can be circumvented when a particular combination of events occurs. Proactive identification of possible “loopholes” within the organization could aid in closing those gaps in the defenses through which accident sequences may occur.

The reporting, identification and subsequent investigation of near-miss incidents is the primary focus of the National Interagency Fire Center’s (NIFC) SAFETYNET ’99 pilot program. This program responds to the Tri-Data report’s recommendation (1998) to develop a system for anonymously reporting safety concerns. Another interagency response to Tri-Data’s recommendations is the Center for Lessons Learned that is currently being developed at NIFC. This Center is a focal point and clearinghouse for information related to firefighter safety. Both the safety data reporting system and the Center for Lessons Learned are vital components of the study’s principle “collect reliable safety data and use it”.

Lucas (1991) proposed “systemic safety management” whereby perceived potential problems, as well as near misses and accidents, are actively solicited from throughout the organization as an integral component of the organizational culture. She cited three key elements vital to the success of a systems approach. The employees must have anonymity and freedom from prosecution in reporting near misses. They must have confidence in management’s policy of forgiveness so they have no fear in losing their jobs. And finally, there must be feedback to employees in the form of implemented error control strategies so that they can see the results of their input. Thus, it is critical that organizations objectively evaluate their underlying safety culture if they wish to institute a near miss reporting system and benefit from the results.

In addition, by reducing the number of unsafe acts, agencies can reduce the number of harmful accidents. For every accident reported, there are numerous unsafe acts that were unreported (Reason 1991). This reduction of unsafe acts could be accomplished by clearly defining employee (firefighter) tasks, adequately teach them how to do each task, validly measure performance of each task, and subsequently reward workers for high-quality completion of the assigned tasks (Kenney 1993). This is clearly a line management responsibility and function. These include, but are not limited to, safety, environmental, and risk management activities. Acquisition and allocation of resources, proper training, and stewardship programs are subsequently dependent on the decisions of top level managers and administrators (Vaughan 1996b, Kenney 1993).

Another avenue of approach that can add more reliability and validity to accident investigations is one in which the National Transportation Safety Board (NTSB) has a prominent role. As Johnson (1999) points out, the NTSB position is outside the Federal regulatory mechanisms that protect the agencies and companies that are investigated. Thus the Board has the independence and autonomy to analyze managerial and regulatory practices that may otherwise be overlooked. Any agency that investigates itself may very well be suspect in its conclusions (whether right or wrong), particularly when the agency itself may be lacking in adequate policy, regulations/standards, and oversight. As in previous wildland firefighter accident investigations, wider issues pertaining to organizational and administrative practices have been generally obscured by more prominent causal factors such as high workload, situational awareness, distributed cognition, and mode confusion (Johnson 1999).

As environmental conditions change adversely, fire ground complexities increase, and agencies continue to be subject to changing political climates, a comprehensive, systematic process to reveal possible failure points is vital. Investigations of entrapment near-misses would provide the insight as to where the system needs reassessment and updated controls. It would likewise provide insight into the adaptive/coping skills that may have kept a near-miss from becoming an accident. There is a need to learn from the lessons that occur without the resulting disaster so that firefighter safety is not so much reactive to major disasters but continually adapting to the dynamic nature of the overall work environment. Reason (1997) calls for an informed culture, one comprised of a reporting culture (accident/incident reporting), a just culture (where rewards and punishments are viewed as just), a flexible culture (where an organization shifts from a bureaucratic conventional operating mode to professional, task expert control during emergency situations), and finally a learning culture (where the organization has the willingness and competence to identify correct conclusions and implement needed reforms). Adequate investigations of near-miss incidents and the development of avoidance/coping skills at vital trigger points (boundaries) could greatly decrease the harmful outcomes to wildland firefighters. By identification of boundaries where successful recovery actions cannot be implemented and the necessary coping/recovery skills near these boundaries, harmful accidents could be significantly reduced. The STEP method used as a primary investigation tool and followed up by Fault Tree Analysis could greatly aid in the ongoing search for a safer firefighting work environment. Both methods offer the means to incorporate more abstract causative factors and more specific root causes as new “composite” error identification models become functional.

This thesis showed, not only the two most applicable accident investigation methods for wildland firefighter entrapments, but the critical need for requiring a method to be utilized. In order to determine the proximate causes of entrapments, the trends, and monitoring of safety measures, it seems equally critical to develop the data base of human factors causes and near miss incidents. To develop and institute a safety culture through a learning culture, a total commitment by management must be a leading priority. The current trend in firefighter entrapments can only be reduced by top level administrative support and low level firefighter dedication to safety. A change in the way we do business, a change in the safety culture, can be and must be achieved by the combined efforts of all levels of the wildland firefighter community.