Note: comments are based on personal experiences as a member of NTSB's Cross Modal Group and later as Hazardous Materials Division Chief who conducted, directed, participated in, provided technical support to about 45 investigations and public hearings in all modes at the Board; observed, documented and analyzed investigation processes and reported results in numerous papers; and developed curricula for and taught investigators, among other activities. I am retired and have no commercial interest in the comments. For more data see http://www.iprr.org/admin/lbcv.html

Overall impressions

1. First let me say that I believe the project to be a very desirable undertaking - the professional investigator issue needs airing, with notions of investigator certification looming, and can give direction to numerous follow-on works, so I hope nothing I say will in any way discourage the authors from pursuing their work. My work indicates it would be a major disservice to all concerned if future investigator certifications are based on unchallenged and unverified assumptions, perceptions and opinions about the tasks and functions necessary to produce desired investigation results by governmental boards, which seems to be the direction some are moving. I commend the authors for suggesting research is needed, and hope my observations will help them.

2. Based on the description of the methodology and its reported application, I have the impression that the content seems more heavily influenced by experienced opinions than by primary observations of the tasks and functions involved, and the rigorous analyses of such observed data. One potential consequence is the uncritical acceptance by the authors of much of the "conventional wisdom" espoused by "experts" and academicians which I have seen impede demonstrable breakthroughs in the investigation field. For example, until outside experts got involved with the scientific investigation of the wind shear phenomenon, the domain experts using domain-accepted investigation techniques typically attributed such approach accidents to pilot error. Another consequence is the use of a large number of abstractions and some cliches in the paper, resulting in ambiguities and occasional internal contradictions, depending on how the ambiguities are resolved.

3. While the authors devote much attention to the social responsibilities of the Board and IIC, I found little or no attention to Boards' responsibility to individuals who may have had some role in the incident. My observations of investigations have disclosed that two significant responsibilities of the IIC are to ensure that anything reported about what such individuals did be accurate, verifiable and fairly stated, and to ensure that the risks to the reputations, livelihood and mental health of such individuals are not exacerbated by what the IIC does or allows to be done. These obligations should to be explicitly noted, based on my observations of the consequences of failures to do so, and examples I am sure the authors could find or recall.

4. I noticed that the discussion of domains does not include "accident investigation" as a discrete domain, although numerous aspects of the report suggest that investigation as practiced by Boards have many of the attributes of a distinct domain, including Boards' unique role in a society, specialized knowledge, skills and training required, public trust, and authority, further supported by the authors' conclusion that investigation is a profession. This position creates some internal inconsistencies within the thesis. The authors may wish to rethink this position, and its many implications for their study, future investigation policies and practices, and training and certification of investigators within the investigation domain.

5. In the discussions of the Board activities, one aspect of Board learning activities that is commonly overlooked - and was here - is their role in bringing about general improvements in the "learning from investigations" process practiced by the all other accident investigators in a domain. Would it not seem appropriate for Boards to set as an objective the advancement of the state of the art of investigation for that purpose? Without that point, it must be assumed that the state of the art is deemed satisfactory by the Boards. That issue may be beyond the scope of the thesis, but it would seen that IICs need to be aware of the "best" tools (as shown by the results they produce) and their use both in their investigations and in other investigations. Wouldn't the inadequate quality of prior investigations by others be a legitimate area of inquiry? I didn't notice that this was mentioned. If it is raised, a Board needs to be prepared to assert what would be better.

Comments about report

My approach to the review was to read the summary, recommendations and conclusions in that order, and then review the main body of the work.

As a reader of the document, I discovered I had difficulties coupling some of the recommendations to the data or arguments developed in the report. It would be most helpful to note for each recommendation the specific conclusions or report sections that support it.

The first recommendation rested on the OECD recommendation, which posed no problem for me - it was in keeping with a governmental directive. The second recommendation about domain emphasis seemed to rely on the unproven assumption that domain knowledge is essential to build trust, when my observations frequently revealed that can be more of a handicap than an advantage because of the many unchallenged assumptions that are embedding in domain knowledge and experiences - which leads to the group think about which the authors rightly complain.

I could find no substantive arguments to support the recommendation that ITSA should be tasked for this recommendation, and given the "group think" of present Boards about the investigation process it seems the authors could mount a persuasive argument against it.

While acknowledging the argument for different levels of investigation competence, I found little to support the grades specified, and in fact have described findings that would suggest a different sort of competence differentiation.

The fifth development recommendation appears on its face to be somewhat redundant to 3.

On reading the Conclusions, I experienced a similar difficulty trying to couple the conclusions to sections of the text. While the points made seem to be valid, it would be desirable for the authors to more closely couple the conclusions to specific sections of the work, to both verify the arguments as an internal quality check, and reassure the reader of their validity.

Comments about report sections

I found it interesting -and disappointing - that the authors report and describe major problems with their current investigation process (e.g., the Voskhod case reinvestigation, reanalysis, group think, delays, etc.) and then accept that process as model for their description of the process IICs should manage. This seems to reflect the generally uncritical acceptance of the current thinking and investigation models like the aviation model among governmental boards. Although the authors indicate more research is needed to establish criteria for investigators, no mention is made of the shortcomings in or the need for research of the investigation process IICs manage. It is evident from the contents of the paper that the latter must be undertaken before investigator qualification criteria can be legitimately established.

A second problem I encountered was with definitions.

Within this framework definitions are critical. The authors use wikimedia as their primary definition source. Missing from that source are elements from dictionaries which have a determinative bearing on the thesis. Two examples are professional and domain.

For example, a crucial but missing explicit element in "professional" is "instruction in the scientific, historical or scholarly principles underlying such skills and methods" included in Webster's International Dictionary (english language.) I discussed this issue and the need for basic investigation "theory" for the "professional" Air Safety Investigators at some length in a 1975 paper published in the SASI Proceedings. Its absence (in 1975) was lamented as deficiency that should and could be addressed, and it has been - although not yet to a consensus level.

The relevance to the thesis is how the scientific, historical or scholarly principles underlying such skills and methods (theory, for our purposes) affect the domain knowledge and skill questions, and "shared assumptions" the authors mention. If the theory is a general investigation theory, that knowledge and related skills should enable an investigator to conduct any kind of investigation by applying the theory with the tools it produces. On the other hand if no such theory is available, then knowledge of the domain within which the phenomenon occurred is the fall-back expertise on which the investigator must rely. That however creates certain subtle - perhaps even unconscious -but very important investigation biases such as uncritical acceptance of implicit assumptions within the domain, the conventional wisdom, and long standing practices, limits of knowledge prevalent in a domain, thus impeding discovery of "unknown unknowns" needed to bring about substantive changes in behaviors of people, objects or energies. An investigation theory -based process should not suffer these impediments. There has been nothing as illuminating in my investigation experience as domain-independent questions like "I don't understand - could you explain that to me?"of a "domain expert" until I could visualize in my "mental movie" what was being described. (It was always interesting to me when we got to the point where they would lose their tempers because they were beginning to recognize they couldn't explain it either: It didn't work they way they thought it should!) But I ramble.

I have previously mentioned the domain definition problem.

A third problem is that the paper has some internal contradictions. Worth repeating is the apparent disconnect between Appendices I and E. I describes problems encountered by the IIC with the process used, yet E seems to accept that process as the basis for a description of the process. The Appendix I Voskhod investigation review and analysis, which contains the preponderance of the primary data about the IIC's investigation tasks in the thesis reinforces this point. Discovering investigation problems when writing the final report demonstrates grave deficiencies in the investigation process and practices (and the tools map) leading up to that stage of the investigation, which in turn reflects the problems with the underlying "theory" on which they are based. Am I missing something here?

An area lacking demonstrated proofs is the assertion that Board trust built and confidence by satisfying domains. Having observed domain experts undermine such trust and credibility due to contentious or subjective investigation findings, usually in the selection of "probable cause" but in other aspects as well, I have been forced to conclude that trust is built by the excellence and unassailability of the investigation and its findings, rather than the reputation for its domain expertise. This of course shifts the emphasis from domain expertise to investigation expertise. The authors may wish to to reconsider their arguments in support of the domain expertise model.

The Appendix I Voskhod investigation review and analysis, which contains the preponderance of the primary data about the IIC's investigation tasks in the thesis reinforces this point. Discovering investigation problems when writing the final report demonstrates grave deficiencies in the investigation process and practices (and the tools map) leading up to that stage of the investigation, which in turn is a reflection of problems with the underlying "theory" on which they are based.

As an aside, my inquiries have revealed that investigation practices have been introduced largely by other disciplines like engineers, pilots, lawyers, chemists, statisticians, etc., in the absence of emphasis on fruits of investigation research. While the influence of the Reason, Perrow, Rasmusson, Vaughan and others is noteworthy for safety thinking, their research was not based on direct observation, documentation, experimentation and testing of investigations or investigation processes. The authors' 'experience with the Vozkhod investigation illuminates the consequences of such influences, and how they can lead to group think. For example, the "fact finding phase before analysis" (3.2) while suitable for preparing a jurisprudence case does not reflect the iterative progress of efficient safety-oriented accident investigations; in another internal contradiction, the report seems to accept this (p 19 vs p 45.)

The authors did a good job of explaining the development of their graphics. While some of the graphics that were developed are interesting to me, it is unclear how the outputs were validated against observed or observable investigation practices. For me, one of the elements of the study I was hoping to see, to support the argument for "professional" were the task flows and a set of IIC task descriptions and specifications, developed from observations of IICs performing those tasks. The RASCI chart provided a portion of this data, but the level of abstraction of this and the other outputs like the various maps left the tasks vaguely defined, at best, and inadequately defined for defining investigation procedures called for in the paper, based on my observations of and experience with procedures documentation tasks and their assessment.

Random notes

As I read through the paper, I made some rough contemporaneous random notes about thoughts that occurred to me, which may or may not be of interest to the authors.. Anyway, here they are.

p8-9 The fundamental principles lack any mention of the investigation technology used by the Boards. While in part a reflection of the deficiency cited earlier in the definitions, it probably also reflects an unawareness of the significance of investigation technology choices among Board members and probably among many investigators, as I have observed. The fact that it is not discussed constitutes what I am persuaded to view as a major deficiency. The thesis methodology used would be unlikely to detect this deficiency, based on my research. Since the Meijers Institute seems focused on "Jurisprudential Research" it is not surprising that this important principle was not recognized: legal research methods do not lend themselves well to process investigations, particularly with their heavy reliance on second hand testimony or indirect "evidence" and its analysis and "truth testing" methods.

p 8 et seq - What's the difference between sector and domain as used in this paper? Use is about evenly divided.

p 9 The focus on the investigators is a rational approach, IMO. The manual could be a logical and readily constructed follow-up, provided the research suggested by this thesis has been completed, thoroughly vetted and proven (reinforces the need to get it right.)

p 10 Re "professional" definition deficiency, discussed above, the problem carries through to the "readiness" considerations of IICs and investigators.

p 12 skill map is very abstract, open to multiple interpretation

p 13 structure study to get "objective view" - interviews structure is subjective? How to maintain objectivity of data from interviews is IIC's (and authors') challenge. Apparently study method relied on focus group for discovery p 19 life cycle description of investigation is flawed - when observed, it is demonstrably different than that linear depiction - it's iterative as good investigators know. I would point out that "failure" is a conclusion drawn about certain aspects of a scenario that has been documented, much like error and other abstract and pejorative terms widely used among uncritical investigators without the accompanying standard against which actions are judged. See note about respecting the rights of those involved in accidents.

p 24 It's not crisply clear what the authors think constitutes "success" for an IIC when he functions as a professional - successfactors is not very definitive. p43 suggests success is leading to new knowledge or skills.

p 29 Canadian precedent raises another IIC task challenge: how does IIC keep parties from introducing spurious hypotheses and irrelevant data into an investigation? (Ways to do this are available and might be mentioned.)

p 30 - 4.2 Employing subject experts. On ICC task basis, IIC needs to be able to specify task, methods and outputs delivered to IIC by experts, and know reliably how their work products will advance the investigation. I have seen numerous cases where the test or inspection planning deficiencies lead to misdirection, delay and contention. (I call one such incident my example of a "$300,000 mistake!) Don't remember this being mentioned.

p 30 Nemeth et al make some statements that relate to above point - it can be controlled by the IIC (and has been) without the intervention of the folks mentioned in the quote. Using the quote without supporting evidence or qualifications give unwarranted credence to their allegation. 1 covers a lot of ground, and brings together a wide range of information about public board investigations

p 31 "Progress is main issue... " how is that progress best measured or demonstrated? BTW, I was glad so see content labeled as opinion in this section.

p 32 I would be interested to know how the theme list here compares with the authors' experiences in actual investigations. As individual responsible for the organization, conduct and control of an investigation, IIC's success is development of an unassailable description and explanation of what happened (3.2), objectively, efficiently and expeditiously. This requires team leadership, but, when successful, satisfies Board corporate requirements, establishes and maintains confidence and trust, and meets public demands for understanding of incidents. The study shows why the emphasis for the IIC needs to be on the investigation process and direction of the investigation, rather than domain knowledge and skills.

p 35 re shared delusions - I love that! It describes perfectly what every IIC has experienced during investigations where one of the participants tries to - and sometimes succeeds in - selling others on a speculative hypothesis that will not stand logical scrutiny against the data available. I would however take issue with some of the other assertions about the data collection and handling during an investigation, again based on observations of the more successful investigation practices.

p 36, 37 6, Ambiguity between the "fact finding" and "analysis" "phases" of an investigation masks a problem I had with the study. The authors may have been misled by the SEReDA study's ambiguity. This should be clarified, because my inquiries have disclosed that disambiguation is essential to overcoming the problems cited in the Voskhod case, with its need to go back for more investigation - twice. Along with avoiding "group think", rectification of the process which produced those problems is one of the IIC's major challenge, but I could not find that explicitly addressed in the paper. The report that Tripod was used twice is a clear signal that it does not meed an IIC's needs. So, what does? (this is a challenge I have addressed in past works)

p 44 Fig 7 assertion that IIC should be able to to use method is not supported by arguments or data in the paper that I could find. In the absence of such rationale, readers are likely to view such assertions as dogma.

I don't remember what triggered this thought as I was reading the paper, but I think the - technical investigation methodology selection issue needs to be recognized if this is function of IIC - it relates to "shared basic assumptions" mentioned in the Culture section, but unfortunately what is included in the "shared" assumptions is not enumerated in much detail. Left untreated, this is a noteworthy gap in the study.

p 37 what data can be provided to support the pseudographs shown? How were they derived, and are suggested quantitative values supported by data in any way? I was particularly interested in the discussion starting on p41, as that can easily become the basis for subsequent certification efforts, and thus merits very careful presentation. A few of my observations follow:

p 42 The authors do not demonstrate with arguments or data or prove that it is necessary to develop a likely hypothesis or initial diagnosis during an investigation. As shown by their Voskhod case, the damage this can do to an investigation can be severe, poisoning the wells for the data search and organization, misdirecting task assignments, biasing other observations, and more. Hypotheses are useful for guiding speculations to bridge gaps in data arrays during investigations, but not for speculatively describing on the scenario. I caution the authors about falling into this academically attractive but performance debilitating trap.

p 43 infers that successful inquiry leads to an increase in knowledge or in skills; is this the measure of IIC success? I think not, and I suspect the authors don't think so either.

p 43 great deal of discussion about systems, but no argument is made to make the case that unless data developed during the investigation drives direction of investigation toward various system elements, system thinking by IIC is demanded. It is not demonstrated that the Rasmussen abstract model has a role in investigation process for IIC, and what that role should be, for example.

p 44. not clear what "investigator should look into SMS to find out why barriers failed" means; isn't the purpose rather to compare the predictive analysis with what was found? The SMS simply defines expected performance, against which actual performance can be compared, but it is a valid data source for establishing expectations. Barriers failed promotes "group think"

p 44 Fig 7 is example of some of the dogma in the thesis - assertion that IIC should be able to to use SADT method is not supported by objective data in paper

p 45 - was "logic tree" deliberately left out as an analysis method widely used during investigations? (I have seen it used for "gap filling" during investigations, and have used it myself in circumstances where data could not be found.)

p 46 re 8 - this section seems to overlook a critical part of the "evidence" gathering function (I find data gathering more definitive than evidence gathering- deciding whether something is "evidence" requires a subjective judgment during much of an investigation) From my observations, what is missing is the documentation aspect of the data acquisition task, particularly the transformation of observations into natural language describing what happened or why it happened. This is not a trivial issue.

p 47 doesn't IIC direct all the parties to investigation? Governmental Agency should "own" the investigation, as a matter of national policy, and if this is not the case, it could be a useful point to raise in the study. Board should not just be one of the investigating organizations, otherwise IIC's authority is undermined.

p 48 re Domain knowledge. The assertion is not supported by arguments leading to the conclusion at the bottom of the page. I recognize it is a widely accepted perception, but only because it has not been challenged by "professional" investigators, in whom confidence is built by the results they produce. A strong case can be made that "domain knowledge" can thwart IICs during investigations because of others' uncritical acceptance of the domain's perceptions, practices and "conventional wisdom." (Voskhod case) Domain knowledge can be purchased by the IIC as needed, but that purchase must be managed to meet the IIC's needs for a specific part of the case, as I have tried to explain. Identifying and managing "domain knowledge" needed is one of the core issues in the role vs profession dichotomy. p48 15 There are two investigation quality control issues - the quality of the investigation process and the quality of the work product. IIC must manage both using quality assurance tools which he or she must master. This is not clear from this text.

p 49 16 It is good that you elected to avoid detailed discussion of the grade issues!!

p 66 No case is made for classifying data during the investigation - what is the purpose other than to raise the level of abstraction, which masks understanding of concrete behaviors that need to be understood to bring about observable change. Hard to observe abstract behaviors objectively.

Enough. I got to this point and realized I risk losing my audience. However, I felt the work sufficiently important to invest more time than I usually would in offering comments. i hope the authors take it in the spirit I intended.

Ludwig Benner

return to top