An Overview of Key Points

B. John Garrick | August 26, 2001

Introduction

Under the auspices of the Society for Risk Analysis, the foundations of Chauncey Starr and B. John Garrick sponsored and funded a special symposium on quantitative risk assessment. Dr. Garrick served as Chairman of the meeting and Dr. Starr as Honorary Chairman. The symposium involved leading risk scientists and practitioners in the United States and was held at the National Academies Beckman Center in Irvine, California, May 31-June 2, 2001. Participants were John F. Ahearne, Elizabeth L. Anderson, George Apostolakis, Vicki M. Bier, Gary Boyd, Robert J. Budnitz, Andrew C. Campbell, Richard T. Carson, Robert F. Christie, Allin C. Cornell, Louis A. Cox, Jr., Paul Davis, B. John Garrick, Willard (Bill) C. Gekler, Yacov Y. Haimes, Jon Helton, David H. Johnson, Stan Kaplan, Harold W. Lewis, Robert J. Loomis, Richard A. Meserve (dinner speaker), Joseph A. Murphy, D. Warner North, David Okrent, Elisabeth Paté-Cornell, Paul Portney, Chandra M. Roy, Michael G. Stamatelatos, Chauncey Starr, Ross Starr, Theo Theofanus, Detlof von Winterfeldt, Chris G. Whipple, David W. Whittle, Richard Wilson, and Edwin L. Zebroski.

The purpose of the symposium was to provide the foundations with substantive source material to guide them in their pursuit of ideas, programs, and activities that will advance the theory and application of quantitative risk assessment (QRA) in the service of the public good. The goal is assurance that the discipline of QRA evolves on the basis of solid and defensible technical and scientific principles, and that it does not get lost in the other important but more administrative risk disciplines having to do with communication and management. This goal is based on the belief that the underpin and quality of meaningful risk communication and risk management of complex systems is ultimately dependent on a competently performed and scientifically based quantitative risk assessment.

The approach to the 2.5 day symposium was to keynote several risk topics with discussion papers. There were 10 invited papers to keynote the discussions. The papers covered QRA methods, practices, applications, regulations, quality, criticisms, and the QRA and science debate. The symposium closed with a half-day session on what it all means. The "what it all means" part of the symposium focused on what the participants thought was necessary to support the goals of keeping the discipline scientific and in the main stream of serving the public good. Some of the highlights of the topics and issues presented follow.

Status of the Methodology for Quantitative Risk Assessment

The symposium began with a paper that was designed to sharpen the focus of the meaning of quantitative risk assessment and to provide an overview of just where the discipline is in its maturity cycle. Some of the key points of the paper are highlighted.

• Generally, the QRA methodology has matured as a technical discipline and is judged to be in reasonably good shape. The shortcomings appear to be more a property of the method of application than the scientific principles of the methodology. Part of the continuing problem is what is described as "linguistic chaos" in the sense that the language of the risk sciences has not fully matured, although there has been progress. In particular, there lacks uniformity in the definition of such fundamental terms as probability, uncertainty, and variability.
• QRA's position in the applied science field is a component of the decision sciences. QRA is key to better decision-making, but the challenge is to get people to buy into better decision-making. In decision analysis QRA not only quantifies the always important public and worker safety issues, but it provides a methodology for quantifying the other attributes, such as costs and benefits, that may affect the outcome of a decision. In particular, QRA does the probabilistic arithmetic and produces the output curves that characterize the various decision options. While the "set of triplets" definition of risk is evolving as the definition of preference for technological systems, it is undergoing refinements. The refinements are primarily to make the definition more rigorous and responsive to scientific scrutiny. An important contribution is some new developments in the theory of structuring scenarios.
• The parameters of QRA are quantified from the evidence, by processing all the evidence through Bayes' theorem. Evidence takes the form of direct evidence from random sampling of the system being analyzed and indirect evidence such as related, but not direct experience. The latter evidence requires creative methods of analysis and is often the primary evidence for low probability, high consequence events.
• A soft spot identified in the QRA and decision analysis framework is in what was described as the "affect" area. Affect, refers to the class of phenomena containing emotions, feelings, intuitions, etc. No easy answers were offered on how to quantify these "affects" in QRA. The general rule of telling the truth and including as much as reasonably possible in the scenarios was encouraged.

The Practitioners' View of QRA

Four discussion papers were presented from QRA experts mostly from the view of practitioners. Three important programs were the focus of the practitioners' papers, nuclear, space, and chemical stockpile disposal. The fourth paper crosscut several issues, but emphasized the QRA quality considerations. It is clear that the nuclear industry is the most advanced in their use of QRA, the space industry appears to be the most ambitious, and the chemical stockpile program, perhaps the most pragmatic and in some respects, the most successful. Among the key points coming out of those papers were the following.

• Regarding the nuclear program, QRA while most advanced still lacks total commitment especially in the U.S. Nuclear Regulatory Commission (USNRC). The most committed are the QRA practitioners at the plants, followed by the QRA groups within the USNRC. The theme for the immediate future is risk-informed regulation, which has not played out enough to know just what it means, but there are worries that it means added burden, i.e., all the old stuff plus QRA to meet licensing requirements.
• The evidence is strong that the QRA work at the nuclear plants has paid off. Some of the performance indicators are better maintenance, improved equipment availability, increased generation, reduced predicted core damage frequencies, outage reductions in frequency and duration, and improved technical specifications.
• In the space program QRA had a very rocky start and was rejected for over two decades. Re-acceptance has been slow, but started with revamping some of the safety practices following the Challenger accident. The current program is robust and ambitious. All the major programs (space shuttle, international space station, Mars sample return mission, etc.) are aggressive in their use of QRA. The priority is to achieve a total QRA capability in-house. As the National Aeronautics and Space Administration has put it, we are safer, better, and cheaper (higher productivity) as a result of applying the principles of QRA.
• The chemical weapons stockpile disposal program represents one of the most successful applications of QRA to real world problems. QRA has been a key element supporting risk management while meeting all program requirements. QRA has been applied to all major elements of the chemical stockpile disposal program including public and worker safety, storage, and disposal of chemical weapons. Basic methods are from the nuclear program with new challenges in such areas as explosion effects modeling, source term characterization (multiple agents and munitions), and human dose-response for chemical agents.
• All of the practitioners spoke to the importance of numbers. During the incubation of QRA applications, there was emphasis on the "insights" obtained from the QRAs, not on the numerical results of the risk measures. This was thought to be an easier sell. Now there is a strong feeling that insights are indeed an important result of the QRA process, but the numerical results matter as much because they shift perception into actions providing the numbers include the quantification of the uncertainties.
• Specialization and organizational culture were offered as two obstacles to the greater application of QRA. The technical nature of QRA sometimes obscures its role in the larger picture of risk management. Culturally, things change slowly. For example, experience with a set of regulations not tuned to the QRA thought process creates mind-sets that are hard to change, especially if the regulators think that what they are doing is working just fine. After all, there view is that we have a large investment with the current system and it seems to be working, why change it? The answer probably lies in cost-benefit considerations that come from decision analysis, not risk assessment by itself.
• There was considerable discussion at the symposium that a QRA should generate realistic and reasonable assessments of the risk, not conservative or bounding results. For example, QRA should provide a benchmark for the regulators to use as a scientific basis for how conservative they should be in implementing their regulations. Without such an approach there is no calibration of safety margins. The participants were careful to point out that adopting the philosophy of realism for QRA does not imply that there isn't a role in safety and risk assessment for bounding analyses. To be sure, an efficient risk assessment has its scope defined by the use of bounding analyses to eliminate the consideration of many unimportant contributors to risk.
• With respect to future applications of QRA, one sobering observation at the symposium was the lack of excitement among university students about the importance and role of risk assessment as a scientific discipline. One line of thought was that this may be a sign of the maturing of the discipline in the sense that the excitement comes not from risk assessment but how QRA greatly enhances the implementation of the traditional disciplines such as structural analysis, engineering design, and decision analysis.
• A point of discussion was the need for "better analyses." One of the problems is that the methodology of risk assessment far outpaces what is often employed in specific assessments.

QRA from the Perspective of QRA Thought Leaders, Decision Analysts, and Senior Scientists

Five discussion papers were presented on various topics concerning the role of the risk sciences. The topics covered regulatory considerations, QRA quality, decision analysis, the debate on risk assessment and sound science, and the reasons against the acceptance of risk assessment. Several key points were made, some of which are highlighted.

• The regulatory agencies are cautious about becoming over dependent on QRA and continue to practice an integrated decision making process; a process that must be risk-informed, but a process that must include extensive open deliberation. The approach is considered reasonable by the regulators and in their view it relieves the burden of expecting QRA to be perfect. In the regulatory world progress has been slow towards regulatory change based on QRA. Generally, regulators have been unwilling to give up existing regulations. Regulators are still struggling with several issues such as metrics for risk and how to get public acceptance of rules based on risk.
• The complexity and specificity of a QRA complicates the development of prescriptive standards to assure QRA quality. The traditional "design-to" standards tend not to be compatible with the QRA process and the varied nature of risk-informed decisions. As noted later, the development of a protocol for practicing quantitative risk assessment appears to be a more productive path forward.
• There was complete agreement among the symposium participants that the context of QRA should be to aid the decision-making process. A number of recommendations were offered on the best approach. These included assurance of an appropriate decision frame, separate exposure of all facts and values, the making of value judgments explicit and transparent, the importance of public acceptance, the need and problem of constructing scales for each attribute, and the importance of explicit consequence assessment. There was much discussion about the role of politics and conflicting interests in decision analysis. The question was left open as to whether they should be excluded or forced into the analysis. Disappointment was expressed on the lack of using QRA in setting public policy and public decisions.
• There was much discussion about the importance of properly "framing" a decision analysis. This applies at the level of the risk assessment as well. There is the tendency to imply much more scope than is actually implemented. The result is disenchantment about the completeness of the analysis. For example, the total risk of a system even from just a safety point of view might be much more than is represented by the safety or risk measure adopted. The public may be thinking the risk from all threats when the analyst is really only considering a limited set or a single threat.
• The topic of "hypothetical fears" and the role of QRA in addressing them was presented and discussed. The fear of radiation even of extremely low levels was an example cited where rational thought processes such as the application of QRA have not been successful in exposing to the satisfaction of the public the context of the risk relative to other threats to public health and safety. Nevertheless, the point is made that the discipline of QRA needs to be applied to such large issues as global warming and genetic modification while "recognizing the intangible nature of many public fears" and the limited effect it may have on public policy.
• Risk assessment and the sound science debate was a topic of the symposium. The point was made that the sound science debate "is mostly unsound and that discussions about junk science are mostly junk." Supporting this point was the observation that many people think the science is sound if it supports their view and it is unsound if it suggests a different course of action. No one is for unsound science it would seem. The term seems to be the result of the need for a "whipping boy" and has little scientific basis. The discussion emphasized the need to practice sound science in decision making using all the existing tools such as QRA and cost/benefit/risk analysis, but in the end the only really important consideration is what's acceptable to the public.
• Why is QRA not more successful? The discipline suffers from misunderstanding and mistrust. The public struggles with its mathematical image and especially with the concept of probability. The struggle between classical statistics and Bayesian methods doesn't help matters by leading to confusion of what is meant by something as basic as "probability", which brings us back to the subject of linguistic chaos. Also, Congress has not helped by its own failures in bringing QRA into the legislative process. Finally, there is the erroneous belief among many that there is an alternative to risk assessment and it is "certainty". Societies exist on the basis of numbers that don't convey uncertainty and, thus, we are lulled into thinking certainty in the face of uncertainty. Changing such cultural impacts requires patience.

The Path Forward

The symposium was considered very successful in that many ideas and exchanges were presented on how best to achieve the goal of advancing the risk sciences for the public good. The underpin of the discussions was on how to keep QRA moving in the direction of a discipline founded on strong scientific principles. It was clear that this was an outstanding group of leaders in the risk field. There was a character about the meeting that suggests that there is more work that this group wants to do and the dialogue should continue. The continuation of the dialogue is taking the form of exchanges between the participants and organizers of the symposium prompted by this initial pass at identifying issues and calling for the development of a protocol for quantitative risk assessment. Other similar symposiums may be planned in the future.

One of the outputs from the symposium was the beginning of something to be called, "A Protocol for Developing and Practicing Quantitative Risk Assessment." The thought is that while it may not be practical to develop a set of standards for quantitative risk assessment, it surely should be possible to develop some basic guidelines along the lines of a protocol for QRA. Such a protocol could go a long ways towards addressing the "linguistic chaos" discussed at the symposium. For the sake of continuing that thought and initiating an exchange between participants of the symposium, the following principles on which to base a protocol are offered. The principles are as discussed at the symposium with minor edits reflecting some of the comments received.

Principles on which to base a Protocol for Developing and Practicing Quantitative Risk Assessment

• Risk is defined as the set of answers to three questions: What can go wrong? What is the likelihood of that happening? What are the consequences if it does happen? This definition is known as the "set of triplets" definition of risk.
• Quantitative risk assessment is part of the decision making process and is thus, a component of the decision sciences.
• The role of quantitative risk assessment in the decision process is to assess the outcomes of all the options being considered in the decision.
• Since these outcomes are in the future, there is always some degree of uncertainty in their assessment. This uncertainty should be quantified as part of the assessment.
• The form in which uncertainty should be quantified is a probability curve over the space of possible outcomes of the option.
• The task of QRA is to produce these probability curves for each decision option. This is what we mean by "quantifying risk." QRA by itself is not a decision analysis, but an analytical process that (1) calculates and quantifies the risk component of a decision, and (2) the uncertainty of other factors considered in the decision.
• In carrying out this task the QRA should use all the relevant evidence available. A QRA carried out in this way is "an evidence based" QRA.
• QRAs should strive to be more evidence based, rather than based on opinion, obscure assumptions, politics, personality, wishful thinking, etc. It should tell the truth about our actual state of knowledge rather than present conservative or bounding results. It should be the point of reference for introducing conservatism and "defense-in-depth" in our decisions.
• QRA is thus the linkage (logic) between the desired probability curves, which are the input to the decision analysis, and all the relevant evidence, especially our experience with contributing events.
• QRA is an open system in terms of the "outcome variables" in the decision process. Whatever outcome variables are of interest to the decision maker(s) should be quantified as part of the QRA process. The specifics of the QRA process are thus tailored to the specific decision problem at issue.

Excerpts from a letter from Chauncey Starr to John Garrick commenting on the above Overview of the Starr/Garrick Symposium (August 28, 2001):

"The QRA is like a yardstick for risk. It doesn't construct public policy….we would not expect a house to be built without carpenters using … a measuring device, but we leave it to the architect to design the structure. This has to be clarified to those fearful of risk analysis taking away their power of decision."

"(We) need (to emphasize the) quantification of all inputs, even so-called intangibles. This includes empirical data, analogs, scale of 1-10 for intangibles, etc. (The) objective is transparency of the modeling process vs. hidden personal bias for predetermined result."

"A science based QRA should disclose the uncertainty (probability distribution) of all in the inputs and outputs…"

"It is not a matter of educating the public about probability theory – rather it is the explanation of what the results mean and (how design philosophies such as) 'defense in depth' reduces public risk."

"Not enough is said about the monetary costs associated with the consequences. … Not enough is said about the 'benefit' side of risk analysis."