Also, post-hoc analyses of a few events are plagued by the problems of hindsight bias. The many day-to-day predictions that were correct are ignored, especially true negatives (e.g., no credit is given for not having invaded other countries that did not have these weapons). The dangers of forecasters being evaluated on the basis of a few events are obvious. For example, public assessments of the IC in this century are largely based on missing the 9/11 terrorist attacks and falsely claiming that Iraq had weapons of mass destruction (WMD). These events are more likely to be failed rather than successful predictions. Without scorecards and assessment of accuracy of the many forecasts an individual or organization makes, judgments of forecaster performance are likely to be based on a few spectacular, newsworthy, atypical events. It is reasonable to expect similar benefits if these methods are applied in intelligence analysis. As the medical examples illustrate, rigorous evaluation of physician judgments and practices using the methods proposed in this chapter have improved medical outcomes substantially. That, combined with time pressure, can generate considerable stress for the person making the forecasts and detections. In both medicine and intelligence analysis the stakes are often very high. Just as physicians often have to make quick assessments based on limited and sometimes conflicting information sources with no two cases ever being quite the same, so too intelligence analysts evaluate and characterize evolving situations using partial information from sources varying in credibility.
Although medical judgment tasks are not perfectly analogous to IC analyses, there are enough strong similarities to make the examples useful. The chapter relies extensively on recent advances in assessing medical forecasts and detections, where signal detection theory specifically and evidence-based medicine more generally have led to many advances. This chapter suggests methods for improving the assessment of accuracy. As discussed extensively in Tetlock and Mellers (this volume, Chapter 11), Intelligence Community Directive Number 203 (Director of National Intelligence, 2007) emphasizes process accountability in evaluating IC performance rather than accuracy.
However, in a problem that is not unique to the intelligence community (IC), forecasters are notoriously reluctant to keep scorecards of their performance, or at least to make those score-cards publicly available. Being able to assess prediction accuracy is especially important when evaluating new methods believed to improve forecast performance.
Those making the predictions need to know how well they are doing and if they are improving. Consumers of these predictions need to know the expected accuracy of the forecasts and the confidence with which the predictions and putative detections are made. Examples include stock analysts, weather forecasters, physicians, and, of course, intelligence analysts.
Many individuals and organizations make predictions of future events or detection and identification of current states. Use of Signal Detection Theory as a Tool for Enhancing Performance and Evaluating Tradecraft in Intelligence Analysis
0 kommentar(er)
