27. An expert’s estimates which are not based on strict calculations
cannot serve as a measure of real probability

Unlike in the stock markets, where the average estimate of the best experts is used as a forecast of market behavior, we cannot select our experts and average them based on their track record of predicting human extinction, because there is no track record of such an event. If it had happened, we would all be dead, and quite incapable of predicting anything.

Slovic, Fischhoff, and Lichtenstein (1982, 472)²¹, as cited in Yudkowsky (2008)²² observed:

A particularly pernicious aspect of heuristics is that people typically have great confidence in judgments based upon them. In another followup to the study on causes of death, people were asked to indicate the odds that they were correct in choosing the more frequent of two lethal events (Fischoff, Slovic, and Lichtenstein, 1977) In Experiment 1, subjects were reasonably well calibrated when they gave odds of 1:1, 1.5:1, 2:1, and 3:1. That is, their percentage of correct answers was close to the appropriate percentage correct, given those odds. However, as odds increased from 3:1 to 100:1, there was little or no increase in accuracy. Only 73% of the answers assigned odds of 100:1 were correct (instead of 99.1%). Accuracy “jumped” to 81% at 1000:1 and to 87% at 10,000:1. For answers assigned odds of 1,000,000:1 or greater, accuracy was 90%; the appropriate degree of confidence would have been odds of 9:1. . . . In summary, subjects were frequently wrong at even the highest odds levels. Moreover, they gave many extreme odds responses. More than half of their judgments were greater than 50:1. Almost one-fourth were greater than 100:1. 30% of the respondents in Experiment 1 gave odds greater than 50:1 to the incorrect assertion that homicides are more frequent than suicides.

The point of this quote is to illustrate that experts are consistently overconfident, often ridiculously so. From Parkin's Management Decisions for Engineers²³:

Generally, people have a displaced confidence in their judgment. When asked general knowledge or probability questions, experimental subjects performed worse than they thought they had (Slovic et al., 1982). Calibration experiments that test the match between confidence and accuracy of judgment, demonstrate that those without training and feedback perform badly. Lichtenstein et al. (1982) found that from 15,000 judgments, when subjects were 98% sure that an interval contained the right answer they were wrong 32% of the time. Even experts are prone to some overconfidence. Hynes and Vanmarke (1976) asked seven geotechnical gurus to estimate the height of a trial embankment (and their 50% confidence limits), that would cause a slip fracture in the clay bed. Two overestimated the height and five underestimated. None of them got it within their 50% confidence limits. The point estimates were not grossly wrong but all the experts underestimated the potential for error.

Simply put, “experts” are often wrong. Sometimes their performance is equal to random chance, or to that of a person pulled off the street. Statistical prediction rules often outperform experts²⁴. This creates trouble for us when we rely on experts to evaluate the probability and nature of catastrophic global risks.