5. Skill at arguing is harmful

In the book chapter “Cognitive biases potentially affecting judgment of global risks,” Eliezer Yudkowsky makes the point that a skillful arguer may be superior at arguing any arbitrary point, making it more likely for him to anchor on a mistaken hypothesis12. In the study of cognitive biases, this is known as anchoring. The better knowledge of cognitive biases an arguer has, the more readily he can attribute biases to his opponent, while being blind to his own biases. In this way, a smarter thinker might actually use his intelligence against himself, making it more difficult to construct an accurate model of risks. He may also fall prey to framing effects—that is, certain ways of thinking about things—and use his intelligence to reinforce that frame, rather than considering alternate framings13. For instance, someone preoccupied with the idea that advanced Artificial Intelligence would do X may never be argued out of that hypothesis and consider that AI might actually do Y instead.14