A fool thinks himself to be wise, but a wise man knows himself to be a fool. – William Shakespeare
It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so. – Mark Twain
As a team whose principals have spent thousands of hours accumulating academic credentials and experience in a fairly narrow field of expertise – public markets – we are endlessly fascinated with peoples’ optimism about their ability to succeed independently in this hyper-competitive domain. After all, most people wouldn’t feel confident performing an appendectomy on one of their loved ones after watching a season of Grey’s Anatomy. In fact, most people wouldn’t consider representing themselves in court, or creating their own family trust, or executing an estate freeze. Heck, most professionals don’t even file their own taxes.
Yet people from all walks of life think they can simply open up an eTrade account and compete successfully in markets against the top investment firms in the world. Somehow, subscribing to a few newsletters; soliciting tips from friends or taxi drivers; learning to draw trend-lines or interpret a MACD cross; or a few inconsistent hours a week or month reading the Wall Street Journal, the Economist, or watching CNBC, equips them to out-trade GETCO, Renaissance, and Goldman’s proprietary trading desk, with the best information, fastest computers, and most aggressive algorithms. What gives?
Then I discovered the Dunning-Kruger effect (DK). At root, the DK effect explains the observation – ubiquitous across domains of expertise – whereby people who are less competent lack the requisite knowledge about the depth and breadth of the domain in question to recognize their own relative incompetence. Laypersons in any field simply don’t know enough about the complexities of that field to comprehend the magnitude of their knowledge gap.
Dunning and Kruger – both psychologists at Cornell – have identified this effect in a wide variety of fields through direct observation and experimentation, and Wikipedia reminds us of myriad similar observations by wise persons through history. For example, Confucius stated, “Real knowledge is to know the extent of one’s ignorance.” In 1951 Bertrand Russell asserted, “One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision.” Charles Darwin also weighed in on the topic with “ignorance more frequently begets confidence than does knowledge,” and even Shakespeare expressed similar sentiment in As You Like It: “The Foole doth thinke he is wise, but the wiseman knowes himselfe to be a Foole.”
I would refer you to Figure 1., which attempts to illustrate the theoretical basis for the Dunning Kruger effect. We aren’t the first to propose this illustration, but I couldn’t easily find the progenitor. At any rate, the figure captures the relative knowledge gap between a novice and an expert in a complex field in the relative size of the novice’s yellow circle and the expert’s green circle within a much larger circle meant to represent the entire set of existing knowledge within that domain. The novice’s smaller yellow circle clearly demonstrates he possesses a fraction of the expert’s knowledge about this broad field.
Figure 1. Who is more aware of what he doesn’t know? The expert or the novice?
In their papers, Dunning and Kruger suggest that novices in a field are not equipped with the requisite knowledge to evaluate their own relative or absolute level of expertise. How well equipped is a fish to evaluate a cormorant’s knowledge of the sky? How fit is a childless friend to offer guidance on parenthood?
Extending this logic, Figure 1. prompts the question, “Who is more aware of what he doesn’t know? The expert or the novice?” This is really the question that drives to the heart of the Dunning Kruger effect, because a person’s ability to evaluate his level of expertise in a field is a function of his awareness about the ratio of his personal knowledge in the domain to the amount of total available knowledge.
If we examine Figure 1. we note that, while the absolute knowledge differential between an expert and a novice is captured in the relative sizes of the yellow and green circles, awareness of this differential is actually a function of the size of the interface between the circles. That is, a person can only be aware of his ignorance to the extent that his knowledge interfaces with the set of available knowledge. These are the terms and concepts that the novice has casually encountered while dealing in this domain, but about which he has no meaningful understanding. In the diagram, the perimeter of the yellow circle describes the novice’s casual exposure to elements of knowledge possessed by the expert. The perimeter of the expert’s circle is obviously larger than the perimeter of the novice’s circle, so the expert’s knowledge interfaces to a larger extent with ‘the unknown’ portion of entire field of knowledge.
What does this mean? First, an expert is relatively well equipped to evaluate his own expertise relative to a novice, but a novice is not well equipped to evaluate his level of ignorance relative to the expert. Second, a genuine expert is more aware of what he doesn’t know relative to the entire field of knowledge than the novice. While false experts are given to frequent opinions asserted with steadfast conviction, real experts are nuanced and circumspect.
Most people are uncomfortable with nuance; it’s difficult to convey in 140 characters, and is often less dramatic. For this reason, the media tends to shy away from genuine experts and gravitate toward false experts who give dramatic opinions with tremendous conviction. Great entertainment, but bad for your pocketbook.