All I Know Is That I Know Nothing
A reflection after reading Leonard Mlodinow's "The Drunkard's Walk"
Socrates is credited as saying that, “all I know is that I know nothing” and, after reading Leonard Mlodinow’s The Drunkard’s Walk, I feel like I should probably have that tattooed on my arm as a constant reminder. Mlodinow’s thesis was certainly not to make readers feel like the world is unknowable so my assertion needs a great deal of context.
The author, a Caltech physicist, strives to give readers access to several of the statistical concepts that underpin modern people’s lives. Along the way, he also dispels many illusions about how well people think they understand the world around them. It isn’t that the world is too complex, nor is it that the world is completely random. The problem is that the complexity and the randomness of the world come together in ways that are frequently beyond understanding. So humans create illusions in order to feel they understand and feel in control. As Mlodinow puts it,
Perception requires imagination because the data people encounter in their lives are never complete and always equivocal. (pp. 170-171)
That’s why I want to remind myself that “all I know is that I know nothing”. I want to keep myself honest about what I do know and what I don’t know. I want to feel okay with not knowing. I want to keep myself honest about what I do control and what I don’t control. I want to feel okay with not being in control.
Mlodinow makes a convincing argument when he writes about stock market investors and prognosticators (chapter 9). He highlights that the problem isn’t with how they do their jobs, but how they interpret the work they do. More importantly, he highlights how they overlook what they don’t do. They miss stuff because they want to believe it is possible to strategize your way into large returns on your investments. But…
Research has shown that the illusion of control over chance events is enhanced in financial, sports, and, especially business situations when the outcome of a chance task is preceded by a period of strategizing, when performance of the task requires active involvement, or when competition is present. (p. 188)
Why does the illusion of control take hold? Because people want to feel agency. That’s why people tend to fall victim to the Texas sharpshooter fallacy, because they’re good at seeing order and agency where there isn’t any. But the investors aren’t the only ones to believe in the illusion of control. The people who study those investors believe it too. That belief shows in the questions asked about them.
Imagine you’re comparing the results of different investors and you notice one whose results stand out. You could rightly assume those results are improbable, which they are. The question you ask is “what is the probability that this exact investor has this much success in this exact time frame?” That question makes the improbability seem to be a product of something about the investor. That’s the illusion of control.
Why is it an illusion? Because you have unwittingly ignored a great deal of randomness to anoint this one investor as a genius. You’ve noticed a cluster of holes on the side of the barn and drawn a target around them. You were so busy making the investor a Texas sharpshooter that you failed to notice all the other holes in the side of the barn. The question that exposes the randomness is “how likely is it that any investor has that much success in any period of similar length?” The answer to that question is far less improbable. It suggests there isn’t necessarily anything special about the one investor whose results stood out.
Those results could have happened randomly. But maybe they didn’t. Maybe the investor is special, maybe they’re not. The problem is there may not be any evidence one way or the other but the illusion of control has you believing in their “skill” without even searching for any evidence. People assume causality when rare events happen. As far as they’re concerned, the proof is in the rarity rather than in any evidence. But the truth is you don’t know.
But there are tools to help you figure out if success was made or just happened. There are tools to help you separate the ordinary from the out of the ordinary. In research, the common phrase to describe out of the ordinary is “statistically significant”. That term has definitely escaped the field of statistics but, unfortunately, the understanding of the term did not escape with it. What coaches think when they see “statistically significant” is not just “out of the ordinary” but also “special”. And coaches think that special means “nonrandom”. But, Mlodinow writes, that’s not what statisticians think.
When I calculate that something is statistically significant, I have not determined that the outcome was nonrandom. I have only calculated that the outcome is very unlikely to happen. But those are two different things. The problem is people tend to infer from my calculation that the outcome was nonrandom rather than just rare. People assume that rare things don’t just happen. “Because if events are random, we are not in control, and if we are in control of events, they are not random” (p. 186). This, to me, is an expression of the illusion of control. There has to be a reason when something rare happens. That investor? They had to have done something to create those rare returns.
The graph above is a good representation of what statistical significance looks like. But it’s also a good representation of what normal distribution looks like. Events are considered to be statistically significant when they are far enough away from the tallest part of the distribution curve. But the thing about normal distributions is that there are always events out in the tails of the curve. When p values like 0.05 are thrown around, remember they just mean something happens about 1 in 20 times. Mlodinow reminds you that, even if you choose a stricter standard, like 0.03, that, “if you test 100 nonpsychic people for psychic abilities…you ought to expect a few people to show up as psychic…” (pp. 172-173). Are those few actually psychic? Probably not, but the point is that just finding statistical significance doesn’t mean you’ve found anything more than that.
Let me give a different example. I have a solitaire game I like to play on both my phone and tablet. At one point in time, on the tablet, I had won 46% of the games I had played while I had won 50% of the games I had played on my phone. It’s the same game and the same player so shouldn’t the percentages be about the same? Only if I have more control over the outcomes than I really do. Maybe I screwed up some chances to win but it is much more likely that I just didn’t get as many deals that were winnable and that’s random. If I reset the stats on both devices and started over, it is likely that the percentages would still be different but it’s hard to say just how different they would be. The higher winning percentage on one device isn’t due to anything I did. It just happened on its own.
I think Mlodinow’s lesson about randomness is important because it frees you from having to find reasons for everything that happens. To quote from a different author, Cassie Kozyrkov, “Analytics has only one golden rule: stick to the data and don’t go beyond it.” Let the data be just the data. Let it describe and don’t force it to also explain. In my job as a performance analyst, I would often find myself thinking I knew more than I did, just because I had the data. I spoke as though the data was itself an explanation. As I sometimes put it, “I’m right because I have a pile of numbers.” But data doesn’t always give you the reasons, so don’t assume they do.
You could argue that my job was to draw conclusions from data without having to be concerned about if those conclusions were causally justified. Before I read Mlodinow’s book, I agreed with that assessment of my job. Before, I assumed statistics gave me the justification. After, I saw that statistics, if anything, should steer me away from feeling justified. It’s not that a performance analyst can’t or shouldn’t do the job, it’s that they need to include the appropriate caveats when they do. It’s important to provide context about how rare or random an outcome may be. It’s important to can give nuance to the balance between what is chaotic and what is controlled. What Mlodinow taught me is that things are more chaotic than I care to admit.
Researchers concluded that ‘people have a very poor conception of randomness, they do not recognize it when they see it and they cannot produce it when they try,’ and what’s worse, we routinely misjudge the role of chance in our lives and make decisions that are demonstrably misaligned with our own best interests. (p. 174)
Like I said, all I know is that I know nothing.