Beyond Intuition with Bayes Theorem

Intuition Fails Us

In his book, Thinking Fast and Slow, the late, great Daniel Kahneman proposes the following thought experiment (Kahneman 2012).

An individual has been described by a neighbor as follows: “Steve is very shy and withdrawn, invariably helpful but with little interest in people or in the world of reality. A meek and tidy soul, he has a need for order and structure, and a passion for detail.” Is Steve more likely to be a librarian or a farmer?

When I first encountered this, my intuition led me believe that Steve is most likely a librarian. Why? Well it’s all in the description. Surely Steve’s neighbor provided an accurate description. Steve is shy, withdrawn, meek, but likes order. Steve must a librarian, right?

The point of this thought experiment is to examine how we form a given judgment that irrationally contradicts the laws of probability.

Let’s take a step back. We don’t know anything about Steve. What is the likelihood that a random person is either a librarian or a farmer? That’s where Bayes Theorem comes into play.

Bayes Theorem

Bayes Theorem offers a (friendly) mathematical framework to update our beliefs based on new evidence. It calculates the probability of an event based on prior knowledge of conditions that might be related to the event. In mathematical terms, it expresses how a belief should rationally change to account for new evidence.

Applying this to our thought experiment, we consider the prior probability of any random individual being a farmer or a librarian, weighted against the likelihood of Steve fitting the description given his profession. In the U.S., farmers vastly outnumber librarians, which significantly shifts the initial odds in favor of Steve being a farmer, despite the stereotypes suggested by his description.

By using Bayes Theorem, we can quantitatively adjust our assumptions: we factor in how many people are actually farmers versus librarians and apply the description as a condition influencing these roles. This approach moves us from a baseless assumption to a reasoned estimation, illustrating how Bayes can be pivotal in making informed decisions in scenarios with incomplete information.

Thus, Bayes Theorem not only corrects our intuitive biases but also enhances decision-making in areas as critical as early stage innovation, where the landscape is often ambiguous and data-driven insights are crucial.

In the future, I’ll publish another blog about Bayes and it’s application to early-stage innovations.

For those that want to dive deeper, I highly recommend this fantastic summary from 3Blue1Brown:

References