Correlation Does Not Imply Causation
Correlation Does Not Imply Causation
Don't be fooled by good-looking charts.
Sometimes two similar lines on a chart connected by a story can make us think irrationally and believe in something false.
Take this example: ice cream sales are correlated with people getting sunburned. So, ice cream should be banned to decrease sunburns and the risk of skin cancer? Well, no.
We deal with data in the form of charts when viewing server metrics, stock prices, or temperature graphs. But, to make good decisions and use data effectively, we must understand what is happening behind the data, often hidden from plain sight.
Both ice cream sales and sunburn are correlated, but it is not because ice cream people get burned. There is a third invisible thing, temperature — with higher temperatures people buy more ice creams, and go outside without much cover which leads to more cases of sunburn.
Correlation
Correlation is a measure of the relationship between two variables. The better we can predict changes in the second variable by looking at the first one, the stronger the correlation is.
Working as an engineer manager I heard dozens of requests from clients to add more developers to move faster. They had seen a correlation between hours spent working and features produced. Therefore, they assumed that twice as many developers must deliver the project twice as fast. Software engineering isn't like this:
“No matter how great the talent or efforts, some things just take time. You can’t produce a baby in one month by getting nine women pregnant.”
— Warren Buffet
It comes with the simple assumption that if one developer can write x lines of code, then n developers can write n * x lines of code; which simply does not work for software engineers.
It is important to understand that correlation is just a statistical measure of two variables, nothing else. Two variables can be correlated without one causing the other.
Spurious Correlations
Spurious correlation occurs when two variables appear to be statistically related but have no real-world connection. The relationship is purely coincidental.
Take a look at these charts (all charts are made by Tyler Vigen, author of the database of spurious correlations):
I look at these unrelated things and feel a sense of pleasure, even though I’m sure it's all random. Searching for meaning where there is no meaning is a human trait, it may influence our perception of reality.
No matter how beautiful the graph looks, it does not mean it represents causation.
Causation
Causation is the relationship between two things where one event directly influences the other. To understand the reality we need to understand causation, and while correlation can help us with this, it shouldn't alone act as the defining measurement.
The example of ice cream consumption affecting sunburn rates illustrates how looking beyond data can help us understand reality. When thinking in systems, we treat sunburn, skin cancer, and ice cream sales as part of a larger system.
How do we find the cause? We need to understand how humans behave: when they reach for an ice cream, and when they get sunburned, which leads us to the answer — temperature rise, and sunny weather cause both events.
This is an example of systems thinking, where we aim to connect and understand all the components and interactions within the system: human behaviour, the effects of weather, and the roles of ice cream and sunburn.
Summary
Correlations are valuable as they point out where we can look for solutions. However, we have to understand what is happening behind the data, hidden from plain sight.
Reality is far too complex to put in a simple chart and assume that's all there is.
Thanks for reading,
— Michał
P.S. The Database of spurious correlations prepared by Tyler Vigen is an amazing source of correlations that are just random coincidences.
Post Notes
Introduction to Systems Thinking:
Mentoring Sessions
I'm mentoring on adplist.org — I'd love to help you make the most of your career or serve as your decision-making sparring partner.
Discord
With
, , and — we launched a Discord community: Engineering & LeadershipYou can be one of its first members to join us for Q&A sessions, virtual meetups, and events. I’m also planning to run a book club focused on leadership topics.
Discover Weekly — Shoutouts
Articles that might help you explore new perspectives, which I have read recently:
"How to Unlock Critical Thinking?" from
by"When your PM drives you crazy" from
by"How To Go Founder Mode As An Engineering Manager" from
by"Win that Communication" from The Curious Soul's Corner by
Connect
LinkedIn, Substack DM, or respond to this email.
Not knowing this is how superstitions are born.
In the management world, it ends up being the reason for stupid processes and rituals.