Instead of relying on opinions, intuition, or whoever speaks the loudest in the room, we look at customer behavior, conversion, revenue, retention, and hundreds of other signals before making a decision.
And in principle, that is a good thing. Data can help us challenge assumptions, understand what users are actually doing, and make decisions based on evidence rather than instinct.
But being data-driven does not automatically mean being objective.
Sometimes the data is completely real, the dashboard is working, the SQL query runs without errors, and we still arrive at the wrong conclusion.
The hidden trap is not always bad data. Sometimes it is how we choose, define, calculate, and interpret that data.
01We choose what to measure
Most companies do not suffer from having too little data. They often have the opposite problem: hundreds, sometimes thousands, of metrics.
Revenue. Conversion. Retention. Click-through rate. Daily active users. Average order value. Session duration. Customer acquisition cost.
But which one should we look at? The moment we choose one metric over another, we are already making a subjective decision.
Imagine a product team launches a new feature. Engagement increases by 20%. That sounds like a success.
But what if retention stays flat? What if customers use the feature more because it adds unnecessary steps? What if engagement increases while conversion decreases?
The metric itself may be correct. The problem is that we selected one number and treated it as the full story.
Before asking, “What does the data say?”, perhaps we should first ask:
Why did we choose this particular metric to represent success?
02A metric name can hide a complicated definition
Some metrics sound so familiar that we rarely question them.
Take “active users.” What is an active user? Someone who logs in? Someone who opens the app? Someone who performs a specific action? Someone who spends more than a certain amount of time using the product?
Different teams can use the same metric name while measuring completely different behavior. The same applies to revenue, conversion, churn, retention, or even customer.
A dashboard might simply display:
Conversion Rate: 4.8%
But behind that number are many decisions.
What counts as a conversion? What is the denominator? Are cancelled transactions included? Are returning customers treated differently? What time period is being used? Which timezone? Are test accounts excluded?
A metric can be technically correct and still be misunderstood. For important metrics, understanding the definition is just as important as reading the value.
03A query can run perfectly and still be wrong
A SQL query can execute successfully and still produce the wrong result.
- A join can duplicate records.
- A filter can unintentionally exclude a customer group.
- NULL values can behave differently from what we expected.
- A date condition can shift transactions into the wrong period.
- A timezone issue can make yesterday's activity appear today.
The number may still look reasonable. That is exactly what makes these mistakes difficult to detect.
| order_id | order_total |
|---|---|
| 101 | $150 |
| 102 | $200 |
| 103 | $100 |
| order_id | items |
|---|---|
| 101 | A, B |
| 102 | C, D, E |
| 103 | F, G |
| order_id | item | order_total |
|---|---|---|
| 101 | A | $150 |
| 101 | B | $150 |
| 102 | C | $200 |
| 102 | D | $200 |
| 102 | E | $200 |
| 103 | F | $100 |
| 103 | G | $100 |
| SUM(order_total) | $1,100 | |
An obvious error is usually easy to catch.
A plausible but incorrect number is much more dangerous.
This becomes even more relevant as AI tools help analysts generate SQL. AI can make us much faster: it can help us write queries, explore unfamiliar databases, or build prototypes in minutes.
But speed does not remove the need for validation. A query generated in seconds can still contain assumptions that do not match the business logic.
The question should not only be “Does this query run?” It should also be:
Does this query actually answer the question I am asking?
04Numbers without context can be misleading
Imagine someone tells you:
“Conversion dropped by 15%.”
Should you be worried? Maybe. But we still do not know enough.
Compared with yesterday? Last month? The same period last year? Is the drop bigger than the normal week-to-week movement, or is it just noise?
Did the traffic source change? Was there a promotion last week? Did mobile traffic increase? Did a large marketing campaign bring in lower-intent visitors? Did the definition of conversion change?
A number in isolation rarely explains what happened. Trends, segments, historical comparisons, seasonality, customer mix, and operational changes can completely change the interpretation.
05Sometimes we use data to confirm what we already believe
This may be the hardest trap to recognize, because the problem is no longer inside the database. It is inside our interpretation.
Imagine you believe a new feature is performing well.
You open the dashboard. Engagement is up 20%. Great. Hypothesis confirmed.
But retention is slightly down. Support tickets increased. And one customer segment is using the feature much less than before.
Which number gets mentioned in the meeting? Probably the +20%.
We naturally pay more attention to evidence that supports what we already believe. This is confirmation bias, and having more data does not automatically protect us from it. In some cases, having more metrics simply gives us more opportunities to find one that supports our preferred story.
A useful question is:
What data would make me change my mind?
That question forces us to look for evidence against our hypothesis, not just evidence supporting it.
06The dashboard is the beginning, not the conclusion
Dashboards help us monitor performance, detect unusual changes, and quickly understand what is happening across a business.
But a dashboard usually answers what happened? It does not always answer why did it happen?
That second question often requires deeper investigation: segmenting customers, checking operational changes, reviewing the underlying data, talking to users, looking at qualitative feedback, running experiments, and understanding the business context.
The purpose of a dashboard should not be to eliminate questions. A good dashboard often helps us identify better questions.
A simple way to challenge important numbers
For metrics that influence important decisions, I find it useful to think about five layers:
- SelectionWhy this metric, and not another one?
- DefinitionWhat exactly are we measuring?
- CalculationHow was the number produced?
- ContextWhat else could explain this change?
- InterpretationWhat story are we telling based on the number, and what alternative explanation might exist?
- And one more question: what is this metric not telling us?
Every metric is a simplified representation of reality. No single number captures everything about customers, products, or businesses.
Being data-driven should mean being curious, not just confident
The goal is not to trust data less. It is to understand it better.
Data is valuable precisely because it helps us challenge assumptions. But that only works if we are also willing to challenge the assumptions behind the data itself:
- The metric we selected.
- The definition behind it.
- The query that calculated it.
- The context surrounding it.
- And the story we created after seeing it.
Being truly data-driven is not simply saying “The numbers say this.” Sometimes it means asking:
How much should we trust this number, and what else could it be telling us?
Data does not have to be false for us to reach the wrong conclusion.
Sometimes the mistake happens in the question, the measurement, or the story we build around the number.
References & further reading
Where the ideas in this post come from. It's a working synthesis, not original research, and these are the sources worth going to directly.
See moreShow less13 sources — books and papers
Books
- [1]
Muller, J. Z. (2018). The tyranny of metrics. Princeton University Press.
How the choice of what to measure shapes what an organisation pays attention to (section 1).
- [2]
Kimball, R., & Ross, M. (2013). The data warehouse toolkit: The definitive guide to dimensional modeling (3rd ed.). Wiley.
Shared metric definitions and declaring the grain of a table, the discipline that prevents the double counting in section 3.
- [3]
Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press.
Twyman's law, “any figure that looks interesting or different is usually wrong” (section 3), and guardrail metrics (section 1).
- [4]
Wheeler, D. J. (2000). Understanding variation: The key to managing chaos (2nd ed.). SPC Press.
How to tell a real change from routine variation; the shaded band in section 4.
- [5]
Pearl, J., & Mackenzie, D. (2018). The book of why: The new science of cause and effect. Basic Books.
From “what happened” to “why it happened” (section 6), with a chapter on Simpson's paradox.
- [6]
Kahneman, D. (2011). Thinking, fast and slow. Farrar, Straus and Giroux.
“What you see is all there is”: we build a confident story from whatever evidence is in front of us.
- [7]
Harford, T. (2021). The data detective: Ten easy rules to make sense of statistics. Riverhead Books.
A readable companion to the whole post.
- [8]
Spiegelhalter, D. (2019). The art of statistics: How to learn from data. Basic Books.
Plain-language grounding in how data turns into claims.
Papers
- [9]
Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society: Series B, 13(2), 238–241.
The original description of the paradox in section 4.
- [10]
Bickel, P. J., Hammel, E. A., & O'Connell, J. W. (1975). Sex bias in graduate admissions: Data from Berkeley. Science, 187(4175), 398–404.
A real case where the overall numbers pointed the wrong way.
https://doi.org/10.1126/science.187.4175.398 - [11]
Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220.
The standard review behind section 5.
https://doi.org/10.1037/1089-2680.2.2.175 - [12]
Goodhart, C. A. E. (1975). Problems of monetary management: The U.K. experience. In Papers in monetary economics (Vol. 1). Reserve Bank of Australia.
Origin of Goodhart's law: a measure used for control stops behaving the way it used to.
- [13]
Strathern, M. (1997). ‘Improving ratings’: Audit in the British University system. European Review, 5(3), 305–321.
Source of the popular phrasing: “When a measure becomes a target, it ceases to be a good measure.”
Accessed September 2026. Figure values are illustrative, not from real data. Where a claim matters to a decision, go to the source rather than citing this post.