Essay 1
When Your Gut Is Right (and When It Isn't)
What Kahneman and Klein figured out together.
A fire lieutenant leads his crew into a burning house. They start fighting the fire in the kitchen. After about a minute he feels something he later describes as the floor being too hot. He doesn’t know why. He shouts at his crew to get out. Seconds after they do, the floor collapses under them. The fire had been in the basement the whole time. When the lieutenant was interviewed about the case years later, he couldn’t fully explain what tipped him off.
In the months leading up to the autumn of 2008, confident financial analysts were on television explaining why the U.S. housing market was healthy and a real downturn was unlikely. Most of them had decades of experience. By the end of the year, the global financial system was effectively in crisis.
Both groups were professionals making predictions in real time. Both were drawing on years of pattern-recognition built up through experience. One was right when it mattered most. The other, collectively, was famously wrong.
What’s the difference?
The popular answers are unsatisfying. Trust your gut sounds wise until you remember the analysts. Don’t trust your gut, think carefully sounds wise until you remember the lieutenant. There was no time to think carefully, and any deliberation would have killed him. Both pieces of advice are sometimes true and sometimes catastrophically wrong, which means neither is really advice at all.
There’s a better answer, and it has been sitting in the academic literature for almost twenty years. Two psychologists worked it out, almost reluctantly, from opposite directions. The story of how they reached the same conclusion, and what that conclusion actually is, is the foundation of everything that follows.
The first of the two was a man named Daniel Kahneman. If you’ve read one book about how the human mind makes mistakes, it was probably his: Thinking, Fast and Slow. Kahneman spent his career, mostly with his long-time collaborator Amos Tversky, documenting all the ways human judgment goes wrong under uncertainty. They were thorough. They identified dozens of systematic errors, mostly through clean laboratory experiments in which university students estimated, predicted, compared, and gambled. The results were stark. People consistently misread probabilities, overestimated their own knowledge, were swayed by irrelevant numbers, remembered the past as more predictable than it had been, and trusted vivid mental images over accurate ones. In 2002 Kahneman won the Nobel Prize in Economics for this work; by then Tversky had been dead for six years, and the prize is not awarded posthumously. The message from this whole body of research looked clear. Intuition was a parade of errors waiting to happen. The careful thing was to slow down, to bring in the deliberate, methodical part of the mind and override the snap judgments.
The second researcher was named Gary Klein. He spent his career, by contrast, studying experts in high-pressure professions: firefighters, ICU nurses, military commanders, chess masters. He watched them work. He interviewed them after critical decisions, asking how they had known what to do. The lieutenant in the opening of this essay is one of the cases Klein documented. The pattern he found was the opposite of Kahneman’s. The experts almost never deliberated. They didn’t weigh options, run probabilities, or override their first impressions. They looked at a situation, recognized it as belonging to some category they’d seen many times before, and acted. When they were wrong, they updated quickly. When they were right, which was often, it looked like magic to anyone watching. Klein called this pattern recognition-primed decision-making. By his account, intuition wasn’t a parade of errors. It was the engine of expertise.
So you had two well-respected researchers, with two large bodies of evidence, reaching two opposite conclusions. Kahneman: intuition fails systematically; slow down, double-check, run the numbers. Klein: intuition works beautifully in expert hands; don’t second-guess it, don’t fight it.
For a long time, the field treated this as a kind of standoff. People in the laboratory tradition kept producing studies showing how unreliable gut feelings are. People in the field tradition kept producing studies showing how powerful expert judgment is. Both were right. Neither would budge.
And then in 2009, Kahneman and Klein sat down to write a paper together.
The remarkable thing about the 2009 paper isn’t that two opposed researchers met and compromised. They didn’t compromise. The paper isn’t a negotiated settlement where each side gives a little ground. What happened was more interesting, and in academic terms more honest: they looked at each other’s data and agreed all of it was correct.
This is the move I want you to keep in your back pocket as you read on, because I’ll do it more than once. When two careful people reach opposite conclusions, the most useful question is rarely who’s right? It’s under what conditions is each of them right?
For Kahneman and Klein, the answer turned out to be unsurprising once they saw it. Intuition is reliable in some environments and unreliable in others, and the difference is precise enough to predict.
The reliable kind has two features. First, the environment has to contain real patterns: situations that repeat, with relationships between causes and outcomes that are stable enough to be learned. Second, the person needs to have received enough quick, clear feedback on their judgments to actually have learned those patterns. Chess satisfies both. Positions recur, every game ends decisively, and you find out fast whether your move was good. So does firefighting. So does diagnosing certain common medical conditions. So does landing an airplane. In environments like these, intuition develops into something the rest of us think of as wisdom: fast, often unconscious, hard to articulate, and reliable when it matters.
The unreliable kind is everything else. Forecasting political events two years out. Picking stocks. Evaluating a candidate you’ll only really get to know after they’re hired. Predicting how a child will turn out. The pattern in these environments is the same: either the situation never repeats in the same form, or the feedback comes so late and so muddled that it can’t actually teach you anything. You can do these things for decades and still not know whether your judgments are well-calibrated. Many people who have, aren’t. They’re just confident.
A researcher named Robin Hogarth, working independently around the same time, gave these two kinds of environments names: kind and wicked. From here on I’ll call them something slightly friendlier and more visual: clear environments, where the world tells you when you’re wrong soon enough to learn from it, and obscured environments, where it doesn’t.
Diagnosing the environment you’re in
Why does any of this matter outside an academic paper?
Because the test of whether your gut is trustworthy isn’t how strong the feeling is. That’s the trick most popular bias writing gets wrong. The test is what kind of environment your gut was trained in. And almost every situation you face in adult life is a mix of clear and obscured stitched together.
Think about a doctor seeing a patient with familiar symptoms. Clear environment, intuition probably solid. The same doctor speculating about a new policy’s long-term effect on public health is in an obscured environment; even decades of expertise won’t have calibrated their gut on something that has never happened before. Same person. Same brain. Two completely different reliability profiles for what feels, from the inside, like the same kind of professional judgment.
Or take a hiring manager who has run hundreds of interviews. They feel certain they can tell good candidates from bad ones in the first five minutes. The interview itself is a clear environment in our sense, with feedback within minutes and lots of repetition. But the thing the manager actually wants to predict, which is whether this person will succeed in the role over the next three years, sits in a deeply obscured one. The feedback comes years later, mixed with the effects of teammates, market conditions, life events, and a hundred other things. By the time the manager finds out whether the hire worked out, they’ve forgotten the impression and rationalized whatever happened. Their gut, despite hundreds of interviews, was never given the data it would need to be reliable on the part that matters. It is confidently miscalibrated, and the confidence is the dangerous part.
This is the pattern I’ll keep coming back to. People don’t have bad guts or good guts. They have guts trained on certain environments. And the reliability of any specific intuition is roughly the reliability of the environment that trained it.
So how do you tell which one you’re in? Three honest questions:
How fast does the feedback come, and how clear is it when it does? A chess move is decided in minutes. A hiring decision is judged in years. A diet decision is judged in months but blurred by everything else you’re doing. Faster, clearer feedback is the strongest single sign that the environment is clear.
Does the situation repeat in the same form? A weekly meeting with the same five people contains real patterns that build expertise. A once-a-decade career decision doesn’t repeat in any meaningful sense; whatever pattern-matching you do is from secondary data (other people’s careers, advice, your own imagination), not from your own track record.
How tangled are the confounders? If your decision’s outcome is also influenced by economic conditions, other people’s choices, random events, or long causal chains, your feedback is heavily contaminated. Even when the outcome eventually comes, you can’t reliably attribute it to your decision. The longer the chain, the more obscured the environment.
Below is a small tool that walks you through this. Pick a situation from your own life, or use one of the preset examples, and see how the diagnosis comes out:
When gut and situation don’t match
The interesting cases — the ones worth writing about — are the cases where you have a strong gut feeling about a situation that turns out to be obscured. The strength of the feeling is doing none of the work it pretends to do. The environment never gave your gut the feedback it would have needed to be reliable. The certainty is decorative.
There are two specific moves to make when this happens, and neither requires you to “be less emotional” or “think harder.” That’s the kind of advice that sounds like advice but doesn’t actually mean anything.
The first move is to demote the gut feeling to a hypothesis. Instead of “I think this candidate is the one,” it becomes “my initial impression is that this candidate is the one. What would I expect to see if that’s true, and what would I expect to see if it isn’t?” The reframing converts a verdict into a question, which is something you can actually investigate. This sounds trivial. It isn’t. Most people, presented with a strong intuition, spend the rest of their effort gathering reasons it might be right rather than reasons it might be wrong. The reframing is the smallest thing you can do that systematically tilts the cognitive economy away from confirming yourself.
The second move is to import the outside view. Even when your own gut is poorly calibrated on a specific situation, there’s usually a wider set of comparable cases you can borrow from. For the hiring manager: instead of “does this candidate feel right to me?”, try “what fraction of candidates who interviewed like this one have actually succeeded in the role?” The phrasing isn’t elegant, but the redirection is the point. You stop asking your gut for a verdict about this case and start asking the historical base rate for a verdict about cases like this. The outside view will be uncomfortable. It is supposed to be. If your inside view feels much rosier than the outside view, the inside view is usually wrong.
Neither move is a magic trick. Both are also far less marketable than the popular alternatives like trust the universe, channel your inner CEO, follow your authentic self. But neither is asking you to suppress the gut feeling or pretend it didn’t happen. The gut feeling stays in the room; it just isn’t allowed to vote without showing its homework.
Where this framework runs out
The clear-versus-obscured distinction is the most useful single idea I’ll give you in these essays, but it has limits worth being honest about.
The first limit is that the diagnosis itself can be wrong. People who have spent decades in an obscured environment often feel like they’re in a clear one. A pundit who has been confidently wrong for thirty years tends to remember the times they were right, forget the times they weren’t, and feel like the environment has been teaching them all along. The environment hasn’t been teaching them. They have been teaching themselves a flattering story. Diagnosing your own environment requires an honesty that most of us have to work at, and we’ll come back to that honesty in a later essay on calibration.
The second limit is that expertise transfers unevenly. A surgeon who has done 5,000 operations of a specific kind is in a clear environment for that operation. But if you ask them about cancer screening policy, you’re asking them to make predictions in an environment where they’ve gotten no specific feedback at all. Same person. Different reliability. The internal feeling of expertise is global; the actual expertise is local.
The third limit is that most decisions are mixed. Buying a house is a single decision, but it’s stacked out of several different sub-decisions, each in a different environment. Negotiating a price is a clear environment if you’ve done it a few times. Predicting whether you’ll be happy living in the neighborhood for ten years is obscured. Estimating renovation costs is obscured in a specific way called the planning fallacy, which we’ll meet again. The skill is in pulling these apart, applying intuition where it earned its keep, and imposing structure where it didn’t.
The fourth limit, and the one that takes longest to internalize, is that knowing all of this doesn’t automatically make you better at it. There’s a phrase that comes up a lot in the bias literature — knowing is half the battle — and one of the most interesting findings of the last fifty years is that it isn’t. Not even close. Knowing about a bias and not having that bias are very different states. We’ll come back to this directly when we talk about what actually helps you think better.
Everything that follows builds on this. The next essay zooms out from intuition specifically to the brain’s broader habits: the long catalogue of biases that everyone has heard of, organized into a shape that makes them legible instead of overwhelming. After that I want to get to the most useful and most honest question in this whole field, which is what actually helps you think better in practice. Then a look at the everyday forecasting all of us are doing all the time. Then how all of this is now scaling, through our own habits, into the machines we’ve built.
The conditions, as I said at the start, are unsurprising once you see them. The hard part is what to do about them.