Superforecasting

Can we truly get better at predicting the future? Philip Tetlock’s Superforecasting argues that we can—if we treat forecasting not as mystical intuition but as a measurable, improvable skill. He and coauthor Dan Gardner show that individuals and teams can meaningfully improve their accuracy by applying empirical discipline, probabilistic reasoning, and continual feedback. Forecasting, Tetlock insists, is akin to medicine before randomized trials: dominated by confident experts who rarely checked whether their treatments worked. The cure is the same one medicine adopted—a culture of testing, measurement, and learning from data.

Across the book, you meet ordinary people—engineers, filmmakers, retirees—who outperform intelligence analysts and pundits precisely because they think scientifically about uncertainty. These 'superforecasters' represent what Tetlock calls the next frontier of evidence-based judgment. The book traces how tournaments run by IARPA (the U.S. intelligence research agency) revealed differences in skill, what cognitive habits made some forecasters reliably better, and what organizational practices help entire teams and institutions learn from their errors.

From Confident Guessing to Tested Forecasting

Tetlock begins with the analogy to medicine. For centuries, doctors prescribed based on prestige and story, not evidence. Archie Cochrane’s painful near-miss diagnosis—being told he was terminal until pathology proved otherwise—illustrates what unfolds when expertise goes untested. Tetlock argues that forecasting suffers the same disease: pundits fill airwaves with confident predictions, but rarely score or revisit them. The remedy is measurement—turning vague statements into clear probabilistic estimates and evaluating them with proper scoring rules. Once forecasts are defined and tracked, improvement becomes possible.

The Tournaments That Changed Everything

To prove forecasting could be studied systematically, Tetlock and Barbara Mellers built the Good Judgment Project (GJP) and entered IARPA’s multi-year forecasting tournament. Thousands of participants answered real-world geopolitical questions under controlled conditions—same wording, timelines, and scoring metrics. The result was revolutionary: ordinary volunteers consistently beat intelligence analysts with classified information. The secret wasn’t secret data—it was disciplined thinking and collaborative learning. By “extremizing” crowd forecasts and weighting top performers more heavily, GJP achieved the best accuracy among research teams, showing that forecasting can be trained.

What Makes a Superforecaster

Superforecasters share recognizable habits rather than extraordinary IQs. They think probabilistically, revise frequently, and approach beliefs as hypotheses. They are 'foxes'—eclectic thinkers who integrate many small insights—instead of 'hedgehogs' driven by one grand theory. Active open-mindedness, numeracy, and curiosity matter more than credentials. People like Tim Minto, Doug Lorch, and Jay Ulfelder embody this mindset: they update their probabilities incrementally, test their assumptions against evidence, and feel no shame in changing their minds. Their calibration—forecasting 70% events that happen 70% of the time—is the empirical mark of mastery.

Cognitive Biases and Identity Resistance

Forecasting isn’t purely intellectual—it confronts emotional and identity barriers. Tetlock draws on Kahneman’s research on heuristics, showing how System 1 narratives mislead while System 2 disciplined probabilistic reasoning corrects. But beliefs are also structural: his 'Jenga tower' metaphor reveals how deeply anchored convictions resist change. Experts tied to public reputations or ideological tribes have high-cost identity blocks; superforecasters, by contrast, aren’t tied to defending prior statements, making them freer to update.

Learning and Perpetual Beta

Tetlock’s most optimistic lesson is that forecasting skill can be learned, refined, and extended indefinitely—a 'perpetual beta' state. Like software that constantly patches and improves, expert judgment thrives on practice, feedback, and humility. The best forecasters maintain logs, revisit misses, and seek adversarial collaboration that forces precision. Whether you’re a manager or analyst, the takeaway is clear: treat each forecast as a mini experiment. Score it, review errors, and adjust your models.

From Individuals to Institutions

Tetlock expands the lens from individual talent to collective intelligence. Teams built with psychological safety and independence outperform lone experts by wide margins, provided they preserve dissent and avoid hierarchy. Similarly, leaders—whether military generals or CEOs—must balance decisiveness with adaptability, echoing Moltke’s 'mission command': convey intent clearly but empower flexible execution. Institutions should emulate medicine’s transformation—track forecast accuracy, encourage competition, and create accountability mechanisms. Without measurement, organizational forecasting remains rhetoric dressed as wisdom.

Core message

You can meaningfully improve your vision of the future—if you measure, learn, and revise. Forecasting becomes science when you treat every belief as a testable hypothesis and every prediction as data for your next improvement.

In sum, Superforecasting turns the art of prediction into disciplined empirical practice. It reveals that uncertainty can be managed—not eliminated—through calibration, collaboration, active open-mindedness, and continuous learning. Forecasts will never be perfect, but they can be honest, measurable, and useful—and that’s a revolution worth pursuing.

Tetlock’s first rule of good forecasting is deceptively simple: keep score. You must define forecasts clearly, assign probabilities, and then compare what happened with what you predicted. Without measurement, talk about foresight is unfalsifiable. He parallels this principle with medicine's shift toward randomized trials: before testing, doctors followed stories instead of data. In forecasting, governments, pundits, and even intelligence agencies often mistake procedure for progress—checking whether analysts followed protocols rather than whether their forecasts matched reality.

Why Measurement Matters

When forecasts are measured, performance differences suddenly appear. You can separate eloquent storytellers from those who genuinely add predictive insight. Tetlock’s Good Judgment Project established metrics like the Brier score to quantify accuracy and calibration charts to identify systematic biases. You learn whether 70% forecasts really happen about 70% of the time, and whether you meaningfully distinguish 90% from 60% confidence. Measurement thus converts intuition into feedback—an engine of learning.

Practical rule

Every forecast should have a defined event, a timeline, and a numeric probability—and it should be scored. Vague verbs like 'might' or 'could' destroy accountability.

How Scoring Improves Judgment

Brier scores reward well-calibrated precision. They punish both overconfidence and vagueness. Ambiguous language like 'significant' is the enemy of learning. Once you score forecasts against benchmarks (simple baselines, crowd averages, or 'dart-throwing chimp' randomness), forecasters begin improving through trial and feedback—forecast, measure, revise, repeat. That cycle creates calibration and resolution, the twin pillars of accuracy.

Institutional Resistance

Unfortunately, institutional cultures resist measurement. Intelligence analysts can be punished for being 'on the wrong side of maybe'; banks and pundits fear being judged. Tetlock borrows Ernest Codman’s term 'End Result System'—codified accountability that medicine adopted despite pushback. He challenges policymakers to create prediction tournaments and transparency portals that score real-world judgments. Resistance, he reminds you, is political—'kto-kogo,' who beats whom—but the long-term payoff is institutional learning.

(Note: Tetlock’s focus on measurement foreshadows newer movements like evidence-based policy design and AI forecasting calibration—fields that also treat prediction as a testable craft rather than confident punditry.)

Tetlock’s IARPA tournament brought forecasting into the laboratory. The design was simple: thousands of participants answered identical, precisely worded questions—each with clear criteria and deadlines. The experiment became the first large-scale test of whether forecasting skill exists and can be improved through training, aggregation, and feedback. The Good Judgment Project team won by a wide margin, proving that disciplined amateurs could outperform professionals.

How the Tournament Worked

Questions covered realistic geopolitical events with timelines ranging from one month to one year (“Will Russia annex territory?” “Will the euro fall below $1.20?”). Every day, participants submitted numerical probabilities online. This setup eliminated vagueness and leveled the playing field. Teams tested training modules, aggregation algorithms, and team structures—an unprecedented opportunity to compare methods directly.

What Made the Good Judgment Project Win

The GJP aggregated forecasts using an algorithm that weighted top performers more heavily and then 'extremized' probabilities—pushing them slightly toward certainty to simulate fuller information sharing. That tweak consistently improved Brier scores. The project also trained forecasters in probabilistic reasoning and encouraged frequent small updates. Over four years, GJP forecasts were roughly 60% more accurate than control groups and outperformed even analysts with access to classified data.

Scientific payoff

Forecasting tournaments turned a soft art into an experimental science—each question a controlled test of human judgment under uncertainty.

Lessons for Application

You can emulate this experiment in any setting—corporate strategy, public policy, research. Ask clear questions, require probabilities, log every forecast, and compare results. Tournaments reveal hidden talent and cultivate disciplined practice. As Tetlock puts it, forecasting skill is not mystical; it’s measurable behavior that improves under transparent conditions.

(Parenthetical note: IARPA’s work inspired later policy initiatives that use prediction markets and tournaments to anticipate global events, marking a quiet revolution in intelligence analysis.)

The heroes of this book aren’t oracles—they’re thinkers with habits you can learn. Superforecasters excel because they avoid ideological commitment, think probabilistically, and love to revise. They are not defined by credentials but by curiosity and intellectual humility. Unlike hedgehogs, who see the world through one grand theory, they resemble foxes—agile, eclectic, and comfortable with complexity.

Cognitive Habits That Matter

Superforecasters use 'active open-mindedness'—Jonathan Baron’s term for the willingness to seek disconfirming evidence. They break large problems into smaller, testable components, and express beliefs in quantitative terms. High numeracy helps: they think in fine-grained probability (53% vs 60%), which improves calibration. Lionel Levine and Sandy Sillman exemplify disciplined quant thinkers who bring structure to ambiguity.

Bayesian Intuition and Updating

Good forecasters act like intuitive Bayesians. They update incrementally—many small moves rather than sudden swings. Tim Minto’s 3.5% average change per update kept him near optimal accuracy. Doug Lorch’s Arctic forecast shows what happens when you overreact: dramatic shifts often degrade long-term calibration. Frequent, proportional updates preserve old information while integrating new signals, refining accuracy through evidence-driven nudges.

Behavior over brilliance

Subtle cognitive discipline—self-criticism, curiosity, and patient incrementalism—beats raw intelligence. You can train these habits.

Identity and Ego

Tetlock’s 'Jenga tower of beliefs' metaphor explains why forecasters outside the pundit class fare better: they have less ego invested in being right. When forecasting isn’t tied to identity, updating costs less. Jean-Pierre Beugoms, worried about overconfidence in his military specialty, consciously treated his forecasts as hypotheses. That mindset—a detachment from self—protects judgment from stubbornness.

(Note: Tetlock’s findings echo Carol Dweck’s 'growth mindset': improvement flows from humility and curiosity rather than status or ideology.)

Forecasting isn't always a solitary pursuit. When designed right, teams outperform individuals—by roughly 23%, according to Tetlock’s experiments. But the key is structure. Unstructured groups quickly fall into groupthink; structured 'superteams' foster independent thought, challenge assumptions, and amplify the collective intelligence of members. The Good Judgment Project demonstrated how communication norms make or break team performance.

How Teams Succeed

Success requires psychological safety—the ability to disagree respectfully and openly. Amy Edmondson’s studies of surgical teams taught Tetlock that safety invites candor. Superteams display independence of judgment, precision questioning, and diversity of thought. Members like Marty Rosenthal and Paul Theron modeled 'givers,' sharing information and research instead of fighting for dominance. These norms transform collective forecasting into social learning.

Emergent wisdom

Team intelligence isn’t just the sum of smart people—it emerges from communication patterns that reward accuracy over ego.

The Leader’s Role

Great forecasting leadership follows the principle of mission command—give teams a clear goal but freedom to adapt tactics. Helmuth von Moltke’s dictum, 'no plan survives first contact with the enemy,' applies equally to organizations facing uncertainty. JFK’s management during the Cuban missile crisis—inviting dissent and fostering devil’s advocacy—illustrates this philosophy. Leaders must combine decisive execution with continuous questioning, humility, and feedback systems that make correction safe.

(Parenthetical note: Tetlock’s approach resonates with modern agile management and venture innovation—structures that value adaptability, learning, and clarity of purpose over rigid command.)

Tetlock closes with intellectual humility. Forecasting has boundaries: rare, high-impact 'black swan' events—Nassim Taleb’s domain—defy precision. Daniel Kahneman reminds us that cognitive biases distort even trained judgment. Yet Tetlock insists that between unpredictable extremes and routine simplicity lies a vast middle zone of 'gray swans'—events amenable to structured prediction. Forecasting tournaments prove that realism and discipline can meaningfully reduce uncertainty in this zone.

Reconciling Kahneman and Taleb

Superforecasters incorporate Kahneman’s bias awareness and demand Taleb’s humility about fat tails. They practice scope sensitivity—adjusting probabilities across time horizons—something Kahneman initially doubted was possible. And they acknowledge Taleb’s warning without surrendering to fatalism. As Tetlock puts it, forecasting can’t banish uncertainty, but it can structure your response to it, making you resilient when the unpredictable occurs.

Forecasting for the Real World

Tetlock urges applying forecasting where it’s most effective—short to medium horizons, where facts can be measured and feedback obtained. Beyond that, build adaptive strategies: diversified portfolios, contingency plans, and institutional learning systems. The essence of wisdom is combining precise forecasting with resilience—respecting limits while still pursuing refinement.

A final lesson

Forecasting is neither omniscience nor futility—it is disciplined humility. The goal is not to predict every black swan but to see more clearly and fail more intelligently.

(Note: Tetlock’s synthesis anticipates modern risk analytics—combining probabilistic forecasts for typical outcomes with stress tests for outliers. It’s a balanced approach: rigorous prediction meets resilient design.)