Episode 39 · 27 minutes

Superforecasting: The Art and Science of Prediction

by Philip Tetlock and Dan Gardner, 2015

A government forecasting tournament showed that some volunteers predict world events measurably better than others, year after year, and that brief training and good teams make most people better. The edge holds for clear questions a month to a year out, and much of it turns out to be steadier, less noisy judgment. For neurodivergent leaders: the habits of an accurate forecaster read as weak presence in a room, and a scored track record is what lets them count.

Transcript

The whole episode in text, 113 lines. Miles and Nora are synthetic voices reading a script approved by Logan Williamson, LPC.

Read the full transcript

Miles: Kearney, Nebraska. A man of fifty five, retired, with a long stretch behind him at the Department of Agriculture in Arizona, part of it manual labour and part of it spreadsheets. He once set out to earn a doctorate in mathematics and decided it was beyond him.

Miles: In his spare time he answers questions. Will Russia formally annex more Ukrainian territory in the next three months. About three hundred of those so far, and every answer is dated, recorded and scored by researchers who've never met him.

Miles: Nobody books him for cable news. Nobody makes a decision because of what he thinks. And on that scorecard he's in the top two percent of thousands of volunteers, which in this book earns a title.

Miles: Today: Superforecasting, from Philip Tetlock with Dan Gardner. In two sentences: seeing a little way ahead is a real ability, it can be measured, and it's spread very unevenly. The people who have most of it think in ways you can describe and partly teach, and they're seldom the people we ask. And later, Nora's going to tell you who they resemble in a job interview.

Miles: Sixty Books. Presented by Stoar Consulting and Coaching.

Miles: Book thirty nine. This is Sixty Books, I'm Miles. One management book per episode, sixty of the strongest from the past hundred years, each read with two questions in hand: what it offers leaders and teams who think differently, and what it means now AI has a seat in the working day. Nora has the research, and this time it's most of the book.

Miles: Nora, last time a poker player told us to treat every choice as a wager and to grade the reasoning, not the result. Which left something hanging. If it's all wagers, are some people simply better at placing them?

Nora: This is the book that went and found out, with a government tournament, tens of thousands of volunteers and a scoring rule from nineteen fifty. The answer's a stubborn yes. The good ones stayed good from one year to the next.

Miles: So the skill's real.

Nora: On a particular kind of question, it's real. And the man who showed it is best remembered for a line about a chimpanzee throwing darts, which he spends the opening chapter of this book trying to take back. There was never any chimpanzee.

Miles: No author should have to put that in print.

Nora: He'd agree. It was a joke about random guessing, and the coverage turned it into experts are useless. He calls that nonsense, and this book is his longer answer.

Miles: Start with the tournament, since everything else stands on it.

Nora: The sponsor was IARPA, the research agency of the American intelligence community. From twenty eleven to twenty fifteen it ran forecasting tournaments: real questions about world events, posed in advance and scored once they resolved. Five university research groups competed. Tetlock and Barbara Mellers led one of them, the Good Judgment Project.

Miles: What sort of questions?

Nora: Will a given leader still hold office on a given date. Will two countries sign a particular agreement. Generally more than a month out and less than a year, a window I'll come back to. Over the four years, roughly five hundred questions and more than a million forecasts.

Miles: And who was answering?

Nora: Volunteers with day jobs. The publisher lists a filmmaker in Brooklyn, a retired pipe installer and a former ballroom dancer. The man in Nebraska is Bill Flack.

Miles: And how did they do?

Nora: In the book's figures, the project beat the tournament's official control group by sixty percent in the first year and seventy eight percent in the second, and the competing university teams by thirty to seventy. Margins on an error score, and what matters more is that the group published its methods in peer reviewed journals, with randomized arms, which almost nothing else on our shelf can claim.

Miles: The authors.

Nora: Philip Tetlock, born in Toronto in nineteen fifty four. Bachelor's and master's at the University of British Columbia, a doctorate in psychology from Yale in nineteen seventy nine. Berkeley, Ohio State, Berkeley again, and since twenty eleven the University of Pennsylvania, where he holds a chair in psychology and management.

Miles: And the study before this one.

Nora: Expert Political Judgment, Princeton University Press, two thousand five. Starting in nineteen eighty four and running about twenty years, he gathered roughly twenty eight thousand predictions from two hundred and eighty four experts on politics and economics. On average they did only slightly better than chance. But the average hid a split, named with a line Isaiah Berlin made famous: the fox knows many things, the hedgehog knows one big thing. Experts who ran everything through one big idea did worse than the ones who drew on many and doubted themselves. That book took two awards from the American Political Science Association, and it holds a finding I'd underline: the more famous the expert, the less accurate.

Miles: Why would that be?

Nora: His co-author explained it in an interview a few months after this book came out. The confident hedgehog with one clean story makes good television. The fox who says it depends makes a poor guest. Keep that handy for our segment.

Miles: The co-author, then.

Nora: Dan Gardner. Canadian, a law degree and a master's in history from York University in Toronto, then years as a columnist and feature writer at the Ottawa Citizen. Before this he wrote Risk, on how people misjudge danger, and Future Babble, on why expert predictions fail. His name comes back two books from now, with a different partner.

Miles: And this book.

Nora: Crown, the twenty ninth of September, twenty fifteen. Twelve chapters, an epilogue and an appendix of ten rules. A New York Times bestseller, and on the Economist's list of that year's best books.

Miles: Now the chapters, in the order he wrote them.

Nora: Chapter one: he calls himself an optimistic skeptic. Skeptical, because foresight has hard limits and plenty of people sell more of it than they've got. Optimistic, because in some situations it exists, and an intelligent, open minded, hardworking person can build it.

Miles: Then the obstacle.

Nora: Chapter three, which is the one I'd hand to anybody. You can't improve what you never score, and most forecasts can't be scored.

Miles: Why can't they?

Nora: Vague words. In March of nineteen fifty one, an American intelligence estimate said an attack on Yugoslavia that year should be considered a serious possibility. Sherman Kent, of the board that produced it, later asked his colleagues what odds they'd had in mind. The lowest meant odds of twenty to eighty. The highest meant eighty to twenty. Same document, same phrase, same board.

Miles: Unanimous, then.

Nora: About the wording. So the book's remedy is numbers, and a rule for scoring them. The Brier score, from Glenn Brier, a statistician at the American Weather Bureau, in nineteen fifty. You give a probability, the world comes out yes or no, and you're scored on the squared gap between them, averaged over all your forecasts. Zero is perfect. Fifty fifty on everything earns a middling score for life.

Miles: And what makes a good one?

Nora: Two ingredients. Calibration: when you say seventy percent, it happens about seventy percent of the time. And resolution: you're willing to go out to ninety or ten when you've earned it, instead of parking every answer near the middle. Careful and vague scores badly.

Miles: Then the people at the top.

Nora: Chapter four. Every year the project took its best two percent and put them together on elite teams. Everyone expected them to slide back toward average, as people picked for one good year usually do. Most didn't. Mellers and colleagues, twenty fifteen, in Perspectives on Psychological Science, report them holding high accuracy two years running across hundreds of questions. The book puts it at roughly seventy percent keeping their status from one year to the next.

Miles: So what are they like?

Nora: Chapters five to eight are a portrait. Supersmart? Somewhat. The typical volunteer outscored about seventy percent of the population on intelligence tests, superforecasters about eighty. Brighter, not dramatically. What sets them apart is how they work. They break a question into pieces they can estimate, which the book calls Fermi-izing, after the physicist Enrico Fermi. And they start outside the case. The book's example is a family called the Renzettis, and whether they own a pet. You don't begin with the family. You begin with the share of American households that keep one, about sixty two percent in the book, and nudge from there.

Miles: Base rate first, story second.

Nora: Yeah. Base rate first. Chapter six is granularity. Forecasters who separate sixty from sixty five beat the ones who think in tens, and when the researchers rounded superforecasters' numbers to the nearest five, they got worse. Ordinary forecasters barely moved even when the rounding was four times as coarse.

Miles: And seven and eight?

Nora: Updating, often and in small steps. Then a name for the whole disposition: perpetual beta, treating whatever you currently believe as a version due for improvement. The book calls it the strongest predictor of becoming a superforecaster, roughly three times as strong as intelligence, its nearest rival.

Miles: Is that the book talking, or the data?

Nora: Both. Mellers and colleagues, twenty fifteen, Journal of Experimental Psychology: Applied, seven hundred and forty three forecasters over two years. Accuracy went with intelligence, political knowledge and open mindedness, and with time spent deliberating and how often people updated.

Miles: Then the teams.

Nora: Chapter nine. In the book's figures, teams were twenty three percent more accurate than individuals. And this is where the randomizing sits. Mellers and colleagues, twenty fourteen, Psychological Science: forecasters assigned to conditions, and three interventions improved both calibration and resolution. Training in probability. Teaming, so people could pool information and argue over their reasons. And tracking, which put the top performers together.

Miles: How much training?

Nora: Under an hour. Chang, Chen, Mellers and Tetlock, twenty sixteen, Judgment and Decision Making: across four years of the tournament, one short module on reasoning with probabilities improved accuracy scores by six to eleven percent over the control condition. Every year.

Miles: And chapter ten's the one you flagged.

Nora: The leader's dilemma. He sets the forecaster's habits beside what a leader is supposed to deliver, confidence, decisiveness, a clear direction, and says plainly that succeeding in one role may undermine the other.

Miles: How does he get out of it?

Nora: Through Helmuth von Moltke, a nineteenth century Prussian general whose orders said what to achieve and left the how to the officer on the spot. And a distinction I'd keep. The humility good judgment needs isn't self-doubt, the feeling that you're not good enough. It's intellectual humility, about how hard the world is to read. And chapter eleven puts the obvious challenge in its own title: Are They Really So Super?

Miles: The evidence check. And this time we're starting higher up than usual.

Nora: We are. The central findings come from randomized conditions in peer reviewed journals, so the question is how far the evidence reaches. Four limits: the window, the yes or no format, the headline about intelligence analysts, and what the advantage was made of.

Miles: The window.

Nora: The questions mostly resolved within a month to a year, and they were written to resolve cleanly. Did it happen by this date, yes or no. Every finding I've given lives under that condition, among volunteers the researchers describe as above average in intelligence and political knowledge. The book says so itself. It reports expert accuracy sinking toward chance about five years out, and no evidence that anyone can forecast ten years ahead beyond the blindingly obvious.

Miles: So the choices leaders lie awake over sit outside it.

Nora: A five year strategy, a hire you'll live with for a decade. Yes. The book marks the edge where its data runs out.

Miles: The format.

Nora: An exchange worth putting on the record. Nassim Taleb argues that where rare, enormous events drive the outcome, getting yes or no questions right can tell you little about what you're exposed to. You can be well calibrated on whether a crisis happens and still be wiped out by how large it gets. Tetlock didn't wave that away. In twenty thirteen the two of them wrote a paper together drawing exactly that line, between binary predictions and real exposures.

Miles: So the book's author put his name on the critique.

Nora: Alongside the critic. The tournament's own claim is narrower than it gets quoted: skill on questions that can be scored. Taleb's reply is that what can be scored isn't always what matters. Both hold, and a reader should keep the second in view while using the first.

Miles: The analysts.

Nora: It's the jacket line. Ordinary volunteers beat intelligence analysts who had classified information. That figure first surfaced in a newspaper column in November twenty thirteen: the top forecasters about thirty percent better than the average intelligence analyst. When Tetlock describes it himself, the comparison is with a prediction market run inside the intelligence community and populated by analysts with clearances, and the margin he gives is twenty five or thirty percent. A real comparison, but with a market rather than the analysts' written assessments, and I couldn't find that head to head in a peer reviewed journal. Of everything in the book, it's the line I'd quote with the most care.

Miles: And the fourth.

Nora: The one that redraws the portrait. Satopää, Salikhov, Tetlock and Mellers, Management Science, twenty twenty one, went back into the tournament data and split each forecaster's error three ways. Bias, which leans the same direction every time. Information, meaning knowing something that matters. And noise, the scatter with no pattern in it. In the authors' own summary, about half the improvement from training, teaming and picking superforecasters came from less noise, a quarter from less bias, a quarter from more information.

Miles: So the best forecasters mostly just scatter less.

Nora: Mostly, on that analysis. Less a gift for seeing the future, more a discipline for being consistent about it.

Miles: Tier statement, please.

Nora: A scientist and a journalist writing for general readers, so this is practitioner literature, like every book on our shelf, and not peer reviewed research. Underneath it, though, sits something rare for the genre: a research programme with randomized conditions and published results that held across years. The record supports a skill on clear questions a month to a year away, which persists, which brief training improves modestly and teams improve further. It doesn't stretch to long horizons, to fat tails, or to the classified information line without its footnote. And none of that is a verdict on anybody.

Miles: Our half. You promised me a job interview.

Nora: I did, and it's an argument, because I'm joining literatures no single study has joined. Read the accurate forecaster as things you could see from across a table. Slow to commit. Talks in probabilities. Says on one hand, then on the other. Shifts when the evidence shifts. Admits a miss without fuss. Hand that list to an interview panel, and that's the candidate written up as lacking executive presence.

Miles: Evidence for the panel's side?

Nora: For what groups reward, yes. Anderson, Brion, Moore and Kennedy, twenty twelve, Journal of Personality and Social Psychology. Six studies, naturalistic and experimental. Overconfidence led to higher status in groups, short term and longer term, and one study traced why: overconfident people gave off behaviour that made them look competent. Confidence works as a signal whether it's earned or not.

Miles: And the forecaster's side.

Nora: Karvetski and colleagues, twenty twenty two, International Journal of Forecasting, analysed the written reasoning behind forecasts in several tournaments. Top forecasters showed more dialectical complexity, holding clashing views and blending them, and used more comparison classes. On paper, the on the other hand habit marks the accurate. Across a table, it sounds like someone who can't decide.

Miles: Where does neurodivergence come into it?

Nora: In two places, and both are group findings. First, how a style gets read. Sasson and colleagues, twenty seventeen, Scientific Reports. In the first of three studies, two hundred and fourteen students rated ten seconds of a mock audition by each of twenty autistic and twenty non-autistic adults, in different formats. The autistic adults were rated less favourably on video, on audio, even from a still photograph. From a transcript of the same words, the gap disappeared. And on intelligence and trustworthiness, the ratings didn't differ.

Miles: So it's style, not substance.

Nora: Yeah. Style, not substance, which is the authors' own phrase. Second, how the thinking tends to run. Brosnan, Lewton and Ashwin, twenty sixteen: higher autistic traits among ninety five students, and a small diagnosed group of seventeen young men set against eighteen, went with a lean toward deliberate reasoning over intuitive. And Farmer, Baron-Cohen and Skylark, Psychological Science, twenty seventeen, preregistered. Ninety autistic adults against two hundred and twelve controls, choosing between products with a decoy added to sway them. The autistic group's choices were more consistent and less moved by the decoy, and random responding didn't explain it.

Miles: Which is the noise finding again.

Nora: Which is why I chose it. With the caution the authors attach themselves: the same resistance to context may cost some useful updating, and updating is the habit this book prizes most. So I'm not saying autistic people forecast better. Nobody has tested neurodivergent forecasters in a tournament; I searched PubMed and came back empty. And everything I've cited concerns autism and autistic traits. I found nothing on ADHD I'd stand behind.

Miles: So what's left once the caveats are in?

Nora: A narrower claim I'd defend. Rooms reward the signal of confidence. The tournament rewarded calibration. And there's a study of the moment those two collide. Tenney, MacCoun, Spellman and Hastie, Psychological Science, twenty oh seven, two experiments on witness credibility. Once errors came to light, confident witnesses lost more credibility than unconfident ones, and the less confident could come out ahead.

Miles: But only once there's a record.

Nora: Yeah. Only once there's a record, and an interview has none. A scored track record is the accommodation. Keep score, and the person who reads badly in a room stops paying for it.

Miles: A minute, the way this show reads it. Superforecasting says near term foresight is a measurable skill, that the people who have it start from base rates, think in fine grained numbers, update often and treat beliefs as drafts, and that much of their edge is simply less noise. For anybody leading a mind built to a different plan, or working inside one, here's Monday. Stop asking for confidence and ask for a number, because a probability with a date on it turns a hunch into something you can check. Keep score, since the record is what protects the careful thinker from the room. Hear "it depends" as an opening position, and ask what it depends on. And when you choose whose judgment to trust, read the track record before you watch the delivery. Delivery is how a panel chooses. Accuracy is not what it chooses for.

Miles: Now the tools.

Nora: The standing caution waits at the very end, after the close music, in wording nobody edits. These minutes are for this book.

Miles: Go on.

Nora: Four uses, all from the ten commandments. Ask a model for the outside view before you look at your own case: what usually happens to projects like this, or hires like this. Then check its figures, because it'll offer a base rate with a certainty it hasn't earned. Ask it to break your question into pieces you can estimate. Ask it for the strongest case against the number you're holding. And keep a log, every forecast with a probability and a date, scored when it resolves. A model will cheerfully do the Brier arithmetic.

Miles: Has anybody measured whether it helps?

Nora: Tetlock's own circle has. Schoenegger, Park, Karger, Trott and Tetlock, twenty twenty five, randomized nine hundred and ninety one people to forecast six questions with one of two model assistants, or neither. One assistant was told to give careful superforecasting advice. The other was built to be overconfident and to ignore base rates. Both improved accuracy over the control group, by twenty four to twenty eight percent.

Miles: Even the bad one.

Nora: Yeah. Even the bad one, on six questions, which is a thin base. My reading, not the authors': the gain may come from having a second voice to argue with, more than from good advice. A second paper, Science Advances, twenty twenty four, pooled twelve models on thirty one yes or no questions against nine hundred and twenty five human forecasters in a three month tournament, and the machine crowd was statistically indistinguishable from the human one.

Miles: And the line to leave on.

Nora: The book's own test, turned on the machine. Don't ask whether the tool sounds sure. Make it commit to a number, write down the date, and let the record settle it, exactly as the researchers did for a retired man in Kearney.

Miles: That's Superforecasting.

Miles: Next time: Noise, by Daniel Kahneman, Olivier Sibony and Cass Sunstein. Tetlock spent a career proving that forecasters disagree with one another and that the disagreement can be scored. The next book goes a step further in: give the same forecaster the same case on a different day, and he'll disagree with himself.

Nora: And the tournament data already said half the edge was steadier judgment, so the two books meet in the middle.

Miles: Nora, you made the case and you labelled it. Thanks for both.

Nora: Pleasure, as ever.

Miles: That's this one done.

Miles: Sixty Books is made by Stoar Consulting and Coaching, in the reading room. Logan Williamson is a Licensed Professional Counselor in Texas and Louisiana. What you heard was a close reading of one book, made for leaders and the teams around them. It wasn't coaching, it wasn't counseling, and it said nothing about you in particular.

Nora: Sources for every claim are on the episode page, with the full transcript beside them. Book forty lands in two days.

Miles: For the record, two things. The voices you've heard were generated. Miles and Nora aren't people, hold no licence of any kind, and nothing in this episode was pointed at you. Each script is drafted with help from AI, then edited and approved by Logan Williamson, a Licensed Professional Counselor in Texas and Louisiana, at Stoar Consulting and Coaching.

Nora: And one note about the AI segment. The evidence that these tools help neurodivergent people at work is early. There are controlled trials now, but they measure things like conversation skill rather than how a job actually goes, and for work itself most of what exists is individual providers reporting success. So take it as promising and not proven. The tools also get rebuilt every few months, so whatever you set up today is built on something that moves.

Sources

Every claim in the episode, in the order it comes up, with where it comes from. 35 claims. Studies in psychology and medicine were checked on PubMed; everything else against the publisher, the paper or the original record.

See the sources

  1. The book opens with Bill Flack: fifty five, retired, formerly with the US Department of Agriculture in Arizona ("part pick-and-shovel work, part spreadsheet"), living in Kearney, Nebraska; he had aimed for a PhD in mathematics and decided it was beyond his abilities; he has answered roughly three hundred questions such as whether Russia will officially annex additional Ukrainian territory in the next three months; each prediction is dated, recorded and assessed for accuracy by independent scientific observers; nobody asks him to appear on cable news; the authors call him a superforecaster.

    Tetlock, P. E., & Gardner, D. (2015). Superforecasting: The Art and Science of Prediction. New York: Crown. ISBN 9780804136693 (hardcover), 9780804136709 (ebook), 9780804136716 (paperback). Chapter 1, "An Optimistic Skeptic", opening pages.

    Book excerpt, BookBrowse, https://www.bookbrowse.com/excerpts/index.cfm/book_number/5154/superforecasting (pages 1 to 3), which reproduces the opening: "Bill is fifty-five and retired, although he says if someone offered him a job he would consider it"; "worked for the US Department of Agriculture in Arizona--'part pick-and-shovel work, part spreadsheet'--but now he lives in Kearney, Nebraska"; "He was aiming for a PhD in math, but he realized it was beyond his abilities"; "Bill has answered roughly three hundred questions like 'Will Russia officially annex additional Ukrainian territory in the next three months?'"; "each one of Bill's predictions has been dated, recorded, and assessed for accuracy by independent scientific observers"; "No one bases decisions on Bill Flack's forecasts, or asks Bill to share his thoughts on CNN"; "Bill Flack is a remarkable forecaster."

  2. Superforecasters are the top two percent of the project's forecasters each year.

    Mellers, B., Ungar, L., Baron, J., et al. (2014). See item 23: "Tracking placed the highest performers (top 2% from Year 1) in elite teams."

    Mellers, B., Stone, E., Murray, T., et al. (2015). See item 20.

  3. The book's thesis in two sentences: foresight is real, measurable and unevenly spread; it can be described and partly taught.

    Tetlock & Gardner (2015), as in item 1, chapter 1, where Tetlock states: "I believe it is possible to see into the future, at least in some situations and to some extent, and that any intelligent, open-minded, and hardworking person can cultivate the requisite skills" (BookBrowse excerpt, page 3).

  4. Tetlock's earlier work is popularly remembered for a dart-throwing chimpanzee; he explains in chapter 1 that it was a punch line for "random guessing" and that the message "all expert forecasts are useless" is "nonsense".

    Tetlock & Gardner (2015), chapter 1, via the BookBrowse excerpt (pages 2 and 3): "the average expert did about as well as random guessing. Except that's not the punch line because 'random guessing' isn't funny. The punch line is about a dart-throwing chimpanzee"; "The message became 'all expert forecasts are useless,' which is nonsense"; "the results were far richer and more constructive than the punch line suggested."

    Scott Alexander, "Book Review: Superforecasting", Slate Star Codex, 4 February 2016, https://slatestarcodex.com/2016/02/04/book-review-superforecasting/ , whose reviewer notes that Tetlock takes great pains to confess that no chimps were involved and that the phrasing was a flashier way of saying random. Paraphrased here; the spoken line is "There was never any chimpanzee."

  5. IARPA, the research agency of the US intelligence community, sponsored forecasting tournaments from 2011 to 2015; five university-based research groups competed; Tetlock and Barbara Mellers led the Good Judgment Project.

    Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S. E., Moore, D., Atanasov, P., Swift, S. A., Murray, T., Stone, E., & Tetlock, P. E. (2014). "Psychological strategies for winning a geopolitical forecasting tournament." Psychological Science, 25(5), 1106-1115. PMID: 24659192. DOI: 10.1177/0956797614524255. Abstract: "Five university-based research groups competed to recruit forecasters, elicit their predictions, and aggregate those predictions."

    Mellers, B. A., & Tetlock, P. E. (2019). "From discipline-centered rivalries to solution-centered science: Producing better probability estimates for policy makers." American Psychologist, 74(3), 290-300. PMID: 30945892. DOI: 10.1037/amp0000429. Abstract: "From 2011 to 2015, the U.S. intelligence community sponsored a series of forecasting tournaments ... Our group, the Good Judgment Project, won these tournaments."

    Philip E. Tetlock, Wikipedia, https://en.wikipedia.org/wiki/Philip_E._Tetlock , recording that since 2011, with co-leader Barbara Mellers, he has directed the Good Judgment Project.

  6. Questions were generally more than a month and less than a year out; over the four years there were roughly five hundred questions and more than a million forecasts.

    AI Impacts, "Evidence on good forecasting practices from the Good Judgment Project: an accompanying blog post" (2019), https://aiimpacts.org/?p=1260 : "The questions were generally about potential geopolitical events more than a month but less than a year in the future."

    Wharton Magazine, "How to See the Future, and Know It When You See It", Spring 2016, https://magazine.wharton.upenn.edu/issues/spring-2016/how-to-see-the-future-and-know-it-when-you-see-it/ : tournaments "run between 2011 and 2015 involving tens of thousands of forecasters trying to predict about 500 questions posed by the intelligence community."

  7. In the book's figures, the project beat the official control group by 60 percent in year 1 and 78 percent in year 2, and beat its university-affiliated competitors by margins from 30 to 70 percent.

    Tetlock & Gardner (2015), p. 18, quoted in AI Impacts (as in item 7): "In year 1 GJP beat the official control group by 60%. In year 2, we beat the control group by 78%. GJP also beat its university-affiliated competitors, including the University of Michigan and MIT, by hefty margins, from 30% to 70%."

    Tetlock, P. E., Mellers, B. A., Rohrbaugh, N., & Chen, E. (2014). "Forecasting tournaments: Tools for increasing transparency and improving the quality of debate." Current Directions in Psychological Science, 23(4), 290-295. DOI: 10.1177/0963721414534257. PDF at https://faculty.wharton.upenn.edu/wp-content/uploads/2015/07/2014---forecasting-tournaments-tools-for-increasing-transparency-and-improving-debate.pdf , reporting the best algorithms "outperforming the simple average of the control group ... by 60% and other teams by 40%" in year 1.

  8. The group published its methods in peer reviewed journals, with randomized arms.

    Items 6, 21, 23 and 24. PubMed classifies Mellers et al. (2014) as a randomized controlled trial; Chang et al. (2016) assigned forecasters at random to training or control each year.

  9. Philip Tetlock was born in Toronto in 1954; BA and MA, University of British Columbia; PhD in psychology, Yale, 1979; Berkeley, Ohio State, Berkeley again (Haas School); since 2011 the University of Pennsylvania, where he holds a chair in psychology and management.

    Annenberg School for Communication, University of Pennsylvania, faculty page, https://www.asc.upenn.edu/people/faculty/philip-tetlock-phd : "Leonore Annenberg University Professor of Psychology and Management, Wharton School of Business and School of Arts and Sciences."

  10. Expert Political Judgment (Princeton University Press, 2005): starting in 1984 and running about twenty years, roughly 28,000 predictions from 284 experts; on average only slightly better than chance; foxes outperformed hedgehogs; the distinction comes from a line Isaiah Berlin made famous; the book won two American Political Science Association awards; and accuracy was inversely related to fame.

    Tetlock, P. E. (2005). Expert Political Judgment: How Good Is It? How Can We Know? Princeton, NJ: Princeton University Press. ISBN 9780691123028.

    Tetlock & Gardner (2015), chapter 1 (BookBrowse excerpt, page 2): "It was a long slog that took about twenty years, from 1984 to 2004."

    The Hedgehog and the Fox, Wikipedia, https://en.wikipedia.org/wiki/The_Hedgehog_and_the_Fox : Isaiah Berlin's 1953 essay takes its title from the Archilochus fragment "a fox knows many things, but a hedgehog knows one big thing", and Tetlock drew on the distinction in Expert Political Judgment.

  11. Dan Gardner, in an interview published 1 February 2016, gave the mechanism for the fame finding: hedgehogs make compelling television guests, foxes make poor media personalities.

    ING Newsroom, "The more famous an expert is, the less accurate their forecasts are", 1 February 2016, https://www.ing.com/Newsroom/News/The-more-famous-an-expert-is-the-less-accurate-their-forecasts-are.htm . Gardner: "The data shows an inverse correlation between fame and accuracy: the more famous an expert is, the less accurate their forecasts are", with the explanation that hedgehog-type experts make compelling television guests and fox-type experts poor media personalities.

  12. Superforecasting was published by Crown on 29 September 2015; twelve chapters, an epilogue and an appendix of ten commandments; a New York Times bestseller and one of The Economist's best books of the year.

    Iowa City Public Library catalogue record, https://search.icpl.org/Record/1459851 , table of contents: 1 An Optimistic Skeptic; 2 Illusions of Knowledge; 3 Keeping Score; 4 Superforecasters; 5 Supersmart?; 6 Superquants?; 7 Supernewsjunkies?; 8 Perpetual Beta; 9 Superteams; 10 The Leader's Dilemma; 11 Are They Really So Super?; 12 What's Next?; Epilogue; An Invitation; Appendix: Ten Commandments for Aspiring Superforecasters.

  13. Chapter 1: Tetlock calls himself an optimistic skeptic, skeptical of the limits of foresight and of sellers of questionable insight, optimistic that foresight exists in some situations and can be cultivated.

    Tetlock & Gardner (2015), chapter 1 (BookBrowse excerpt, page 3): "There are shady peddlers of questionable insights in the forecasting marketplace. There are also limits to foresight that may just not be surmountable"; and the optimistic sentence quoted in item 3.

  14. In March 1951 National Intelligence Estimate 29-51 said an attack on Yugoslavia in 1951 "should be considered a serious possibility"; Sherman Kent polled the board, and the readings ran from odds of 20 to 80 to odds of 80 to 20. The book tells this story in chapter 3.

    Kent, S. (1964). "Words of Estimative Probability." Studies in Intelligence, 8(4). Central Intelligence Agency, Center for the Study of Intelligence, https://www.cia.gov/resources/csi/static/Words-of-Estimative-Probability.pdf : the estimate of March 1951 concluded "an attack on Yugoslavia in 1951 should be considered a serious possibility"; when Kent asked Board members, "the low man was thinking of about 20 to 80, the high of 80 to 20."

    Tetlock & Gardner (2015), chapter 3, as quoted by Bryan Caplan, "Superforecasting: Supremely Awesome", EconLog, October 2015, https://www.econlib.org/archives/2015/10/superforecastin.html : "One analyst said it meant odds of about 80 to 20 ... Another thought it meant odds of 20 to 80 - exactly the opposite."

  15. The Brier score was introduced by Glenn Brier, a statistician with the US Weather Bureau, in 1950: the mean squared difference between probability forecasts and outcomes, zero being perfect.

    Brier, G. W. (1950). "Verification of forecasts expressed in terms of probability." Monthly Weather Review, 78(1), 1-3.

    Glenn W. Brier, Wikipedia, https://en.wikipedia.org/wiki/Glenn_W._Brier : statistician with the US Weather Bureau's Office of Meteorological Research from 1939.

  16. A good score has two ingredients, calibration and resolution.

    Mellers et al. (2014), as in item 6: "probability training, team collaboration, and tracking improved both calibration and resolution."

    Tetlock & Gardner (2015), chapter 3, on calibration and resolution as the two components of Brier accuracy.

  17. Each year the top two percent were placed in elite teams; defying expected regression to the mean two years in a row, superforecasters maintained high accuracy across hundreds of questions; the book puts the share keeping their status from one year to the next at roughly 70 percent.

    Mellers, B., Stone, E., Murray, T., Minster, A., Rohrbaugh, N., Bishop, M., Chen, E., Baker, J., Hou, Y., Horowitz, M., Ungar, L., & Tetlock, P. (2015). "Identifying and cultivating superforecasters as a method of improving probabilistic predictions." Perspectives on Psychological Science, 10(3), 267-281. PMID: 25987508. DOI: 10.1177/1745691615577794. Abstract: "culling off top performers each year and assigning them into elite teams of superforecasters. Defying expectations of regression toward the mean 2 years in a row, superforecasters maintained high accuracy across hundreds of questions and a wide array of topics."

    Tetlock & Gardner (2015), p. 104, quoted in AI Impacts (item 7): "roughly 70% of the superforecasters maintained their status from one year to the next."

  18. The average volunteer scored above about 70 percent of the population on intelligence tests and superforecasters above about 80 percent; superforecasters decompose questions ("Fermi-izing") and start from the outside view, the book's example being the Renzetti family and the roughly 62 percent of American households that own a pet.

    Tetlock & Gardner (2015), chapter 5, as summarized by Jason Collins, "Tetlock and Gardner's Superforecasting", https://blog.jcx.au/posts/tetlock-and-gardners-superforecasting-the-art-and-science-of-prediction.html : "Whereas the average Good Judgment participant scored better than 70% of the population on IQ, superforecasters were better than about 80%."

    Tetlock & Gardner (2015), chapter 5, as quoted by Bryan Caplan, "Where Superforecasters Start", EconLog, October 2015, https://www.econlib.org/archives/2015/10/outside_first.html : "A few minutes with Google tells me about 62% of American households own pets. That's the outside view here... Then I will turn to the inside view."

    "Fermi-izing" as a named practice: the appendix's Ten Commandments, which include breaking "seemingly intractable problems into tractable sub-problems", as listed in Life Itself, "Notes on Tetlock and Gardner's Superforecasting", https://lifeitself.org/blog/notes-on-tetlock-and-gardners-superforecasting . The attribution to Enrico Fermi is the book's.

  19. Granularity: forecasters who use finer distinctions are more accurate; rounding superforecasters' forecasts to the nearest 0.05 reduced their accuracy while regular forecasters lost little even from rounding four times as coarse. Perpetual beta is the strongest predictor of becoming a superforecaster, roughly three times as powerful as intelligence.

    Tetlock & Gardner (2015), chapters 6 and 8, as quoted in Caplan, EconLog, as in item 17: "the average forecaster who sticks with tens - 20%, 30%, 40% - is less accurate than the finer-grained forecaster who uses fives"; "superforecasters lost accuracy in response to even the smallest-scale rounding, to the nearest .05, whereas regular forecasters lost little even from rounding four times as large"; "The strongest predictor of rising into the ranks of superforecasters is perpetual beta, the degree to which one is committed to belief updating and self-improvement. It is roughly three times as powerful a predictor as its closest rival, intelligence."

  20. In a study of 743 forecasters over two years, accuracy was predicted by intelligence, political knowledge, open-mindedness, deliberation time and frequency of belief updating; participants were above average in intelligence and political knowledge.

    Mellers, B., Stone, E., Atanasov, P., Rohrbaugh, N., Metz, S. E., Ungar, L., Bishop, M. M., Horowitz, M., Merkle, E., & Tetlock, P. (2015). "The psychology of intelligence analysis: drivers of prediction accuracy in world politics." Journal of Experimental Psychology: Applied, 21(1), 1-14. PMID: 25581088. DOI: 10.1037/xap0000040.

  21. Teams were 23 percent more accurate than individuals (book); in a randomized design, probability training, teaming and tracking each improved calibration and resolution.

    Tetlock & Gardner (2015), p. 201, quoted in AI Impacts (item 7): "On average, teams were 23% more accurate than individuals."

    Mellers et al. (2014), as in item 6. PubMed publication type includes Randomized Controlled Trial. Abstract: "Probability training corrected cognitive biases, encouraged forecasters to use reference classes ... Teaming allowed forecasters to share information and discuss the rationales behind their beliefs. Tracking placed the highest performers (top 2% from Year 1) in elite teams ... probability training, team collaboration, and tracking improved both calibration and resolution."

  22. A cognitive debiasing training module lasting less than an hour improved Brier scores by 6 to 11 percent over control, across four years.

    Chang, W., Chen, E., Mellers, B., & Tetlock, P. (2016). "Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments." Judgment and Decision Making, 11(5), 509-526. Open text at https://dlab.sauder.ubc.ca/sjdm/journal/16/16511/jdm16511.html . Abstract: "Although the training lasted less than one hour, it consistently improved accuracy (Brier scores) by 6 to 11% over the control condition."

  23. Chapter 10, the leader's dilemma: the forecaster's habits sit uneasily with what leaders are expected to deliver; "what is required to succeed at one role may undermine the other"; the book turns to Helmuth von Moltke, whose orders specified what to achieve and left how to subordinates; and the humility good judgment requires is not self-doubt but intellectual humility. Chapter 11 is titled "Are They Really So Super?"

    Tetlock & Gardner (2015), chapter 10, two reader highlights on Goodreads: "But look at the style of thinking that produces superforecasting and consider how it squares with what leaders must deliver" (https://www.goodreads.com/notes/24678534-superforecasting/5860995-azriel/fd87c2cd-2575-40e1-87d5-cbd19bfb361f); "This looks like a serious dilemma. Leaders must be forecasters and leaders but it seems that what is required to succeed at one role may undermine the other" (https://www.goodreads.com/notes/24678534-superforecasting/74509229-wally-bock/742d5f86-3ec0-443e-b6ca-a6eca04fbc4f).

    Tetlock & Gardner (2015), chapter 10, as quoted by CXO Advisory, "A Few Notes on Superforecasting", https://www.cxoadvisory.com/big-ideas/a-few-notes-on-superforecasting : "The humility required for good judgment is not self-doubt - the sense that you are untalented, unintelligent, or unworthy. It is intellectual humility."

    Shortform, "Auftragstaktik: The Brainchild of General Moltke", https://www.shortform.com/blog/auftragstaktik/ , summarizing the book's use of Moltke: the approach "gave superior officers the authority to tell subordinates what to do, but not how to do it."

  24. Tournament questions mostly resolved within a month to a year and were written to resolve cleanly; participants were above average in intelligence and political knowledge; the book reports expert accuracy declining toward chance about five years out and no evidence of forecasting ten years out beyond the obvious.

    Mellers et al. (2015), Journal of Experimental Psychology: Applied, as in item 23 (participants above average).

    Tetlock & Gardner (2015), p. 243, quoted in AI Impacts: "there is no evidence that geopolitical or economic forecasters can predict anything ten years out beyond the excruciatingly obvious ... the accuracy of expert predictions declined toward chance five years out."

  25. Taleb argues that binary (yes or no) predictions are poorly matched to real exposures with variable payoffs in fat-tailed domains; Tetlock co-wrote the 2013 paper making this distinction.

    Taleb, N. N., & Tetlock, P. E. (2013). "On the Difference between Binary Prediction and True Exposure with Implications for Forecasting Tournaments and Decision Making Research." Working paper. https://www.stat.berkeley.edu/users/aldous/157/Papers/taleb_tetlock.pdf . Abstract: "There are serious statistical differences between predictions, bets, and exposures that have a yes/no type of payoff, the 'binaries', and those that have varying payoffs ... Real world exposures tend to belong to the variable category, and are poorly captured by binaries ... variable exposures are sensitive to Black Swan effects, model errors, and prediction problems, while the binaries are largely immune to them."

  26. The claim that ordinary volunteers beat intelligence analysts with classified information first appeared publicly in a newspaper column of 1 November 2013 ("about 30 percent better than the average for intelligence community analysts"); Tetlock describes the comparison as with a prediction market inside the intelligence community populated by analysts with classified information, by 25 or 30 percent.

    Ignatius, D. (2013, November 1). "More chatter than needed." The Washington Post. Reproduced at https://www.belfercenter.org/publication/more-chatter-needed : "the top forecasters, drawn from universities and elsewhere, performed about 30 percent better than the average for intelligence community analysts."

    Tetlock, P. (2015). "Edge Master Class 2015: A Short Course in Superforecasting, Class II." Edge.org, https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii : "the Good Judgment Project outperformed a prediction market inside the intelligence community, which was populated with professional analysts who had classified information, by 25 or 30 percent", citing the Ignatius column.

    "I couldn't find that head to head in a peer reviewed journal" is a statement of the search result, not a claim that none exists. The comparison circulated in a working paper (Goldstein et al.) discussed at https://forum.effectivealtruism.org/posts/JQaeuuWiFfXCGDKNp/comparing-superforecasting-and-the-intelligence-community , which reports that it compared Good Judgment aggregates with the intelligence community market over 139 questions in year 3, not a superforecaster market; that working paper could not be retrieved and nothing from it is spoken.

  27. Decomposing forecaster error into bias, information and noise, noise reduction played the dominant, consistent role; in the authors' institution's summary, about half the accuracy improvement came from less noise, a quarter from less bias and a quarter from more information.

    Satopää, V. A., Salikhov, M., Tetlock, P. E., & Mellers, B. (2021). "Bias, Information, Noise: The BIN Model of Forecasting." Management Science, 67(12), 7599-7618. DOI: 10.1287/mnsc.2020.3882. Abstract: "A four-year series of subjective probability forecasting tournaments ... revealed a host of replicable drivers of predictive accuracy, including experimental interventions such as training in probabilistic reasoning, anti-groupthink teaming, and tracking of talent ... The BIN model reveals that noise reduction plays a surprisingly consistent role across all three methods of enhancing performance."

    INSEAD Knowledge, "The Secret Ingredients of 'Superforecasting'", https://knowledge.insead.edu/strategy/secret-ingredients-superforecasting : "about 50 percent of the accuracy improvements can be attributed to noise reduction, 25 percent to tamping down bias, and 25 percent to increased information"; noise defined as "errors that, unlike bias, have no pattern or system behind them."

  28. Overconfidence leads to higher social status in groups, short and longer term, in naturalistic and experimental designs, through a behavioural signature that makes the person appear competent.

    Anderson, C., Brion, S., Moore, D. A., & Kennedy, J. A. (2012). "A status-enhancement account of overconfidence." Journal of Personality and Social Psychology, 103(4), 718-735. PMID: 22800286. DOI: 10.1037/a0029395.

  29. Top forecasters' written rationales show higher dialectical complexity and more use of comparison classes; training and teaming nudge rationales in the same direction.

    Karvetski, C. W., Meinel, C., Maxwell, D. T., Lu, Y., Mellers, B. A., & Tetlock, P. E. (2022). "What do forecasting rationales reveal about thinking patterns of top geopolitical forecasters?" International Journal of Forecasting, 38(2), 688-704 (bibliographic record at https://ideas.repec.org/a/eee/intfor/v38y2022i2p688-704.html ; DOI not confirmed and so not given). Abstract: "top forecasters show higher dialectical complexity in their rationales and use more comparison classes"; dialectical complexity gauges "tolerance of clashing perspectives and efforts to blend them into coherent conclusions."

  30. Higher autistic traits in 95 students, and 17 young men with an autism diagnosis compared with 18 without, went with a lean toward deliberative over intuitive reasoning.

    Brosnan, M., Lewton, M., & Ashwin, C. (2016). "Reasoning on the autism spectrum: a dual process theory account." Journal of Autism and Developmental Disorders, 46(6), 2115-2125. PMID: 26960339. PMC: PMC4860198. DOI: 10.1007/s10803-016-2742-4.

    *Cut for runtime: Lewton, M., Ashwin, C., & Brosnan, M. (2019). "Syllogistic reasoning reveals reduced bias in people with higher autistic-like traits from the general population." Autism, 23(5), 1311-1321. PMID: 30409029. DOI: 10.1177/1362361318808779. 189 adults, same pattern over and above general cognitive ability. And Brosnan, M., & Ashwin, C. (2023; online November 2022). "Thinking, fast and slow on the autism spectrum." Autism*, 27(5), 1245-1255. PMID: 36325717. DOI: 10.1177/13623613221132437, finding that under time pressure both autistic and non-autistic groups gave more intuitive answers.

  31. In a preregistered study, 90 autistic adults made more consistent choices than 212 controls when a decoy option was added, and the difference was not explained by noisy responding; the authors caution that the price may be a reduction in adaptive updating.

    Farmer, G. D., Baron-Cohen, S., & Skylark, W. J. (2017). "People with autism spectrum conditions make more consistent decisions." Psychological Science, 28(8), 1067-1076. PMID: 28635378. PMC: PMC5548251. DOI: 10.1177/0956797617694867.

  32. When witnesses' errors are revealed, confident witnesses lose more credibility than unconfident ones, and after an error less confident witnesses may appear more credible; people infer calibration.

    Tenney, E. R., MacCoun, R. J., Spellman, B. A., & Hastie, R. (2007). "Calibration trumps confidence as a basis for witness credibility." Psychological Science, 18(1), 46-50. PMID: 17362377. DOI: 10.1111/j.1467-9280.2007.01847.x.

  33. The four uses (outside view first, decomposition, the strongest case against your number, and a scored forecasting log) come from the book's Ten Commandments.

    Tetlock & Gardner (2015), Appendix, "Ten Commandments for Aspiring Superforecasters", as listed in Life Itself (item 21): triage; breaking problems into sub-problems; balancing inside and outside views; updating; looking for clashing causal forces; granularity; balancing under- and overconfidence; learning from errors; bringing out the best in others; mastering the error-balancing bicycle. The application to a language model is the show's, not the book's.

  34. In a randomized study of 991 participants forecasting six questions, both an LLM assistant prompted to give superforecasting advice and one designed to be overconfident and base-rate neglecting improved accuracy by 24 to 28 percent over a control.

    Schoenegger, P., Park, P. S., Karger, E., Trott, S., & Tetlock, P. E. (2025). "AI-augmented predictions: LLM assistants improve human forecasting accuracy." ACM Transactions on Interactive Intelligent Systems, 15(1). DOI: 10.1145/3707649. Preprint: https://arxiv.org/abs/2402.07862 . Abstract: "interacting with each of our frontier LLM assistants significantly enhances prediction accuracy by between 24% and 28% compared to the control group"; one assistant "designed to provide high-quality ('superforecasting') advice", the other "designed to be overconfident and base-rate neglecting"; the effect did not differ between high- and low-skilled forecasters.

  35. An ensemble of 12 LLMs forecasting 31 binary questions was statistically indistinguishable from a crowd of 925 human forecasters in a three month tournament.

    Schoenegger, P., Tuminauskaite, I., Park, P. S., Bastos, R. V. S., & Tetlock, P. E. (2024). "Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy." Science Advances, 10(45), eadp1528. PMID: 39514669. PMC: PMC11800985. DOI: 10.1126/sciadv.adp1528.

All Sixty Books episodes · Subscribe by RSS