Forecasting & Calibration

Syllabus

1. Module 1

   The outside view

   Pick reference classes that fit, turn rates into the probability for your question's window, and estimate quantities you have never seen by breaking them into parts.

   1. Choosing and combining reference classes15 min

      Every question belongs to many classes of past cases. How to choose between them, and how to combine a small, close class with a large, loose one.

      Objectives

      • Choose between competing reference classes by what drives the outcome and how many cases each holds.
      • Combine a narrow and a broad reference class into one starting estimate.
   2. Rates, windows and deadlines14 min

      A base rate is always per something — per meeting, per month, per match. How to turn it into the probability for your question's window, and why 'by <date>' questions decay as the deadline approaches.

      Objectives

      • Convert a rate per period into the probability for the question's time window.
      • Explain how the time left before a deadline changes the probability of a 'by <date>' question.
   3. Fermi estimates for threshold markets15 min

      Many markets ask whether a number will cross a line. How to estimate a quantity you have never seen by splitting it into parts, and how to turn a range into a probability for the threshold.

      Objectives

      • Break an unknown quantity into factors you can estimate and multiply them.
      • Estimate from a low and a high bound with the geometric mean and judge how far a threshold lies from it.

   Module check

   10 questions drawn from the module's pool, pass mark 80%. Retake it as often as you like: every retake draws new questions, and you see the answer and explanation for each one.

2. Module 2

   Updating

   Move your number by the right amount when evidence arrives: Bayes in odds form, telling signal from noise, and the traps — anchors, echoes, over- and underreaction — that pull you off course.

   1. Bayes in odds form16 min

      The one calculation behind every good update: turn your probability into odds, multiply by how much more likely the evidence is under Yes than under No, and turn it back.

      Objectives

      • Update a probability with a likelihood ratio by working in odds.
      • Work out a likelihood ratio from how likely the evidence is under each outcome.
   2. Signal, noise and echoes15 min

      How far a piece of news should move you, how to recognise news that only sounds informative, and why the same story heard five times is still one piece of evidence.

      Objectives

      • Judge how far evidence should move you by how much more likely it is under one outcome than the other.
      • Recognise evidence that repeats what you already counted and update on it only once.
   3. Anchors, overreaction and underreaction15 min

      The numbers that pull your estimate before you have thought, the news you react to too much, and the news you react to too little — with a correction for each.

      Objectives

      • Identify an anchor in a forecasting situation and take a step that neutralises it.
      • Tell overreaction from underreaction and name the correction for each.

   Module check

   10 questions drawn from the module's pool, pass mark 80%. Retake it as often as you like: every retake draws new questions, and you see the answer and explanation for each one.

3. Module 3

   Measuring yourself

   Score forecasts with rules that reward honesty, split a score into what you control and what you do not, and find out whether you are too sure, too timid or tilted.

   1. Scoring rules: Brier, log score and honesty15 min

      Two ways to score a probability, why the log score is merciless with confident misses, and why a proper scoring rule makes reporting what you believe the winning strategy.

      Objectives

      • Calculate a log score and explain why it punishes confident misses harder than the Brier score.
      • Explain why a proper scoring rule gives the best expected score for reporting what you believe.
   2. Reliability, resolution and uncertainty16 min

      Split a Brier score into three parts — how hard the questions were, how calibrated you were and how much your forecasts told apart — and read a calibration table band by band.

      Objectives

      • Split a Brier score into reliability, resolution and uncertainty and say what each part measures.
      • Read a calibration table and say in which range forecasts ran too high or too low.
   3. Too sure, too timid: the confidence gap and extremizing15 min

      Measure over- and underconfidence the way the calibration lab does, correct it, and learn when pushing a forecast away from 50% improves it and when it only makes it worse.

      Objectives

      • Measure over- or underconfidence as the gap between stated confidence and hit rate.
      • Apply extremizing to a forecast and say when it helps and when it hurts.

   Module check

   10 questions drawn from the module's pool, pass mark 80%. Retake it as often as you like: every retake draws new questions, and you see the answer and explanation for each one.

4. Module 4

   Forecasting in practice

   Combine forecasts without spoiling them, keep a journal you can actually learn from, know how many questions a conclusion needs, and keep your numbers current as the deadline approaches.

   1. Combining forecasts15 min

      The mean, the median and the mean of log-odds — and the two situations in which averaging makes a forecast worse instead of better.

      Objectives

      • Combine several forecasts with the mean, the median or the mean of log-odds.
      • Decide when averaging forecasts improves them and when it makes them worse.
   2. A journal you can learn from14 min

      What to write down with every forecast so you can score it later and find out why it went right or wrong — and how many forecasts a conclusion about yourself really needs.

      Objectives

      • Record a forecast with every field you need to score it and learn from it later.
      • Judge whether a track record is large enough to support a conclusion, using the standard error.
   3. Keeping forecasts current and improving them14 min

      When to revisit a forecast, how far to move it, and how to turn a scored track record into one concrete change — the loop that makes a forecaster better.

      Objectives

      • Decide when to revisit a forecast and how far new information should move it.
      • Turn a scored track record into one concrete change to your process.

   Module check

   10 questions drawn from the module's pool, pass mark 80%. Retake it as often as you like: every retake draws new questions, and you see the answer and explanation for each one.