# Human research against automated forecasting URL: https://www.market-guy.com/learn/prediction-analysis-vs-bots Updated: 2026-08-26 The interesting question is not whether a machine can forecast. It is which questions it is good at - because the answer is specific, published, and points at exactly where a person should be spending their time. ## Two different things get called a bot The first is a trading program: it reads the order book, enforces consistency between related prices, and quotes both sides. It has no opinion about the world and is not trying to have one. That is the subject of the companion guide on who is on the other side of your trade. The second is a forecasting system: it reads sources, reasons about a question, and outputs a probability. This is a genuinely different activity, competing with human analysis rather than with human market making, and it is the one worth understanding if you are trying to work out whether careful research still pays. Confusing the two produces bad conclusions in both directions. The fact that trading programs dominate order books says nothing about whether machines can forecast. The fact that forecasting systems are improving says nothing about who is quoting the market you are looking at. Key point: A quoting program enforces consistency. A forecasting system attempts correctness. They are different machines solving different problems. ## What automated forecasting is genuinely good at Breadth, first and most obviously. A system can produce a considered estimate on every open market simultaneously, which no person can. Where the edge in a market comes from nobody having bothered to look, that advantage is decisive and it compounds - the thousandth question costs the same as the first. Speed on structured questions, second. Where an answer follows from published data - a data release, a scheduled event, a threshold against a known series - retrieval and arithmetic beat judgement, and a machine does both instantly. Several of the analyses in this library are exactly that shape: a submission date plus a review clock, a distance divided by a weekly rate. Consistency, third, and this is underrated. Human forecasters are inconsistent across similar questions, susceptible to how a question is framed, and prone to updating too much on vivid news. A system applying the same procedure every time avoids all three, which is worth real accuracy even without being cleverer about any individual question. - Breadth: every market at once, at constant marginal cost. - Structured retrieval: where the answer is in a published series. - Consistency: no framing effects, no fatigue, no overreaction to a vivid headline. ## What it still struggles with Ambiguous resolution wording is the most useful weakness to know about, because this library keeps finding money in it. Deciding what a contract actually means - which of four candidate authorities settles it, whether an announced resignation counts before its effective date, whether a walkover voids a market - is an interpretive judgement about a specific text, and it is where the most reliable human findings in the research library come from. Genuinely novel situations are the second. A forecasting system reasons from patterns in what it has seen; a situation with no precedent gives it very little to work with, and it tends to fall back on superficially similar cases that are not actually analogous. A restructured competitive circuit, a legal authority substituted for another within days, an office with a contested succession - these are exactly the cases where pattern-matching misleads. Source judgement is the third. Deciding that a widely repeated figure comes originally from an attacker's leak site, or that a corporate announcement carries no obligation behind it, requires knowing how a particular information ecosystem generates its numbers. That knowledge is specific, tacit and unevenly written down. ## FAQ Q: Can AI forecast better than humans? A: Not yet on the hardest questions, and the gap is narrowing. Published benchmarks that score models and human forecasters on the same live questions put the best human forecasters ahead of frontier models, with the difference small and shrinking — but the machine advantage is concentrated on questions whose answers already sit in a dataset. Q: What is the difference between a trading bot and a forecasting system? A: A trading program reads the order book and enforces consistency between prices without any view about the world. A forecasting system reads sources and outputs a probability. They compete with different things — market making and human analysis respectively — and conflating them produces bad conclusions in both directions. Q: Where do humans still have an advantage? A: Interpreting ambiguous resolution wording, reasoning about genuinely novel situations with no clean precedent, and judging where a widely repeated number originally came from. All three are specific, tacit and poorly represented in the patterns a system learns from. Q: Does this mean research is pointless on liquid markets? A: On a liquid market about a scheduled release with unambiguous wording, the honest expectation is that you have no edge. Trading it anyway means paying the spread for the privilege of being average — which is a good reason to spend the effort elsewhere. ## Sources - [ForecastBench — A dynamic benchmark of AI forecasting capabilities (Wharton)](https://faculty.wharton.upenn.edu/wp-content/uploads/2026/02/ForecastBench_A_Dynamic_.pdf) - [Forecasting Research Institute — LLMs are closing the gap on human superforecasters](https://forecastingresearch.substack.com/p/llms-are-closing-the-gap-on-human) - [Good Judgment — What superforecasters actually said about ForecastBench](https://goodjudgment.substack.com/p/what-superforecasters-actually-said) - [AI forecasting in 2026: what 11 analyses say](https://forum.nunosempere.com/posts/Spyz3wESZu2eeqhDj/ai-forecasting-in-2026-what-11-analyses-say) --- Research tooling for prediction markets. Not financial advice.