How our signals are scored
How the smart-money verdict and the AI estimates are checked against resolved markets — the rules, and what the scoring cannot see.
Last updated Oct 11, 2026
What is measured
When a qualified-wallet cohort takes a directional position, that statement is recorded together with the market price at that moment. Once the market resolves, the pair is scored: was the direction right, and — the number that actually matters — was the signal closer to the outcome than the price it disagreed with?
The second question is the point. A hit rate of 61 percent against a price that already said 65 percent is a failure, and the bare hit rate would report it as a success. Both numbers are therefore always taken together, the signal's Brier score next to the market's on the same cases.
What the scores are used for
The scores are used internally, to check and tune the cohort rules and the signal thresholds. They are not published on this site. Read the verdict as context next to the price, not as a forecast.
The rules stay fixed regardless of the outcome. A signal strength is only scored once at least 30 cases stand behind it, the strengths are never pooled to reach that number, and cases recorded under the wallet rules before their correction in August 2026 are excluded rather than mixed in.
What the scoring cannot see
Snapshots are only taken on the markets the coverage job reaches, which are the highest-volume open ones. Whatever the scores say, they say it about those markets — not about Polymarket as a whole. This is the same boundary that applies to the cohort itself, and it does not disappear once the sample is large enough.
The calibration of the AI probability estimates is a separate measurement. Every analysis has been logged with its probability since July 2026, and since October 2026 these estimates are scored internally against resolved outcomes, next to the market price at the time. That result is not published either: the sample is still small, and the instructions behind the estimates changed in October 2026, so each version is now measured on its own.