Learn

The Disagreement Test: how to evaluate a betting model in two checks

If you want to know how to evaluate a betting model, the Disagreement Test asks two things: does the model really disagree with the market price, and do those disagreements beat the closing line over many bets. Accuracy alone can't settle it, because betting odds are already hard to beat at forecasting.

Updated October 21, 2026 · 6 min read

Most model checks start with accuracy: how many games did it get right. That is the wrong first question for betting, because the market is already a very good forecaster. The Disagreement Test replaces it with two checks you can run on any model, from a spreadsheet you built to an AI predictions product.

The two checks, in the same order as the video:

  1. It disagrees with the price. The model's chance for an outcome differs from the market's fair chance by enough to matter, and not only on the games where everyone already agrees.
  2. Its disagreements beat the close. When you take the prices it likes, those prices keep beating the closing line over many bets.

Why accuracy is the wrong starting point

In a study published in PLOS ONE, Wunderlich and Memmert note that betting odds are frequently found to outperform mathematical models in sports forecasting. So a model that tries to win by predicting better than the odds is picking the hardest fight there is.

There is another way. Hubáček and Šír proved in a 2020 paper that it is possible to make systematic profits with an inferior model, as long as its mistakes are not correlated with the market's. Put plainly, a model doesn't have to be smarter than the market. It has to be wrong in different places, and right where it disagrees.

Before you start

The test works the same on any source of predictions. Rithmm, for example, describes AI predictions with a win probability and an edge for each game, plus tools to build or copy models, and the two checks below apply to that output just as they apply to your own spreadsheet or to ours.

You need three things for each game the model covers:

Check 1. Does it really disagree with the price?

Log the model's chance next to the market's fair chance for every game, not just the ones it recommends. Then look at the gap.

Game (demo) Market fair chance Model chance Gap Recommended?
Hawks vs Wolves 52.0% 58.0% +6.0 yes
Bears vs Owls 61.0% 61.5% +0.5 no
Kings vs Jets 47.0% 44.0% -3.0 other side
Stars vs Foxes 55.0% 55.2% +0.2 no

All four rows are a hypothetical example.

What you are checking:

  1. Most gaps are small. A model whose chances sit far from the market on almost every game is usually broken, not brilliant.
  2. The big gaps have a reason. Injury news the market hasn't priced, a rest spot, a matchup the model handles differently. Write the reason next to the row.
  3. It isn't just copying the price. If every gap is near zero, the model adds nothing. That is the correlation problem from the 2020 paper.

Convert prices to chances with the moneyline formula: price divided by price plus one hundred for minus prices, one hundred divided by price plus one hundred for plus prices, then remove the margin.

Check 2. Do the disagreements beat the close?

The edge number a model shows is its claim. The closing line is the market's last word, and grading against it gives you an answer on every bet instead of waiting for hundreds of results.

For each bet the model recommends:

  1. Record the price you took and the closing price for the same side.
  2. Mark whether your price beat the close. A better number than the close counts as a beat.
  3. Keep the rule fixed. Decide in advance which gaps you bet (for example, gaps of three points or more) and don't change it mid-test.
  4. Count across many bets. One week tells you very little. Look at the share of beats after a few hundred bets, and at the average size of the beat.

Our closing line value calculator turns each pair of prices into a beat or a miss.

The worksheet

Copy this into a sheet. One row per recommended bet.

Date Game Model chance Market fair chance Gap Price taken Closing price Beat the close?
(date) (game) 58.0% 52.0% +6.0 -108 -115 yes

The example row is hypothetical. Fill the rest from your own bets.

What to watch out for

  • Survivorship. If you only log the recommended games, you can't run check 1. Log every game.
  • Tool-shopping. If you test several models and keep the best-looking one, some of that result is luck.
  • Moving the goalposts. A model that "would have" won with a different threshold hasn't passed.
  • Account limits. A model that keeps beating the close can get your account limited, so plan stakes with that in mind.

How to tell it worked

A model passes when two things are true at once: its disagreements with the market are mostly small and explainable, and across a few hundred recommended bets your prices beat the close more often than not. If it passes check 1 but fails check 2, its disagreements are noise. If it fails check 1, there is nothing to test. Every model on Quant+ has a scorecard that shows how often it beats the close, so you can run the same test on ours.

21+. For informational purposes only. Nothing is guaranteed. If gambling stops being fun, call 1-800-GAMBLER.

Questions people ask

Does a model need to be more accurate than the odds to make money?

Not necessarily. Hubáček and Šír showed in 2020 that a model that is worse at predicting can still profit if its errors are not correlated with the market's. What matters is where it disagrees, and whether those spots are right.

What is the closing line and why use it?

It is the last price before the game starts, when the most information is in the market. If the prices you take keep beating it, that is evidence your disagreements are real, and you can measure it on every bet.

Is an AI prediction with an edge number enough?

An edge number is the model's claim. Treat it as a hypothesis and grade it against the closing price over a large sample before you size bets on it.

Does Quant+ promise that its models win?

No. Quant+ publishes a scorecard for each model, including how often it beats the close. It does not place bets, and nothing is guaranteed. 21+.

Sources

  1. The Betting Odds Rating System: Using soccer forecasts to forecast soccer, PLOS ONE (Wunderlich, Memmert)
  2. Beating the market with a bad predictive model, arXiv (Hubáček, Šír)
  3. Rithmm, Rithmm
  4. What a sports betting model outputs, Rithmm
  5. Moneyline odds, Wikipedia