A season scoreboard from eval.horse titled The Season, In One Bar Each: one horizontal bar per model showing points earned as a share of points available, running from deepseek-v4-flash at 32.5 percent down to Claude Opus 5 at 11.1 percent, with the morning line and random baselines drawn in blue among them.

Introducing eval.horse

Aug 27, 2026 · 6 min read ai evals infrastructure

I built a new benchmark called eval.horse. It has AI models handicap real horse races using the same race card, research tools, and lock time. Each model ranks the full field from first to last and gives every horse a probability of winning. The picks are published before the race, the race happens, and then they’re scored against the result.

I mostly built it because I thought it would be fun and also a good test for ZeroSignal. I’ve been spending more time around horse racing lately and was curious whether the models would be any good at handicapping.

Most benchmarks are built from questions where the answer was already known when the benchmark was created. That’s obviously useful, but it also creates problems around contamination, memorization and eventually saturation. The recent OpenAI / Hugging Face incident is a good example of how messy benchmarking can get.

A horse race has a convenient property: the answer doesn’t exist yet.

When the models handicap a race nobody knows which horse is going to win. The result can’t be sitting somewhere in their training data because the event hasn’t happened. And there’s no chance a model can actually influence the result (atleast yet).

An unknown answer on its own would make for a terrible benchmark. The useful part is that these questions resolve quickly and unambiguously. Each race is basically a new prediction problem generated by the real world, with ground truth arriving shortly after the models commit.

Horse racing also gives them quite a lot to reason over. There are speed figures, recent form, class changes, pace, distance, surface, post position, trainers, jockeys, workouts, pedigree, weather, scratches and whatever else the model decides is useful. Much of the information is public and reasonably structured, but there’s no formula that tells you which horse crosses the line first.

I don’t think this isolates some pure concept of model “reasoning,” nor am I sure that’d even be possible here. A model can search badly, choose bad sources, interpret good information poorly, or be badly calibrated despite reaching the right conclusion. eval.horse measures that whole process, which is closer to how people actually use these models anyway.

How it works

Every model gets the same race card and the same output contract. They also get web search and web read tools, so they can research the race before answering.

I didn’t want this to become a test of which model happened to memorize the most horse racing data during training. If one model thinks trainer statistics matter and another wants to spend its time reading workouts or looking at pace, they’re free to do that. The tools are the same; what they do with them isn’t.

Every model has to rank every horse from first to last. It can’t sit out a difficult race, only give me the three horses it likes, or refuse to make a call. The first three positions become its win, place and show predictions.

The predicted order of finish for one race, one model per row, each field ranked left to right by the win probability that model assigned. The actual finishing order runs across the top, and each of the first three seats is marked as a hit or a miss against it. Gemini 3.7 Flash has the top three exactly right; most of the rest have the winner somewhere down the list.

It also has to assign a win probability to every horse. I think those probabilities may eventually be more interesting than the picks themselves. Picking a 3/5 favorite to win and giving it a 60% chance is very different from picking the same horse and giving it 20%. Over enough races we can start looking at calibration, not just how often the horse at the top of the list happened to win.

The eval.horse page for Gemini 3.7 Flash, showing score, points, win/place/show hit rates, exact 1-2-3 rate and log loss, with a season score line over graded races and a calibration plot of every probability it published, binned against the perfect-calibration diagonal.

Before the first post time the predictions are published and hashed. Once the race starts, nothing can be revised with the benefit of hindsight.

One race tells us almost nothing. Horse racing is noisy, and a good prediction can lose because a horse breaks badly, gets boxed in, or simply has a bad day. The benchmark gets more interesting as the sample grows.

The scoring itself will probably evolve too. Winner accuracy is obvious and easy to understand, but it shouldn’t be the only thing that matters. I want to look at probability calibration, the quality of the full rankings, and eventually compare the models against some boring baselines: always picking the betting favorite, using market-implied probabilities, and so on. If a model is ahead after hundreds of races but can’t reliably beat the morning line, that’s important context. If one stays ahead across several measures and the difference persists as the sample grows, that’s more interesting.

I also wouldn’t call whoever sits at the top of eval.horse the smartest model. It’d mean it’s been better at this particular combination of research, tool use, prediction and uncertainty. That’s narrower, but still useful.

Why not let them gamble?

I considered giving every model a bankroll and letting them bet. I still think that could be fun, but it changes the question.

A model can be an excellent handicapper and a terrible bettor. Once money is involved it has to decide whether its estimated probability differs enough from the market to have an edge, how large the edge is, how much to wager, and how much variance it’s willing to tolerate. Now I’m evaluating betting strategy on top of handicapping.

For the first version I wanted fewer moving parts. Every model calls every race and gets graded on the prediction.

The market comparison can come later. In fact, one of the results I’d find most interesting is whether any model can consistently produce probabilities that are better calibrated than the betting market. I’m nowhere near having enough races to say anything useful about that yet.

ZeroSignal

There’s also a ZeroSignal connection to all of this.

I built the site using my current favorite coding agent (PI) pointed at the ZeroSignal proxy. When I needed to actually run all of the models for the benchmark, I ended up using ZeroSignal there too. It allowed me to set-up a harness to test dozens of models at once.

eval.horse can send the same request to a bunch of different models through one interface, and those models can use web search and page reading during inference. I didn’t have to build and maintain a different provider integration for each model I wanted to evaluate.

What happens now

I’ve got no idea what the leaderboard will look like after a meaningful number of races. One model might separate itself. They might all converge around the same obvious horses. One might pick more winners while another is much better calibrated. They might mostly recreate what the betting market already knows. They might all suck.

The current standings, sorted by score, with win, place, show, exact and log loss columns beside each model. Every row is flagged as a thin sample, and a morning line baseline sits fifth, ahead of most of the models.

For now I mostly want to let eval.horse accumulate races and see if anything real starts to emerge from the noise.