ChessLFM: 2000 Elo in 230M params

mlabonne20 Sept 20264,101 viewsEnglish (US)

Chess engine Lichess Software Development

Is Stockfish distillation all you need?

LLMs are famously bad at chess. Sooner or later, they play an illegal move, resurrect a captured piece, or lose track of the position entirely. And even when the moves stay legal, they’re still quite bad: 100B+ models can still lose to casual players.

I like chess LLMs because they apply NLP techniques to an artificial, narrow language that tests their limits. Two years ago, I created the ChessLLM with data, training, and eval scripts to build tiny chess models. Small models have improved a lot since then, so I wanted to see how far I could push LFM2.5-230M with specialized training.

The final solution reaches 2004 Elo (95% CI 1972-2039) on a Stockfish-anchored ladder and runs locally in your browser via WebGPU. In this article, we'll see how to turn a tiny general-purpose decoder into a chess engine with a strong teacher, a bit of RL, and a lot of search.

A game against ChessLFM running in the browser, sped up.
Fable 5.1 (white) getting destroyed by our 230M model 🔪

One move = one token

The most important question we need to answer is: how should the LLM output a chess move?

The naive approach is text, which is what I used two years ago. Adam Karvonen's chess-GPT also trained a nanoGPT on raw PGN strings and reached roughly 1300 to 1500 Elo. It works, but the model burns capacity tracking the board state, moves span several tokens, and nothing prevents illegal moves.
DeepMind's ChessBench showed a better alternative: feed an explicit board encoding and predict Stockfish's evaluation directly. They used a 270M encoder model with an action-value head and achieved grandmaster-level blitz performance!

ChessLFM borrows ChessBench's action space and its fixed-position board encoding, but keeps a decoder-only model. Chess has only 1,968 geometrically possible moves, so we can transform each of them into a vocabulary token like <m:e2e4>. The board becomes 64 square tokens at fixed positions plus a few state tokens (side to move, castling, en passant, halfmove clock, repetition count). Another 64 tokens <v:0>...<v:63> discretize the win probability. The surgery adds 2,106 tokens in total: append rows to the embedding matrix and initialize them from the mean and covariance of the existing embeddings (classic stuff).

Why not a policy head + a value head like ChessBench or AlphaZero? Because a vanilla decoder-only model is directly deployable with ONNX export, transformers.js, and WebGPU. Honestly, I mostly wanted to make a cool WebGPU demo, so that was the easiest option.

This recipe has many advantages. For example, the prompt is a fixed 80 tokens, so the model never processes free text. Illegal moves are impossible, because we mask them before selecting our move. And the policy is an explicit distribution over moves, which is exactly what distillation and RL need later.

SFT with Stockfish distillation

SFT distills Stockfish evaluations over 126M positions. I used three sources:

Lichess position evaluations (CC0), based on engine analyses requested by Lichess users.
Lichess puzzles (CC0) provide tactics, which user-requested analyses lack.
Synthetic endgame drills cover basic mates and conversion, which almost never appear in the other datasets.

The puzzle and endgame sets are re-annotated with a local Stockfish 17.1 at MultiPV-10 (its ten best lines instead of one), so they carry a full move ranking to distill.

The SFT loss applies at only two token positions: the value token and the move token.

Term by term:

The value term must predict the correct win-probability bin token as the target (cross-entropy).
The move term must predict the single move Stockfish would play (cross-entropy again).
The KL term is classic in knowledge distillation: the student must match the teacher's full ranking over candidate moves, not just the best one.

After SFT, the model plays at about 1450 Elo. I tried longer training runs and different hyperparameters, without significant improvement. The remaining gains had to come from somewhere else...

Test-time search

On its own, the model looks exactly one move ahead, so it misses any tactic longer than that. Not great for chess! Chess engines fix this with search: play out candidate moves a few turns into the future, evaluate the positions they lead to, and pick the move that holds up best.

AlphaZero showed how a neural network keeps that search small. The network does two jobs:

Its move probabilities say which candidates are worth exploring, so the tree stays narrow.
Its position evaluation scores the end of each line, so the search never has to play out to checkmate.

ChessLFM produces both a move distribution and a value token, so it supports exactly this kind of search.

In practice, the search is small to keep things fast. It explores the 12 most likely moves at the root and the top 6 moves after that, stopping at depth 3. The value head scores the ~430 positions this reaches, and the scores travel back up the tree assuming both sides always pick their best move (minimax).

Perhaps unsurprisingly, search is a huge boost worth about +300 Elo over the raw policy.

With test-time search, evaluation quality becomes the main bottleneck. To improve it, we need to rerun SFT over all the full 126M positions (+67 Elo). The value token is a 64-way classification: the model picks one of 64 win-probability bins. The issue is that standard cross-entropy treats every wrong bin the same, so predicting bin 37 instead of 38 costs as much as predicting bin 5.

HL-Gauss (Imani & White 2018, used at scale in Stop Regressing) replaces the one-hot target with a small Gaussian centered on the true bin (σ = 1.5 bins), so neighboring bins share the probability mass. With this, predicting bin 37 instead of 38 is penalized less than predicting bin 5. The model learns smoother evaluations that rank positions more reliably. Retraining the value token with HL-Gauss cost gave another modest +60 Elo boost.

GRPO with Stockfish

The last training stage is GRPO with Stockfish as our verifier. The way the problem is defined makes RL very simple: an action is a single move token, and Stockfish can quickly score any move.

One step samples 8 legal moves per position, scores each one with Stockfish (the reward is the win-probability delta it causes), and pushes the policy toward the moves that beat the group's average. I used KL divergence with the frozen SFT policy so the distribution sharpens instead of collapsing onto one move.

Funnily, GRPO barely changes greedy play (within the confidence interval), but adds a small +16 Elo whenever the policy is used as a search prior. What actually improves is that the true best move lands inside the expanded top-6 more often.

What didn't work

I threw a bunch of ideas at this problem, so here are the ones that didn’t work:

Denser teacher data changed nothing. I re-annotated the whole mixture at MultiPV-10 so that every position carried a full soft ranking, but Elo stayed within the confidence interval.
A smarter search budget did worse. Gumbel MuZero (Danihelka et al. 2022) allocates evaluations with sequential halving, spending more of the budget on promising branches. But it lost to plain uniform depth-3 search at the same budget.
Self-improvement failed too. Expert iteration (Anthony et al. 2017) retrains the model on its own search output, like what AlphaZero and MuZero's Reanalyse use. Here, training the policy on its own searched moves degraded the Elo score by 47 points.

In hindsight, self-improvement was slightly ambitious. The teacher is overwhelmingly stronger than the student here (roughly 1200 Elo above), so distillation provides strictly better labels. It was fun to try, though!

🧑‍🍳 The full recipe

To wrap up: the final model reads a tokenized board and answers with two tokens (an evaluation token and a move token). Getting there took four training stages plus some test-time search:

Tokenizer surgery: every chess move = one token, and the board becomes a fixed 80-token prompt.
SFT distillation: imitate Stockfish's best move, its ranking of the alternatives, and its evaluation, over 126M positions.
HL-Gauss value retrain: swap the value loss for a soft-target version.
GRPO polish: per-move RL against a Stockfish reward, sharpening the move distribution.
Test-time search: a depth-3 lookahead on top of the model at inference.

Here’s the full recipe:

The final model scores 49% head-to-head against the 2000-rated anchor, on a scale that is conservative above 2000. You can test the model and give it a like using the links below: