evals

CORE Bench

The DCLM CORE metric: 22 in-context-learning tasks, each accuracy centered against random chance and averaged. The same number nanochat speedruns chase — GPT-2 1.6B sits at 0.2565. Scores are comparable only within a tier (the per-task example budget): smoke 50, standard 500, full everything. How to hook a model up.

paste into your coding agent — it implements the one scoring endpoint your model needs
CORE over timeone measurement scale at a time —
0.00.10.2GPT-2 1.6B (nanochat target)GPT-2 124MGPT-2 124M (reference)karpathyVidyaVidya ternary
GPT-2 124M (reference)karpathyVidyaVidya ternary

Standings

Each model's newest complete attempt, per tier.

ModelCORETierCheckpointWhenHistory
GPT-2 124M (reference)0.0796smokeparity gate2026-08-20 08:36attempts
karpathy0.0610smokestep-109270, Alpha sm_86; greedy-pair prompts over 768 ids keep their last 7682026-10-11 07:01attempts
karpathy0.0578standardstep-109270, Alpha sm_86; greedy-pair prompts over 768 ids keep their last 768 (524 of 3,742)2026-10-11 07:42attempts
Vidya0.0436smokeruns/polar-dense: polar dense 4L/32H, 30000 updates, 1.8469 bpb, 3.62 M MAC/byte, 2.14e+16 FLOP2026-10-21 20:21attempts
Vidya0.0182standardruns/polar-dense: polar dense 4L/32H, 30000 updates (983 MB), 1.8469 bpb, 3.62 M MAC/byte, 2.14e+16 FLOP2026-10-21 20:36attempts
Vidya ternary-0.0106smoketernary 4L/64H polar dense, 3K updates, lr 4x, 1.9818 bpb, 3.90 MB, 13.64 M MAC/byte (96% ternary adds)2026-10-23 11:27attempts