CORE Bench
The DCLM CORE metric: 22 in-context-learning tasks, each accuracy centered against random chance and averaged. The same number nanochat speedruns chase — GPT-2 1.6B sits at 0.2565. Scores are comparable only within a tier (the per-task example budget): smoke 50, standard 500, full everything. How to hook a model up.
paste into your coding agent — it implements the one scoring endpoint your model needs
CORE over timeone measurement scale at a time —
GPT-2 124M (reference)karpathyVidyaVidya ternary
Standings
Each model's newest complete attempt, per tier.
| Model | CORE | Tier | Checkpoint | When | History |
|---|---|---|---|---|---|
| GPT-2 124M (reference) | 0.0796 | smoke | parity gate | 2026-08-20 08:36 | attempts |
| karpathy | 0.0610 | smoke | step-109270, Alpha sm_86; greedy-pair prompts over 768 ids keep their last 768 | 2026-10-11 07:01 | attempts |
| karpathy | 0.0578 | standard | step-109270, Alpha sm_86; greedy-pair prompts over 768 ids keep their last 768 (524 of 3,742) | 2026-10-11 07:42 | attempts |
| Vidya | 0.0436 | smoke | runs/polar-dense: polar dense 4L/32H, 30000 updates, 1.8469 bpb, 3.62 M MAC/byte, 2.14e+16 FLOP | 2026-10-21 20:21 | attempts |
| Vidya | 0.0182 | standard | runs/polar-dense: polar dense 4L/32H, 30000 updates (983 MB), 1.8469 bpb, 3.62 M MAC/byte, 2.14e+16 FLOP | 2026-10-21 20:36 | attempts |
| Vidya ternary | -0.0106 | smoke | ternary 4L/64H polar dense, 3K updates, lr 4x, 1.9818 bpb, 3.90 MB, 13.64 M MAC/byte (96% ternary adds) | 2026-10-23 11:27 | attempts |