evals

karpathy

completestandardstep-109270, Alpha sm_86; greedy-pair prompts over 768 ids keep their last 768 (524 of 3,742)

CORE 0.0578

2026-10-11 07:42 · max 500 examples/task · scorer v1 · bundle abc150c04fa3 · endpoint http://127.0.0.1:18045

TaskAccuracyCenteredΔ centerednTime
agi_eval_lsat_ar0.23040.0380—230119s
arc_challenge0.2160-0.0453—500238s
arc_easy0.32800.1040—500234s
bigbench_cs_algorithms0.40600.4060—5002371s
bigbench_dyck_languages0.00400.0040—5002346s
bigbench_language_identification0.27600.2035—500262s
bigbench_operators0.05710.0571—210952s
bigbench_qa_wikidata0.08200.0820—5002232s
bigbench_repeat_copy_logic0.00000.0000—32142s
boolq0.4720-0.3895—500254s
commonsense_qa0.22800.0350—500241s
copa0.62000.2400—10042s
coqa0.06000.0600—5002246s
hellaswag0.26600.0213—500365s
hellaswag_zeroshot0.26400.0187—500206s
jeopardy0.00000.0000—5002223s
lambada_openai0.23800.2380—5002264s
openbook_qa0.2420-0.0107—500214s
piqa0.59400.1880—500223s
squad0.00600.0060—5002431s
winograd0.50180.0037—273138s
winogrande0.50600.0120—500204s

Every example's outcome (per-option losses, greedy matches, correctness) is stored with this attempt.