evals

karpathy

completesmokestep-109270, Alpha sm_86; greedy-pair prompts over 768 ids keep their last 768

CORE 0.0610

2026-10-11 07:01 · max 50 examples/task · scorer v1 · bundle abc150c04fa3 · endpoint http://127.0.0.1:18045

TaskAccuracyCenteredΔ centerednTime
agi_eval_lsat_ar0.24000.0500—5028s
arc_challenge0.2400-0.0133—5026s
arc_easy0.30000.0667—5026s
bigbench_cs_algorithms0.38000.3800—50225s
bigbench_dyck_languages0.00000.0000—50221s
bigbench_language_identification0.18000.0979—5028s
bigbench_operators0.00000.0000—50231s
bigbench_qa_wikidata0.10000.1000—50220s
bigbench_repeat_copy_logic0.00000.0000—32143s
boolq0.62000.0000—5028s
commonsense_qa0.34000.1750—5025s
copa0.62000.2400—5022s
coqa0.04000.0400—50222s
hellaswag0.28000.0400—5029s
hellaswag_zeroshot0.2400-0.0133—5027s
jeopardy0.00000.0000—50221s
lambada_openai0.26000.2600—50219s
openbook_qa0.2200-0.0400—5023s
piqa0.52000.0400—5024s
squad0.00000.0000—50225s
winograd0.4800-0.0400—5024s
winogrande0.4800-0.0400—5024s

Every example's outcome (per-option losses, greedy matches, correctness) is stored with this attempt.