karpathy
completestandardstep-109270, Alpha sm_86; greedy-pair prompts over 768 ids keep their last 768 (524 of 3,742)CORE 0.0578
2026-10-11 07:42 · max 500 examples/task · scorer v1 · bundle abc150c04fa3 · endpoint http://127.0.0.1:18045
| Task | Accuracy | Centered | Δ centered | n | Time |
|---|---|---|---|---|---|
| agi_eval_lsat_ar | 0.2304 | 0.0380 | — | 230 | 119s |
| arc_challenge | 0.2160 | -0.0453 | — | 500 | 238s |
| arc_easy | 0.3280 | 0.1040 | — | 500 | 234s |
| bigbench_cs_algorithms | 0.4060 | 0.4060 | — | 500 | 2371s |
| bigbench_dyck_languages | 0.0040 | 0.0040 | — | 500 | 2346s |
| bigbench_language_identification | 0.2760 | 0.2035 | — | 500 | 262s |
| bigbench_operators | 0.0571 | 0.0571 | — | 210 | 952s |
| bigbench_qa_wikidata | 0.0820 | 0.0820 | — | 500 | 2232s |
| bigbench_repeat_copy_logic | 0.0000 | 0.0000 | — | 32 | 142s |
| boolq | 0.4720 | -0.3895 | — | 500 | 254s |
| commonsense_qa | 0.2280 | 0.0350 | — | 500 | 241s |
| copa | 0.6200 | 0.2400 | — | 100 | 42s |
| coqa | 0.0600 | 0.0600 | — | 500 | 2246s |
| hellaswag | 0.2660 | 0.0213 | — | 500 | 365s |
| hellaswag_zeroshot | 0.2640 | 0.0187 | — | 500 | 206s |
| jeopardy | 0.0000 | 0.0000 | — | 500 | 2223s |
| lambada_openai | 0.2380 | 0.2380 | — | 500 | 2264s |
| openbook_qa | 0.2420 | -0.0107 | — | 500 | 214s |
| piqa | 0.5940 | 0.1880 | — | 500 | 223s |
| squad | 0.0060 | 0.0060 | — | 500 | 2431s |
| winograd | 0.5018 | 0.0037 | — | 273 | 138s |
| winogrande | 0.5060 | 0.0120 | — | 500 | 204s |
Every example's outcome (per-option losses, greedy matches, correctness) is stored with this attempt.