i'm turning language models into superforecasters. on the side i work on evals, rl and inference; these are the questions i've been testing.
each result links to committed evidence. more experiments and write-ups at alpcetin.com.
-
tithe — does instruction tuning trade calibration for truthfulness? on the same 779 questions with shared prompts, 95% intervals resolve ece worsening in 8 of 9 label-scored pairs and none of the full-string-scored pairs. write-up
details
it depends on how i score the answer: on the same 779 questions and answers, instruction tuning worsens ece in 8 of 9 pairs when i score the a/b label, but improves it in 6 and worsens it in none when i score the two answers as full strings (shared prompts, 95% intervals). the prompts and scored text still differ, so this doesn't show what causes the flip.
-
rayleigh — how many questions separate two models? in simulation, planning from a 128-item pilot's observed gap reached 15–68% power instead of 80%. calculator
details
in simulations where i know the truth, planning from a 128-item pilot's observed gap reached 15–68% power instead of 80%, while a 95% upper bound on the discordance rate reached 84–95% at 1.2–2.4 times the median plug-in budget. on fresh items, 23 of 27 planned comparisons between six small models detected a difference, 17 after holm over all 30 planned (nominal tests, mostly large gaps).
-
mendel — how many training seeds are enough? in simulation, a fixed-effect plan with a variance allowance kept power at 79.3–83.3% for an 80% target. write-up
details
in a known-truth simulation, fixing the effect in advance and allowing for noise in the pilot variance kept power at 79.3–83.3% for an 80% target, while planning from the pilot's observed gap gave 36.3–80.5%. my fresh-seed ppo plans used the observed gap and detected 139/307 differences (detection frequency, not power), and cartpole's same-variant null controls rejected 5/36, which my diagnostics could not explain.
-
weeble — does shove training help a robot stay up? go1's 50%-survival impulse improves by 8.8 N·s, and a robustness atlas tests the 12 released policies over 21,600 episodes, all in simulation. demo
details
go1's 50%-survival impulse improves by 8.8 N·s with three seeds per arm. in a robustness atlas of 21,600 episodes over the 12 released policies, every dog seed stayed up in at least 90% of episodes carrying 40% of its body mass, but none did with 40 ms of sensor delay. it's all simulation, not hardware, and ci rebuilds the report and atlas byte for byte on pull requests.
-
coldstart — how soon can a cold process produce a policy? ars qualifies cartpole in 1.08 s median on my laptop, including imports and evaluation. write-up
details
ars qualifies cartpole in 1.08 s median on my laptop, including imports and evaluation (reused eval seeds). i wrote a c environment batch that matches gymnasium's resets bit for bit on 1,010,006 seeds per task and steps cartpole environments 7.7–13× faster than my numpy batch, but the environment is only 22–27% of an ars or cem run, so even a free environment would cut a run by at most about a quarter. i have not timed the c batch inside qualification.
-
quartet — is the tuner limited by its kernel bank? even the best action in my original h100 archive takes 1.608–1.609× torch's latency. write-up
details
in my original h100 archive, the earlier-bank choice adds at most 0.14% geometric-mean latency over the later bank's in-sample best action, and even that best action's latency is 1.608–1.609× torch's. the new triton actions reduce geometric-mean latency by 27.0% versus the old bank, but still take 1.174× torch's time. these are descriptive medians from one h100 collection; the same collector now runs on any cuda gpu under its own run id, so i can repeat it.
-
temperament — can rotation hurt fp8 attention? paired against plain fp8, rotation costs qwen2.5-7b-instruct 18.20 points on hellaswag but gains 5.10 on mmlu. write-up
details
paired on the same items, rotating queries and keys costs qwen2.5-7b-instruct 18.20 points on hellaswag against plain fp8 but gains 5.10 on mmlu, and adding rotation to already-centered keys changes hellaswag by +0.20 points (95% interval −0.25 to +0.65). this is h100 batch scoring, with exploratory intervals and no multiple-comparison correction.
-
abacus — how fast can my own cpu decoder run qwen2.5-0.5b-instruct? it reaches 1.090–1.531× llama.cpp q8_0's paired throughput in all six pre-specified cells on one cloud host, not my laptop. write-up
details
it beat llama.cpp q8_0 in all six pre-specified cells, at 1.090–1.531× paired throughput, on one virtual cloud host with cpu binding, not my laptop. i also made generation stop at end of turn, specialized the batched kernels (speed not measured), added a one-command
make demoand an offline corpus step, and checked the shipped weights on six authored prompts: they pick the unquantized model's top token at 462 of 476 positions (agreement, not task accuracy). -
doomformer — can a doom map do neural arithmetic? fuzzing compared 1,987,632 values with no integer-reference mismatch (testing, not a proof), and the museum map lets you rewire one weight inside a running level. films
details
fuzzing 15,000 nets at the range limits compared 1,987,632 values with no integer-reference mismatch, and readback is now guarded, so a dropped page can't pass with stale values (testing, not a compiler proof). in the museum map, 136 voodoo dolls run a small hand-written network that reads a digit from floor pads, and rewiring one weight inside the running level turns a 7 into a 1, with all 148 values exact; ci builds the wad so you can download it and play without python (so far i've run it only in vizdoom).
over 80 merged pull requests, including vllm beam search, cupy chirp, sglang docs, torchtitan streaming state and sentence-transformers hard negatives. my pytorch dynamo fix also landed on main.