ASLA: Algorithm-Selection Leaderboard Audit

Small-compute scaling-law extrapolation is routinely used as a decision procedure: fit each training intervention on cheap runs, project to the target budget, ship the projected winner. ASLA audits that procedure. It treats intervention selection as a fixed-budget decision problem and measures, with known or measured ground truth, when extrapolation-based selection fails — noise close-calls, functional-form misspecification, and crossovers that happen after the largest fitting budget — and what better decision rules buy.

Everything runs offline on deterministic synthetic scenarios, on the ASLA-Bench problem generator, or on your own run tables converted to a canonical parquet schema. W&B harvesting is optional.

What ASLA provides

• Decision-level audit (asla audit): top-1 accuracy, regret, pairwise accuracy, rank correlations for each decision rule, with cell-wise seed bootstrap confidence intervals; seed-noise bands; under-seeded-cell warnings; statistical ties at the target (truth_ties); per-intervention fit diagnostics (R², RMSE, dof).
• FDR-controlled crossover detection: per-pair Welch tests at the target with Benjamini–Hochberg correction (crossovers_fdr); single-seed pairs are reported as untestable rather than significant.
• Misspecification-aware ensemble projection: power-law, saturating, and damped-power-law fits weighted by leave-largest-budget-out extrapolation loss, with an extrapolation reliability score ρ = cross-family disagreement ÷ seed-noise band. ρ ≫ 1 is the quantitative version of "the fit is blind here".
• Cost-aware sequential selection (racing): advance interventions up the budget ladder, eliminating by projected-interval dominance; every decision rule reports compute_spent, so rules are compared on the compute–regret Pareto frontier, not a single operating point.
• Conformal projection intervals and a deterministic conformal gate, plus an empirical calibration study of bootstrap vs. conformal coverage.
• ASLA-Bench (asla benchmark): parameterized problem families (close-call, late-crossover, saturating) with interpretable difficulty knobs and known ground truth. See docs/BENCHMARK.md.
• Markdown reports (asla report or asla audit --report).
• Theory notes — a detectability lower bound and racing correctness — in docs/THEORY.md.

Findings

Generated by scripts/render_readme_findings.py from committed JSON in results/; every number below is reproducible by re-running the corresponding script. Pre-registrations live in PREDICTIONS_TASK_*.md and were committed before the numbers they score.

1. On DataDecide, extrapolation never beat ranking at the top rung

Single-scale ranking at the largest fitted budget mis-selects 1.33% of pairs on DataDecide (c4_en_bits_per_token). Across 48 matched ladders in a specification curve, plain projection never beat it (max_excess_among_winning = 0.0). The excess tracks the lever arm, not the number of fitted budgets: Spearman +0.564 (p = 3.2e-28) against log lever arm, versus -0.136 against budget count.

Scope: this is measured on DataDecide C4 bits-per-token with a 1B target, across every design and estimator we tested. It is not a proof that extrapolation can never help, and the regime where it should help --- a lever arm large enough that no run near the target exists --- is exactly where the ground truth needed to check it does not exist either.

None of the alternatives we built beat a baseline that fits nothing and extrapolates nothing: better estimators (shared-exponent, empirical-Bayes shrinkage, ensemble, checkpoint-augmented), a better objective, and a better ladder. That is a statement about point estimates. Under the calibrated procedure (results/target_scoring/restated_claims.json), with recipes as a sample and corrected across the paper's 38 inferential claims, no C4 estimator comparison survives; ensemble is nominally worse than single-scale. Conditional on these 25 recipes, ensemble is worse after correction. The only claim that survives the paper-level correction: projection excess rises with log lever arm (expected-error scoring).

• Optimal allocation. Solving the cost-constrained transductive design and evaluating at matched compute leaves an excess over single-scale of +0.028 to +0.117 at the 1B target. Under the calibrated procedure the excess is nominally significant at 3 reduced budget fractions and survives correction at none; the optimised design never does better than single-scale.
• Design for decision = design for estimation. Under the linear model the target-gap variance is exactly twice the target-level variance, so the two objectives share an argmin. Measured: maximum cost-share difference 0.00058 against a pre-registered threshold of 0.02. Changing the design objective cannot change the design chosen, so it is not a lever.

2. Model error dominates seed noise, and is structured in compute

Residual scatter of cell means about a fitted power law is 2.90x the seed standard error, an 8.40x variance inflation, so the dominant term in projection error is model error that more seeds do not shrink. (Finding 3 below shows a large part of the out-of-range component is a ladder design confound rather than an inadequacy of the power law; this in-range scatter is not explained by that confound.) A single variance-inflation scalar cannot absorb it: phi varies 2.7-8.9x across ladders and under-predicts held-out structure by 12x.

Whether that error cancels in a pairwise gap is metric-dependent and predictable in advance from the PC1 loading CV, which separates the suites with no overlap: 0.145-0.219 for offset-type metrics against 1.370-2.153 for scaled-type ones.

3. A large part of the extrapolation error is a ladder confound

Projections overshoot the observed 1B target by +0.0877, with 24 of 25 recipes on the same side. Residual curvature is real and unanimous (quadratic coefficient signed 25/25, p = 6.0e-08; Spearman -0.887 against the overshoot) but does not explain it: taking the curvature hypothesis at face value removes only 22% of the absolute error.

The dominant source is a design confound. DataDecide's tokens-per-parameter holds near 100 up to 90M and then drifts to 85.0 at the 1B target, correlating with log compute at -0.80, so a compute-only fit attributes one effect to the other. Adding log(tokens/param) cuts absolute projection error by 60%.

The confound is common-mode: the two projection sets agree at Spearman 0.9977 (3.00% versus 2.00% mis-selection), so it shifts the level without reordering. Selection results are unaffected; coverage and centring results are what it explains.

4. Current confidence statements fail in two quantified ways

• Precisely wrong, not merely narrow. Bootstrap projection intervals cover the observed target 0% of the time (0/25) against a 90% nominal level. The band is 4.73x narrower than its own centring error. Conformal covers 88% by being 15x wider at the same centring, not by being better placed.
• A nominal guarantee that is not the achieved one. Nominal delta = 0.05 becomes an effective 0.500 under bootstrap widths.

5. Certification is free where it matters, and unavailable where it is tempting

A delta-PAC rule that abstains rather than guessing is cheap at the top of the ladder: at the 1B target with 3 seeds it certifies 292 of 300 pairs with 0 certified errors, declining only 1.7% of the calls the baseline gets right (single-scale mis-selection here is 0.78%).

The complement is the constraint. Across the audited leaderboards, 437 of 468 adjacent orderings (93.4%) are not resolvable at the seed budget actually used, and 28 are unidentifiable at any budget. A leaderboard that always returns a ranking is running a procedure with no correctness guarantee.

6. A reference-distribution defect in standard practice

With n seeds the standard deviation is estimated from n points, so the reference distribution is Student's t on 2n-2 degrees of freedom. Under a Bonferroni correction over 300 comparisons at delta = 0.05, the t quantile at 3 seeds is 13.65 against the normal's 3.76, a factor of 3.63x even in the equal-variance best case. With the Welch degrees of freedom measured per pair on DataDecide, the Gaussian threshold is anti-conservative by a median of 6.77x (3.69x to 18.27x, 10th to 90th percentile) at exactly the seed count leaderboards use. The best case closes to 1.04x only by 50 seeds. Anyone reporting significance on a 3-seed grid with a Gaussian threshold inherits this. See results/confidence/STUDENT_T_DEFECT.md.

Reproducing these numbers

Every document under results/ is rendered from committed JSON by a script in scripts/. To check that none has drifted from the results it claims to be generated from:

python scripts/verify_reproducibility.py --verbose

The studies themselves are re-runnable but slow (hours on a laptop); each driver takes --fast for a smoke test and --resume to reuse completed work. The JSON they produce is committed, so the documents and this section can be regenerated without re-running them.

Reading order

document	what it contains
NOVELTY_CHECK.md	positioning against transductive bandits, SL2, and misspecified BAI
FORMULATION.md	the problem as transductive pure exploration, with the h* recovery
results/allocation/RESULTS.md	optimal allocation scored against pre-registered P1-P5
results/abstention/RESULTS.md	abstention versus delta and budget, scored against A1-A5
results/curvature/RESULTS.md	the curvature gate and the tokens-per-parameter confound, C1-C7
results/confidence/	the Student-t defect and the two confidence failures
results/consolidated/	the measurement-artifact practice and model-error structure

Install

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[test,figures,dev]"
command -v asla

Keep the virtual environment active while using the CLI. command -v asla should point inside this repository's .venv. On some Macs a bare asla resolves to PlatformIO, not this package.

Optional extras:

python -m pip install -e ".[figures,wandb]"

Demo

asla demo --scenario saturation_crossover --estimand single_design_seed_sensitivity
asla demo --scenario noise_close_call --estimand pairwise_decisions

Default demo and audit runs use paper-grade counts (n_boot=1000, n_trials=500). For a quick local smoke test add --fast.

The saturation demo reports a wrong projected winner, positive regret, a detected crossover, an explicit fit is blind message, and large ensemble reliability scores ρ for the saturating pair. The noise close-call demo shows the gate and the race reducing mean regret and wrong-pick rate versus plain projection, with compute costs alongside. Demo and audit reports compare the scaling-law projection ranker, the ensemble ranker, and the largest-single-scale baseline with bootstrap confidence intervals.

Validate and audit real runs

Canonical run columns:

• intervention string
• intervention_class string
• compute float
• seed integer
• bpb float
• optional downstream float
• optional params_n float for Chinchilla two-axis fits
• optional tokens_d float for Chinchilla two-axis fits
• optional metric_name string (one value per table) naming what bpb holds when it is not C4-EN bits per byte
• optional nullable tuning_quality string recording how hyperparameters were tuned at that run's scale

asla validate --runs runs.parquet
asla audit --runs runs.parquet --target 64 --budgets 1 2 4 8 --intermediate-budget 16 --estimand pairwise_decisions --out results/audit.json --report
asla figures --runs runs.parquet --target 64 --budgets 1 2 4 8 --intermediate-budget 16 --out results/figures

--estimand is required. Choose single_design_seed_sensitivity for one complete-candidate-set decision or pairwise_decisions for the finite-table average over every unordered intervention pair. Their assumptions and licensed claims are documented in NOTES_ESTIMAND.md; the software does not select a paper headline. Pass fitting budgets explicitly for paper results. A reserved --intermediate-budget is excluded from fitting and makes the top-1-only gate available.

Useful audit flags:

• --weighted — weight fits by per-cell seed standard errors.
• --crossover-q — BH-FDR level for crossover significance (default 0.05).
• --report — write a markdown report next to --out.
• --budgets — explicit projection-fitting budgets.
• --intermediate-budget — held-out gate exploration budget, never fit by the baseline projection.
• --fit-form compute_power_law (leaderboard audits) or --fit-form chinchilla (controlled grids with params_n/tokens_d; fails loudly if those columns are missing).

Fits are unit-invariant: compute, params_n, and tokens_d may be raw counts (FLOPs, parameters, tokens) or consistent relative units.

Benchmark and paper experiments

asla benchmark --fast --out results/benchmark        # difficulty sweep, 4 rules
python scripts/run_pareto_study.py --fast            # compute-regret frontier
python scripts/run_calibration_study.py --fast       # interval coverage
python scripts/run_paper_experiments.py --estimand pairwise_decisions --fast  # explicit human-selected estimand

Drop --fast for paper-grade counts. See docs/BENCHMARK.md.

Public data: DataDecide and Fantastic Optimizers

The first real measurements use released evaluation tables (never model weights). Sizes are checked through the HuggingFace API before download.

python -m pip install -e ".[harvest]"
asla harvest-datadecide --metric c4_en_bits_per_token --out data/datadecide_runs.parquet
asla harvest-datadecide --metric olmes_macro_error --out data/datadecide_runs_olmes_macro_error.parquet
asla harvest-fantastic-optimizers --out-dir data
asla harvest-signal-and-noise --out data/signal_and_noise_datadecide_c4_bpb.parquet
asla validate-known-answer --runs data/datadecide_runs_olmes_macro_error.parquet
asla resolution --entries leaderboard.json --sigma 0.0014  # which orderings are resolvable?
python scripts/run_first_audit.py            # writes results/first_audit/ and FIRST_AUDIT.md
python scripts/run_signal_and_noise_check.py # writes results/signal_and_noise/ and SIGNAL_AND_NOISE.md

Harvested tables carry metric_name (what bpb holds: DataDecide has no bits-per-byte metric, so rows are labelled c4_en_bits_per_token, olmes_macro_error, ...) and a nullable tuning_quality. Raw artifacts are cached under data/raw/ (git-ignored). See KNOWN_ANSWER.md, FIRST_AUDIT.md, AUDIT_ADVERSARIAL.md, NOTES_FIT_BOUNDS_DEFECT.md, PREDICTIONS_TASK_B.md, NOTES_TUNING_CONFOUND.md, SIGNAL_AND_NOISE.md, and RELATED_WORK.md.

No-W&B data collection

On a laptop, create an HPC checklist:

python scripts/make_run_manifest.py \
  --interventions data/interventions_template.csv \
  --budgets data/budgets_template.csv \
  --seeds 3 \
  --out data/run_manifest.csv

Edit data/interventions_template.csv and data/budgets_template.csv before using the manifest for real training. The generated manifest is not audit data; it is a checklist for runs that still need measured BPB values.

After the runs finish and the BPB values are filled into a CSV:

python scripts/finalize_run_manifest.py \
  --manifest data/run_manifest.csv \
  --out-csv data/runs_template.csv \
  --out-parquet runs.parquet
python scripts/check_runs_coverage.py --csv data/runs_template.csv --target TARGET_COMPUTE
asla validate --runs runs.parquet
asla audit --runs runs.parquet --target TARGET_COMPUTE --budgets FIT_BUDGETS --intermediate-budget INTERMEDIATE_COMPUTE --estimand ESTIMAND --fast --out runs_audit.json

The finalizer refuses pending rows and blank BPB values, so incomplete manifests cannot silently become audit data.

The optional HPC scaffold is in hpc/:

• hpc/train_command.template: wrapper command used by the row runner.
• hpc/site_train_command.template: calls hpc/train_and_eval.py, which uses ASLA_SITE_COMMAND_TEMPLATE.
• hpc/site_command.template: local template that calls the script named by ASLA_REAL_TRAIN_EVAL.
• hpc/slurm_array_template.sh: SLURM array template, dry-run by default.
• hpc/ADAPTATION_CHECKLIST.md: step-by-step cluster adaptation guide.

Dry-run one manifest row:

python scripts/run_one_manifest_row.py \
  --manifest data/run_manifest.csv \
  --row 1 \
  --index-base 1 \
  --command-template-file hpc/train_command.template \
  --output-dir results/hpc

Collect per-run result.json files after HPC jobs finish:

python scripts/collect_results.py \
  --manifest data/run_manifest.csv \
  --results-dir results/hpc \
  --out data/run_manifest.csv

Run python scripts/hpc_preflight.py before submission. It verifies that the SLURM array covers every manifest row exactly once and smoke-tests only the plumbing in a temporary directory. Result JSON must contain run_id, intervention, compute, seed, finite positive measured bpb, and completed status. Collection rejects identity mismatches. Existing metrics/results are never reused; a deliberate reviewed retry requires ASLA_OVERWRITE_OUTPUT=1.

Harvest W&B runs

ASLA never guesses W&B field names. First inspect a finished run:

asla harvest --discover --entity-project ENTITY/PROJECT

Then provide an explicit JSON field map from schema column to logged W&B key:

{
  "intervention": "config.recipe_name",
  "intervention_class": "config.recipe_class",
  "compute": "summary.train_flops",
  "seed": "config.seed",
  "bpb": "summary.eval/c4_en_bpb"
}

config. and summary. prefixes select the W&B store explicitly and support nested dotted paths. An unprefixed key is accepted only when it resolves in exactly one store; ambiguity is an error.

Harvest:

asla harvest --entity-project ENTITY/PROJECT --field-map field_map.json --out runs.parquet

Development

pytest        # test suite
ruff check .  # lint (also run in CI)
mypy asla     # type check (also run in CI)

The enhancement roadmap and its status live in PLAN.md. CI runs ruff, mypy asla, and pytest on Python 3.10 and 3.12.