# What conformal calibration costs at regime onset Replication package for the paper *What conformal calibration costs at regime onset: coverage debt, width debt, and a monitored hybrid for electricity price intervals*. Everything reported in the paper can be reproduced from this repository. There is one script, `code/08_check/check_paper_numbers.py`, that recomputes every headline number and prints PASS or FAIL against the value printed in the paper. Run that first if you only want to know whether the numbers hold. ## Quick start ```bash python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt # Verify the paper against the shipped intermediate results (about 30 seconds) python code/08_check/check_paper_numbers.py # Or rebuild everything from the raw panels (about 4 hours, mostly LEAR) bash run_all.sh ``` All scripts are written to be run **from the repository root**, not from the directory they live in. Paths inside the code are relative to that root. ## What is in the box ``` data/ four input panels, enough to rebuild every result code/ the pipeline, in the order it runs output/ intermediate and final results, plus the five figures paper/ the manuscript and the script that builds it ``` `output/` is committed. That is deliberate: it means a reader can check any number in the paper without waiting four hours for LEAR to refit, and it means the checking script has something to check on a fresh clone. ## Data Four files in `data/` drive everything. | File | Contents | Source | |---|---|---| | `entsoe_prices_hourly_2015_2026.parquet` | Hourly day ahead prices, ten European bidding zones, 2015 to 2026 | ENTSO-E Transparency Platform | | `entsoe_exog_hourly.parquet` | Hourly day ahead load, wind, solar and residual load forecasts for the same ten zones | ENTSO-E Transparency Platform | | `entsoe_severity_hourly.parquet` | The same residual load forecasts pivoted to one column per zone, used by the U shape and orthogonality analyses | ENTSO-E Transparency Platform | | `ercot_HBAVG_hourly_2011_2023.parquet` | Hourly day ahead settlement point prices, ERCOT hub average, 2011 to 2023 | ERCOT public data archive | `entsoe_exog_hourly.parquet` replaces the ten per zone CSV files the code was originally written against. It stores the same four columns at full double precision and is verified byte identical to those CSVs for all ten zones; the loader in `code/exog.py` presents it to the scripts in the original shape, so results are unchanged. The CSVs themselves are not committed because they are 60 MB of redundancy. The scripts in `code/00_download/` rebuild the panels from the original sources. The ENTSO-E ones need a free API token from the Transparency Platform; the ERCOT and Ember ones need nothing. You do not need any of this to reproduce the paper: the panels are already here. See `DATA.md` for provenance, licensing and coverage notes. ## Where each exhibit comes from Every number in the paper traces to one file in `output/`, and every file in `output/` is written by one script. Run the script from the repository root. | Exhibit | Produced by | Reads from | Written to | |---|---|---|---| | Table 3 | hand written in the manuscript | | | | Table 4, Figure 3 | `code/02_calibration/c1_baselines.py DE` | `output/lear/lear_DE.parquet` | `output/zoo_DE.csv`, `output/zoo_bands_DE.npz` | | Table 5 | `code/02_calibration/c2_tests.py DE` | `output/zoo_bands_DE.npz` | `output/tests_DE.csv` | | Table 6 | `code/03_episodes/e3_episodes_all.py`, then `code/03_episodes/t1_clustering.py` | prices, `output/lear/` | `output/episodes_all.csv`, `output/episodes_clustered.csv` | | Table 7 | `code/05_verification/v1_fixes.py` | `output/episodes_all.csv`, `output/lear/` | `output/r1_forecaster_split.csv` | | Table 8, Figure 5 left | `code/04_economic/r1_economic.py DE` | `output/zoo_bands_DE.npz` | `output/economic_DE.csv` | | Table 9, Figure 5 right | `code/04_economic/t3_definitive.py` | `output/lear/`, prices | `output/final_main.csv` | | Table 10 | `code/05_verification/w1_checks.py` | `output/lear/lear_ERCOT.parquet` | `output/texas_baseline.csv` | | Figures 1 and 2 | `code/06_figures/u1_figs.py` | `output/lear/` | `output/f1.png`, `output/f2.png` | | Figure 4 | `code/06_figures/u1_figs.py` | `output/episodes_all.csv` | `output/f4.png` | | Table A1 | `code/02_calibration/c3_sensitivity.py DE` | `output/lear/lear_DE.parquet` | printed to stdout | | Table A2 | `code/03_episodes/r3_episode_defs.py` | prices | `output/episode_definitions.csv` | | Table A3 | `code/05_verification/d2_verify_dnn.py` | `output/dnn/`, `output/lear/` | `output/dnn_vs_lear.csv` | | Table A4 | `code/04_economic/s3_monitor4.py` | `output/lear/lear_DE.parquet` | `output/monitor4.csv` | | Table A5 | `code/04_economic/s2_hybrid_robust.py` | `output/lear/lear_DE.parquet` | `output/hybrid_robust.csv` | | Table A6 | `code/03_episodes/r4_orthogonality.py` | prices, exogenous panel | `output/orthogonality.csv` | | Table A7 | `code/03_episodes/a7_ushape_all.py` | prices, exogenous panel | `output/ushape_all_markets.csv` | | leakage audit and placebo test, Section 5.5 | `code/05_verification/c5_leakage_placebo.py DE` | `output/lear/lear_DE.parquet` | `output/placebo_DE.csv` | | stationary bootstrap interval, Section 5.5 | `code/05_verification/w1_checks.py` | `output/zoo_bands_DE.npz` | printed to stdout | Nothing in the paper comes from `code/09_exploratory/`. Those five scripts are an earlier stage, kept because they document how two of the reported negative results were reached. Their intermediates are not committed. See the README in that directory. ## Pipeline order `run_all.sh` executes this order. The forecaster stage dominates the runtime. 1. **Forecasters.** `b1_lear.py {DE,FR,DK1}` fits LEAR exactly as specified by Lago et al. (2021): 247 regressors, per hour LASSO with LARS and AIC, an asinh variance stabilising transform, a 1095 day rolling window and weekly recalibration. `e1_lear_price.py` fits the 103 regressor price only variant for ERCOT, where no public day ahead load and wind forecasts cover the whole sample. `d1_dnn.py DE` fits the feed forward network used as a dose response check. Budget roughly 45 minutes per European market and about 20 for ERCOT. 2. **Calibration.** `c1_baselines.py` runs the nine calibration schemes, `c2_tests.py` the Kupiec, Christoffersen and dynamic quantile tests with the Benjamini and Hochberg adjustment over all 108 tests, `c3_sensitivity.py` the design sensitivity grid. 3. **Episodes.** the automatic detector, the eleven market replication, the clustering of 44 episodes into 21 calendar events, the two alternative episode definitions and the orthogonality check. 4. **Economics and the hybrid.** the currency pricing, the monitored hybrid, the monitor and constant sensitivity grids, and the single consistent run that produces Table 9. 5. **Verification.** the LEAR benchmark check, the network check, the leakage audit and placebo test, the forecaster split, and the Texas monitor state trace behind Table 10. 6. **Figures**, then **the manuscript**. ## Reproducibility notes Random seeds are fixed wherever randomness enters: the cluster bootstrap over 21 events, the stationary bootstrap over rolling coverage, the placebo onsets, and the network initialisation. The two bootstraps use 20000 and 2000 replications respectively. Wilcoxon signed rank tests of coverage against the nominal level are one sided against under coverage throughout. The paired test between the two forecasters in Table 7 is two sided. This is stated in the manuscript and asserted in the checking script. Two checks were carried out on this package rather than asserted. First, `check_paper_numbers.py` was run on a fresh clone, both against the committed intermediates and after regenerating them: 61 checks, no failures. Second, LEAR was refitted for Germany from `data/entsoe_exog_hourly.parquet` and the resulting forecasts are bit identical to the committed `output/lear/lear_DE.parquet`, which is what establishes that replacing the ten per zone CSV files with the packaged panel changes nothing. Two steps were not rerun end to end here for time: LEAR for France and western Denmark, and the feed forward network. They use the same code path as the German fit that was verified. ## Software Python 3.11, with the versions pinned in `requirements.txt`. The manuscript is built with Node and `docx` (`code/07_paper/build_paper.js`); that step is not needed to reproduce any result. ## Licence Code is released under the MIT licence (`LICENSE`). The data panels remain subject to the terms of their original providers, set out in `DATA.md`. ## Citation See `CITATION.cff`. # ConformalOnset