Skip to content

About

Statistical analysis pipeline for Sugar Sugar study results (H1-H5 per study design)

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

sugar-data-processing

Statistical analysis library for Sugar Sugar study results.

It implements Section 7 (Statistical Analysis Plan) of the study design (Human Prediction of Next-Hour Glucose from Prior CGM Context).

The package is split into five stages so each part can be imported as a library module, exercised in a Jupyter notebook, or run end-to-end via the CLI.

Stage Module What it does
1. Data gathering sugar_data_processing.gathering Load prediction_statistics.csv, explode rounds, build person-level table
2. Data verification sugar_data_processing.verification Schema / structural checks + demographic & metric quality flags
3. Statistical tests sugar_data_processing.statistics H1–H5 (§7.3–7.4) on person-level MAE
4. Data comparison sugar_data_processing.comparison Human MAE vs GlucoBench / literature bands (§7.5)
5. Output sugar_data_processing.output PNG figures + one merged markdown / JSON / explorer.html
6. AI scoring sugar_data_processing.ai Rebuild the 3-hour game window, score models, fold into the same report

Orchestration: sugar_data_processing.pipeline.run_analysis.

Hypotheses

Hypothesis Question Test path
H1 PwD vs non-PwD MAE (category first, then generic vs own) Shapiro → t-test / Mann–Whitney
H2 CGM vs non-CGM MAE (category first, then generic vs own) Shapiro → t-test / Mann–Whitney
H3 Diabetes duration (months) vs MAE, generic and own Pearson / Spearman + log exploratory
H4 CGM experience (months) vs MAE, generic and own Pearson / Spearman + log exploratory
H5 Own vs generic MAE (paired) Shapiro on diffs → paired t / Wilcoxon
H6 Human vs baseline models Same 3-hour windows; persistence / linear / SugarOne W5.3 and SugarOne R.28 (ONNX, downloaded with HF_TOKEN) / the other ONNX models below. A local SugarOne 1.0 checkpoint is used only when the W5.3 bundle is absent.

The dashboard uses the model names the paper will use: SugarOne is the W5.3 forecast ONNX (GlucoseDao/FORECAST-SugarOne-W5.3-10.7h-mard_loss-mv1-s42), SugarOne R.28 is the dual-mode arm scored with nothing after the origin, CitricSugar is CITRAS-FM W38.2, Kronsugar is KronLSTM R.80 and Sugarstral is Timestral-28M R.81. On the 9 November 2026 game windows W5.3's person-level MAE was 14.37 mg/dL and R.28's was 21.47 mg/dL, so W5.3 is the primary SugarOne. Both arms are scored, and a publish is refused if either is missing. CITRAS-FM W38.7 is not scored: on the game windows its MAE was 15.07 mg/dL against W38.2's 14.97 mg/dL. GluMind is not either: on the joined Loop + AI-READI test set SugarOne 1.0 has an MAE of 12.40 mg/dL against GluMind's 12.73. The game has no insulin or carbs, so the forecast SugarOne sees those channels as zero.

Person-level MAE (mean of round MAEs) is the analysis unit so repeated rounds from the same participant do not inflate degrees of freedom.

Primary H1/H2 comparisons use two-sided α = 0.025 (Bonferroni for two primary hypotheses). Other layers are exploratory at nominal α = 0.05. See corrected recruitment planning and the dated data assessment. Sample-size targets are conditional on effect size, variance, group balance, and eligibility; the committed synthetic fixture cannot establish those quantities for the study.

The published production briefing summarizes the 24 October 2026 audit in plain language. Production records run through 9 October: 207 runs, 184 browser IDs, 155 linked identity groups and 109 primary-eligible identities (104 in the stricter quality sensitivity). The revised operational target is 320 eligible for a 3.5 mg/dL effect with planning SD 10, or 250 eligible for a 4 mg/dL effect. See the linked analysis plan for group quotas, recruitment attrition and assumptions.

To regenerate the audit and briefing from the local scrubbed, identity-linked production CSV without model downloads:

PYTHONPATH=src .venv/bin/python scripts/assess_production.py

An optional participant_key column supplies pseudonymous identity linkage prepared before contact scrubbing. When present, the loader uses it as the analysis identity and preserves original_study_id; use use_participant_keys=False only for the explicit study-ID sensitivity. Without linkage, study-ID counts can overstate independent participants.

What data it uses

Input Path / source Role
Real export data/raw/prediction_statistics.csv (gitignored) sugar-sugar app statistics export
Synthetic fixture data/fixtures/synthetic_prediction_statistics.csv Committed demo cohort sized for H1–H5
CLI override --csv /path/to/prediction_statistics.csv Explicit file, any location

Required columns (enforced on load + re-checked in verification):

study_id, run_id, timestamp, format, is_example_data, uses_cgm, cgm_duration_years, diabetic, diabetes_duration, rounds_played, overall_mae_mgdl, overall_rmse_mgdl, overall_mape_pct, per_round_metrics

Optional current-app columns (kept when present):

round_context, generic_intervention, challenge_unknown, challenge_unknown_pct, paper_mention, paper_full_name, diabetic_type, predicted_values, real_values, prediction_times

cgm_duration_years may be a bare year (legacy) or value,unit (6,months). Per-round is_example_data / data_source_name (in per_round_metrics or round_context) decide generic vs own — the run-level source is only the last dataset of that session. Email / location are dropped; personal upload filenames are redacted to own_upload.

Analysis population (§7.2)

  • Primary (H1–H4): ≥6 generic segments (format A, or odd rounds in mixed C)
  • Own-data: ≥6 own segments (format B, or even rounds in mixed C)
  • H5: participants meeting both thresholds

Layout

src/sugar_data_processing/
  gathering/       # stage 1 — load CSV, explode rounds, participant table
  verification/    # stage 2 — schema checks + quality flags
  statistics/      # stage 3 — H1–H5 tests + effect sizes
  comparison/      # stage 4 — GlucoBench / literature bands
  output/          # stage 5 — plots + human markdown/JSON/HTML
  ai/              # sequence export + model ingest + AI report
  fixtures/        # synthetic CSV generator
  pipeline.py      # run_analysis() wires stages 1→5 (+ AI export)
  cli.py           # Typer entry point
notebooks/
  study_analysis.ipynb   # thin walkthrough over the library (same paths as CLI)
data/
  raw/             # drop real prediction_statistics.csv here (gitignored)
  anonymized_raw/  # verifier copy of that CSV (gitignored; see below)
  fixtures/        # committed synthetic demo data
  processed/       # parquet/csv intermediates (written by output stage)
output/
  figures/         # PNGs
  reports/         # analysis_report.md + explorer.html + milestone.html
tests/             # pytest (real synthetic data, no mocks)
docs/
  analysis-plan.md # study design §7 → module map

Setup

uv sync

For the notebook (dev group includes Jupyter):

uv sync --group dev

How to run the whole thing

sdp is the short CLI name (uv run sugar-data-processing is the same).

Online study stats (what you usually want)

The public site does not serve the research CSV. Pull it over SSH, scrub contact columns into data/raw/ (gitignored), then run H1–H5.

Once per machine

uv sync
cp .env.template .env

Edit .env (never commit it):

  • SUGAR_REMOTE — the same user@host:/path you already ssh to (the CSV, or the data/input directory that contains it)
  • SUGAR_SSH_IDENTITY — the same key you pass to ssh -i, if you use one

Each time you want fresh online numbers

uv run sdp fetch --analyze

That is the whole online path: scp → blank email / location / paper_full_name → write data/raw/prediction_statistics.csv → reports. What that file contains is below.

Pull only (no tests):

uv run sdp fetch

Re-run H1–H5 on the last pulled CSV (no SSH):

uv run sdp analyze

fetch also looks for prediction_statistics_ai.csv in the same directory as the human file. Keep those two files together and run them as a pair: analyze reads the human path you pass, then loads prediction_statistics_ai.csv from that same folder. In-game rows become in_place and that model is the primary AI comparison. A missing companion is a note; post-factum scoring still runs.

What is extracted from the server

uv run sdp fetch copies one file, prediction_statistics.csv, and writes it to data/raw/prediction_statistics.csv. Consent records, the email-to-study-id map, and the CGM files participants uploaded stay on the server.

On the way in, three columns are blanked:

Column After fetch
email empty
location empty
paper_full_name empty

Everything else in that export is kept, because the tests read it:

Kept What it is
study_id, run_id, timestamp, format, rounds_played which session
age, gender, diabetic, diabetic_type, diabetes_duration, uses_cgm, cgm_duration_years inputs to H1–H4
predicted_values, real_values, prediction_times, per_round_metrics, round_context, the overall MAE / RMSE / MAPE columns the game window and the errors
data_source_name a public corpus name (D1NAMO-…, BIGIDEAS-…) or the participant's own upload filename
challenge_unknown, generic_intervention, paper_mention study flags. The paper's name is the blanked column above

The upload filename is the piece that can still identify someone: it can be a person's name or a Nightscout host, and it is repeated inside per_round_metrics and round_context. It remains in data/raw/. Loading the CSV for a report replaces those filenames with own_upload, so the HTML does not show them. Glucose values and the demographic columns are the health data the statistics are computed from.

uv run sdp anonymize writes a second copy for evaluators. Same rows and the same study_id and metrics. Corpus filenames stay. Each person gets one stand-in address (email1@gmail.com, email2@gmail.com, …). Upload filenames become initials (AL.csv, and AL2.csv when those initials are already used).

How to run with an anonymized file

Give evaluators this repo plus the anonymized CSVs. study_id is unchanged, so the HTML is the same analysis. They put both files in one folder:

data/anonymized_raw/prediction_statistics.csv
data/anonymized_raw/prediction_statistics_ai.csv

Then:

uv sync
uv run sdp analyze --csv data/anonymized_raw/prediction_statistics.csv

analyze reads the human file you pass and the AI file sitting next to it. Do not run the two CSVs as separate commands.

Someone who already has the anonymized human file can drop a later prediction_statistics_ai.csv in that same folder and re-run the same analyze --csv line. The join is study_id / run_id. Human-only HTML still builds if the AI file is missing; H1–H5 do not need it.

Pass --csv. A plain uv run sdp analyze reads data/raw/prediction_statistics.csv when that file exists. Reports land in output/reports/; explorer.html is the full page, milestone.html is the human page. See Where the numbers are.

Build the files you send (from the last fetch). The AI companion is copied when it sits next to the raw human CSV:

uv run sdp anonymize

Synthetic fixture (no credentials)

uv run sdp make-fixture
uv run sdp analyze --fixture

Local sugar-sugar checkout (not the live box)

uv run sdp fetch --sibling --analyze

Or:

uv run sdp analyze --csv ../sugar-sugar/data/input/prediction_statistics.csv

Resolution order for analyze without --csv / --fixture:

  1. data/raw/prediction_statistics.csv if present
  2. else fixture (with a yellow warning)

Where the numbers are

File What it is
output/reports/analysis_report.md Merged report: human H1–H5, then same-person human vs AI by task
output/reports/explorer.html Tabs: Overview, Human, AI, People (actual / human / AI lines)
output/reports/milestone.html Milestone page: human participants only (Overview + Human)
output/reports/analysis_report.json Machine-readable human + AI payload
data/processed/ai/prediction_points.csv Per-point sequences for post-factum model scoring
data/processed/model_cache/forecasts.parquet Trained-model forecasts kept between builds (see below)
output/figures/*.png Plots (also copied under output/reports/figures/)
data/processed/participants.csv Person-level analysis table

Scoring the trained models (CITRAS-FM, KronLSTM, Timestral, SugarOne) on every window takes minutes, so each build keeps their forecasts in data/processed/model_cache/. The next build scores only windows that are new or whose visible three hours changed; new model files or a change to a model's scoring code score every window again. Set SDP_SCORE_CACHE=off to rescore everything, or point it at another file.

Export sequences only (no full report rewrite):

uv run sdp export-ai

After a model writes predictions (join keys study_id, run_id, round_number, point_index):

uv run sdp ingest-ai --predictions path/to/model.csv

evaluation_mode is post_factum for replays of saved games. in_place is the forecast from prediction_statistics_ai.csv when the app recorded it during the game.

AI on the latest participants (gitignored)

Human H1–H5 use the whole export. Model scoring, H6, and the People tab use the most recent participants, including the latest two. Each player is shown with only the models that scored their rounds. The latest day's players are Kronos. Earlier players keep every model scored on those rounds.

uv run sdp milestone-ai

The page is output/milestone_ai/milestone_ai.html (gitignored). It is the same explorer as the full report: Overview, Human, AI, and People. Put prediction_statistics.csv and prediction_statistics_ai.csv in the same folder and pass that human path as --csv. player_models.csv in the dest folder lists the models for each of those players. Pass --predictions <model_predictions.csv> to rebuild from an already scored file instead of running the models again.

Jupyter (same library modules and same report folder)

uv sync --group dev
uv run sugar-data-processing make-fixture
uv run jupyter notebook notebooks/study_analysis.ipynb

Select the sugar-data-processing (.venv) kernel. The notebook only calls library APIs (prepare_session, gathering, verification, statistics, comparison, output). It writes to the same place as the CLI: output/reports/ (no notebook-specific report directory).

Library import

from pathlib import Path
from sugar_data_processing import run_analysis

result = run_analysis(
    Path("data/fixtures/synthetic_prediction_statistics.csv"),
    Path("output"),
)
print(result.verification.passed, result.report_path)

Or stage-by-stage:

from pathlib import Path

import polars as pl

from sugar_data_processing.gathering import load_prediction_statistics, build_participant_table
from sugar_data_processing.verification import verify_dataset
from sugar_data_processing.statistics import run_all_hypotheses
from sugar_data_processing.comparison import benchmark_context
from sugar_data_processing.output import write_report

csv = Path("data/fixtures/synthetic_prediction_statistics.csv")
runs = load_prediction_statistics(csv)
participants = build_participant_table(runs)
verification = verify_dataset(runs, participants)
suite = run_all_hypotheses(participants)
eligible = participants.filter(pl.col("eligible_primary"))
benchmarks = benchmark_context(eligible if eligible.height > 0 else participants)
write_report(
    runs=runs,
    participants=participants,
    suite=suite,
    benchmarks=benchmarks,
    verification=verification,
    output_dir=Path("output"),
    source_csv=csv,
)

What each stage produces

Stage In-memory On disk (via analyze / write_report)
Gathering runs, participants Polars frames —
Verification VerificationReport (schema + quality) Embedded in report JSON / markdown §6
Statistics HypothesisSuite (H1–H5) Report §3–4 + JSON hypotheses
Comparison BenchmarkContext Report §5 + JSON benchmarks
Output report path output/reports/analysis_report.*, explorer.html, output/figures/*, data/processed/* + data/processed/ai/

Verification details

  • Schema: required columns, non-empty IDs, format ∈ {A,B,C}, parseable per_round_metrics, non-negative MAE, rounds_played vs metrics length
  • Quality flags: implausible age/durations, flag mismatches, MAE IQR outliers, short sessions, duplicate run_id, below-threshold participants

High-severity schema issues set verification.schema_ok = False. Quality flags are reported but do not alone fail verification (they inform the report).

Tests

uv run pytest

Notes

  • Models only see the 3-hour game window (24 visible points, left-padded to 128). explorer.html People tab overlays actual CGM, the human line, and each AI forecast.
  • See docs/analysis-plan.md for the §7 → module mapping and format conventions.

About

Statistical analysis pipeline for Sugar Sugar study results (H1-H5 per study design)

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages