A reproducible simulation study of the success–latency trade-off in pretrained visual Diffusion Policy sampling. Based on Diffusion Policy, LeRobot, and gym-pusht.
Completed: 200 closed-loop episodes, 80 supplemental measured planning calls, verified trajectory replays, and a fresh-environment reproduction check. See the research report, raw episode records, CV wording, and learning checklist.
The checkpoint was trained by the LeRobot authors. This project's contributions are compatibility repair, strict checkpoint verification, a reproducible paired evaluation harness, sampling-budget experiments, and evidence-backed analysis. It does not claim a new algorithm, training the upstream model, or real-robot deployment. Implementation and experiment execution are AI-assisted.
Each condition uses the same 50 environment seeds. Timing below comes from a separate fixed-observation, interleaved benchmark, 20 measured calls per condition; the report also preserves the original rollout timing.
| Sampler | Success | Wilson 95% interval | Interleaved median planning |
|---|---|---|---|
| DDPM100 | 33/50 (66%) | 52.2–77.6% | 1310.8 ms |
| DDIM10 | 26/50 (52%) | 38.5–65.2% | 134.7 ms |
| DDIM20 | 37/50 (74%) | 60.4–84.1% | 258.6 ms |
| DDIM50 | 34/50 (68%) | 54.2–79.2% | 645.8 ms |
DDIM20's observed success difference versus DDPM100 is +8 percentage points, paired 95% bootstrap interval −8 to +24. This does not establish improved or equivalent success. Its supplemental median planning time is approximately 5.07× faster on this RTX 5070; this is not a real-robot timing guarantee. DDPM-to-DDIM comparisons change both scheduler and step count.
中文摘要：基于公开预训练视觉扩散策略完成四组、共 200 回合 Push-T 仿真评估，保留成功与失败的逐步轨迹、置信区间和原始计时。DDIM20 在本次样本中成功率为 74%，独立交替计时中位耗时约 259 ms；不能据此宣称成功率显著优于基线或已完成实机部署。
- Prespecified main protocol and supplemental timing protocol.
- Complete report, machine-readable summary, and paired comparison intervals.
- Independent release audit, exact model checks, and fresh-environment verification.
- Videos: first baseline success, seed 10000, first baseline failure, seed 10003, and debug failure, seed 2000. The first two are post hoc illustrations replayed from logged actions with per-step geometry checks; they do not replace aggregate evaluation.
Tested runtime: Python 3.12, Windows, NVIDIA RTX 5070 12 GB, PyTorch 2.11.0 CUDA 13.0. The exact environment is in requirements-lock.txt; CUDA wheel installation needs the PyTorch CUDA 13.0 index.
uv venv .venv --python 3.12
uv pip install --python .venv/Scripts/python.exe --no-deps -r requirements-lock.txt --extra-index-url https://download.pytorch.org/whl/cu130 --index-strategy unsafe-best-match
.venv/Scripts/hf.exe download lerobot/diffusion_pusht --revision 84a7c23178445c6bbf7e1a884ff497017910f653 --local-dir work/upstream
.venv/Scripts/python.exe prepare_model.py --source work/upstream --output work/model
.venv/Scripts/python.exe evaluate.py --model work/model --output work/debug --scheduler DDIM --steps 10 --seed-start 2000 --episodes 1 --video
.venv/Scripts/python.exe run_study.py --model work/model --output work/reproduced-results
.venv/Scripts/python.exe benchmark_latency.py --model work/model --output work/reproduced-timing.json
.venv/Scripts/python.exe build_report.py --results work/reproduced-results --timing work/reproduced-timing.json --report work/reproduced-reportTo recompute the published statistics and figures from the released evidence without running GPU inference:
.venv/Scripts/python.exe build_report.py --results results --report report
.venv/Scripts/python.exe audit_release.pyThe migration extracts the checkpoint's own normalization statistics, strictly loads all learned parameters, and checks exact tensor equality after saving. Evaluation resumes only when the saved manifest matches; every completed episode is saved atomically. Model weights and private CV files are excluded from Git.
The lock is a complete snapshot of the measured Windows environment. Use --no-deps as shown: it preserves the tested opencv-python-headless wheel and prevents LeRobot's broad dependency metadata from adding a second opencv-python wheel over the same cv2 namespace. This snapshot targets the documented scripts, not every optional LeRobot integration. A fresh-environment package/import/replay check is saved in evidence/clean_environment.json.
python -m unittest test_metrics -v checks statistical boundaries. verify_normalization.py --model work/model --output evidence/normalization_checks.json checks processors against explicit legacy formulas. The report generator rejects incomplete conditions, mismatched settings, seed differences or inconsistent episode metrics.
- Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, paper.
- LeRobot pretrained model card: pretrained checkpoint, Apache-2.0.
- LeRobot: policy, processors, normalization migration helpers, Apache-2.0.
- gym-pusht: simulation environment; see its upstream license.
This repository calls upstream libraries rather than redistributing their source or model weights. The original code is MIT-licensed; upstream dependencies and weights retain their licenses. Use CITATION.cff to cite this evaluation software, and also cite the upstream work. See the report for the single-checkpoint, single-task, uncertainty and shared-desktop timing limitations.