Teaching an adversarial driving agent to break highway autopilots, with a curriculum built from Scenic scenarios.
An RL "attacker" car shares a multi-lane highway (CARLA Town06) with five cars, each driven by an adaptive-cruise-control + MOBIL lane-change controller. It earns a reward when it causes one of them to crash or leave the road, and is penalized if it crashes itself. Every episode is a fresh scene sampled from a Scenic program (lane placement, target speeds, politeness, braking limits, following gaps), simulated in MetaDrive.
Most random scenes are either trivially easy or impossible to attack. SIPACL adds Prioritized Level Replay on top of PPO: it keeps a buffer of past scenes, scores each by how much the agent is still learning from it, and replays the most informative ones.
30 paired seeds, 100k training steps each, then 50 evaluation episodes on fresh random scenes (no replay at eval time):
| No replay | PLR (p = 0.75) | |
|---|---|---|
| Mean eval return | 76.2 | 79.1 |
| Std across seeds | 8.5 | 6.2 |
| Seeds where it wins | 9 | 21 |
Paired t-test p = 0.033, Wilcoxon signed-rank p = 0.036, Cohen's d = 0.41. PLR gives higher returns
and less variation from seed to seed. Raw numbers for every run are in
Work/policy/runs_results.csv; python docs/make_figure.py
redraws the figure, and Work/policy/seedbyseed_compare.py runs the full paired analysis.
flowchart LR
S["Scenic program<br/>rl_mobil.scenic"] -->|sample new scene| P{"replay?<br/>prob p"}
B[("Scene buffer<br/>ranked by learning progress")] -->|replay high-LP scene| P
P --> M["MetaDrive episode<br/>attacker + 5 ACC/MOBIL cars"]
M -->|obs, reward| A["PPO agent"]
A -->|steer, throttle| M
M -->|"rewards + critic values"| L["Positive value loss<br/>mean(max(GAE δ, 0))"]
L -->|score scene| B
| Observation (19-d) | lane-line lidar, the nearest car's position in the ego frame, ego speed / heading / controls |
| Action (2-d) | steering and throttle/brake, continuous in [−1, 1] |
| Reward | dense shaping for forward progress and speed; +10 when a target car crashes or goes off-road while the attacker stays clean; −5 if the attacker crashes, leaves the road, or falls behind every target |
| Scene score | positive value loss, mean(max(GAE δ, 0)) over the episode |
After each episode the environment recomputes GAE from that episode's rewards and critic values. A high positive value loss means the critic still underestimates the scene, so the agent has more to learn from it. Replays are drawn by score rank, and the buffer drops its oldest scene once it holds 5,000.
The 30-seed study and the checkpoints in Work/runs/ date from April 2026, before the lane-line
lidar and nearest-car features were added to the observation (September 2026, 277f067).
From the repository root (where Scenic/, metadrive/ and Work/ live):
git clone https://github.com/vitgruib/SIPACL
cd SIPACL
pip install -r reqs.txt # installs the vendored Scenic + MetaDrive (editable) and the pinned depsWatch the best trained attacker (it picks the top mean_eval_return run that has a checkpoint in Work/runs/):
cd Work
python policy/run_best.pyTrain one seed the way the study above did:
cd Work
python policy/ppo.py --no-evaluate-model --seed 1 \
--replay-resample-prob 0.75 --total-timesteps 100000 \
--eval-after-train --eval-episodes 50 --save-model --render FalsePass --replay-resample-prob -1 for the no-replay baseline. Each run appends a row to
Work/policy/runs_results.csv and, with --save-model, writes Work/runs/<run_name>.pt.
python policy/ppo.py --help lists every option. Put saved logs under Work/logs/ (gitignored).
Work/
├── policy/
│ ├── ppo.py # PPO training/eval loop (CleanRL-based) + PLR wiring
│ ├── run_best.py # evaluate the best saved checkpoint
│ ├── seedbyseed_compare.py # paired t-test / Wilcoxon / Cohen's d + diagnostics
│ └── runs_results.csv # one row per run
├── custom/
│ ├── custom_gym.py # Gymnasium env: scene buffer, LP scoring, replay sampling
│ └── custom_simulator.py # Scenic↔MetaDrive bridge, observation (lidar + nearest car)
├── scenarios/rl_mobil.scenic # the highway attack scenario and reward monitor
├── controllers/ # ACC, PID and lateral controllers for the target cars
├── runs/ # trained checkpoints (.pt)
└── CARLA/ # Town01–10 maps (MIT, from the CARLA project)
Scenic/ metadrive/ # vendored dependencies
docs/ # README figures and the script that makes them
Scenic · MetaDrive · CleanRL · CARLA maps · Prioritized Level Replay (Jiang et al., 2021)