Skip to content
vitgruibPublic

About

An adversarial RL driver that learns to make highway ACC/MOBIL cars crash, trained with PPO + Prioritized Level Replay over Scenic-generated MetaDrive scenes.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

29 Commits

Folders and files

Repository files navigation

SIPACL

Teaching an adversarial driving agent to break highway autopilots, with a curriculum built from Scenic scenarios.

Python 3.11 PyTorch Scenic MetaDrive CleanRL PPO

An RL "attacker" car shares a multi-lane highway (CARLA Town06) with five cars, each driven by an adaptive-cruise-control + MOBIL lane-change controller. It earns a reward when it causes one of them to crash or leave the road, and is penalized if it crashes itself. Every episode is a fresh scene sampled from a Scenic program (lane placement, target speeds, politeness, braking limits, following gaps), simulated in MetaDrive.

Most random scenes are either trivially easy or impossible to attack. SIPACL adds Prioritized Level Replay on top of PPO: it keeps a buffer of past scenes, scores each by how much the agent is still learning from it, and replays the most informative ones.

Results

30 paired seeds, 100k training steps each, then 50 evaluation episodes on fresh random scenes (no replay at eval time):

Paired comparison over 30 seeds: PLR mean eval return 79.1 vs 76.2 without replay; PLR wins on 21 of 30 seeds, paired t-test p = 0.033
No replay PLR (p = 0.75)
Mean eval return 76.2 79.1
Std across seeds 8.5 6.2
Seeds where it wins 9 21

Paired t-test p = 0.033, Wilcoxon signed-rank p = 0.036, Cohen's d = 0.41. PLR gives higher returns and less variation from seed to seed. Raw numbers for every run are in Work/policy/runs_results.csv; python docs/make_figure.py redraws the figure, and Work/policy/seedbyseed_compare.py runs the full paired analysis.

Training loop and scene replay

flowchart LR
    S["Scenic program<br/>rl_mobil.scenic"] -->|sample new scene| P{"replay?<br/>prob p"}
    B[("Scene buffer<br/>ranked by learning progress")] -->|replay high-LP scene| P
    P --> M["MetaDrive episode<br/>attacker + 5 ACC/MOBIL cars"]
    M -->|obs, reward| A["PPO agent"]
    A -->|steer, throttle| M
    M -->|"rewards + critic values"| L["Positive value loss<br/>mean(max(GAE δ, 0))"]
    L -->|score scene| B
Loading
Observation (19-d) lane-line lidar, the nearest car's position in the ego frame, ego speed / heading / controls
Action (2-d) steering and throttle/brake, continuous in [−1, 1]
Reward dense shaping for forward progress and speed; +10 when a target car crashes or goes off-road while the attacker stays clean; −5 if the attacker crashes, leaves the road, or falls behind every target
Scene score positive value loss, mean(max(GAE δ, 0)) over the episode

After each episode the environment recomputes GAE from that episode's rewards and critic values. A high positive value loss means the critic still underestimates the scene, so the agent has more to learn from it. Replays are drawn by score rank, and the buffer drops its oldest scene once it holds 5,000.

The 30-seed study and the checkpoints in Work/runs/ date from April 2026, before the lane-line lidar and nearest-car features were added to the observation (September 2026, 277f067).

Quickstart

From the repository root (where Scenic/, metadrive/ and Work/ live):

git clone https://github.com/vitgruib/SIPACL
cd SIPACL
pip install -r reqs.txt      # installs the vendored Scenic + MetaDrive (editable) and the pinned deps

Watch the best trained attacker (it picks the top mean_eval_return run that has a checkpoint in Work/runs/):

cd Work
python policy/run_best.py

Train one seed the way the study above did:

cd Work
python policy/ppo.py --no-evaluate-model --seed 1 \
    --replay-resample-prob 0.75 --total-timesteps 100000 \
    --eval-after-train --eval-episodes 50 --save-model --render False

Pass --replay-resample-prob -1 for the no-replay baseline. Each run appends a row to Work/policy/runs_results.csv and, with --save-model, writes Work/runs/<run_name>.pt. python policy/ppo.py --help lists every option. Put saved logs under Work/logs/ (gitignored).

Repository layout

Work/
├── policy/
│   ├── ppo.py                  # PPO training/eval loop (CleanRL-based) + PLR wiring
│   ├── run_best.py             # evaluate the best saved checkpoint
│   ├── seedbyseed_compare.py   # paired t-test / Wilcoxon / Cohen's d + diagnostics
│   └── runs_results.csv        # one row per run
├── custom/
│   ├── custom_gym.py           # Gymnasium env: scene buffer, LP scoring, replay sampling
│   └── custom_simulator.py     # Scenic↔MetaDrive bridge, observation (lidar + nearest car)
├── scenarios/rl_mobil.scenic   # the highway attack scenario and reward monitor
├── controllers/                # ACC, PID and lateral controllers for the target cars
├── runs/                       # trained checkpoints (.pt)
└── CARLA/                      # Town01–10 maps (MIT, from the CARLA project)
Scenic/  metadrive/             # vendored dependencies
docs/                           # README figures and the script that makes them

Built on

Scenic · MetaDrive · CleanRL · CARLA maps · Prioritized Level Replay (Jiang et al., 2021)

About

An adversarial RL driver that learns to make highway ACC/MOBIL cars crash, trained with PPO + Prioritized Level Replay over Scenic-generated MetaDrive scenes.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages