This repository contains the official implementation of our ICSE 2025 paper, "Decictor: Towards Evaluating the Robustness of Decision-Making in Autonomous Driving Systems." We also include existing baselines in our framework to facilitate the comparison and evaluation of different methods.
Our experiments are conducted on a server with the following specifications:
- CPU: AMD EPYC 7543P 32-Core Processor
- GPU: NVIDIA RTX A5000
- RAM: 256GB
- Storage: 1TB
From experience, we recommend using a machine with minimum specifications of:
- CPU: Intel Core i7-7700K
- GPU: NVIDIA GeForce GTX 1080
- RAM: 32GB
- Storage: 256GB
Our tool only supports Linux systems. Our experiments are conducted with the following software configurations:
- OS: Ubuntu 20.04
- Anaconda (Installation instructions can be found here)
- NVIDIA Driver (Installation instructions can be found here)
- Docker (Installation instructions can be found here)
- NVIDIA Container Toolkit (Installation instructions can be found here)
Make sure you have correct software installed before setting up for Decictor. We recommend following the installation instructions from Baidu Apollo to set up the environment. The instructions can be found here.
We assume the root directory is /workspace. Please replace it with your own directory.
- Recommended (Zenodo)
For convenient configuration, we recommend downloading Apollo from Zenodo.
We have fixed several build bugs in Apollo v7.0.0 and added command-line tools for SimControl.
cd /workspace
# download apollo.zip
unzip apollo.zip
cd /workspace/apollo- Step-by-Step (If the recommended way does not work)
Step 1: Download Apollo from the official repository:
git clone -b r7.0.0 https://github.com/ApolloAuto/apollo.git
cd /workspace/apollo
Step 2: However, there may be some issues during the building process of Apollo v7.0.0. These can be resolved by applying the following fixes:
vim WORKSPACE
# paste the following patch to line 60
http_archive(
name = "zlib",
build_file = "@com_google_protobuf//:third_party/zlib.BUILD",
sha256 = "629380c90a77b964d896ed37163f5c3a34f6e6d897311f1df2a7016355c45eff",
strip_prefix = "zlib-1.2.11",
urls = ["https://github.com/madler/zlib/archive/v1.2.11.tar.gz"],
)
Please refer to ApolloAuto/apollo#14374 and https://github.com/ApolloAuto/apollo/pull/14387/files
Step 3: Move the Decictor/apollo/sim_control directory to the apollo/modules folder.
Note: The following commands may require sudo privileges depending on your Docker settings. If you encounter a "permission denied" error, please try running the command with sudo.
(1) Start docker container
cd /workspace/apollo
bash docker/scripts/dev_start.sh
Once finished, you will see a docker container named apollo_dev_$USER. ($USER is your username)
(2) Compile Apollo
cd /workspace/apollo
bash docker/scripts/dev_into.sh
You are now in the container, shown like $USER@in-dev-docker:/apollo. You need to run the following command to build Apollo:
./apollo.sh build
If the build finishes successfully, you can see the following message:
[ OK ] Done building apollo. Enjoy!
You can exit the container by typing exit.
We provide two sources (Zenodo and Github) for Decictor. Considering that Decictor includes some large external files (e.g., maps), we recommend users start with the Zenodo Source, which contains all the necessary files. If you prefer to use the GitHub Source, you will need to download the maps separately.
cd /workspace
# download Decictor.zip
unzip Decictor.zip
cd /workspace
git clone https://github.com/MingfeiCheng/Decictor.git
# download data
cd /workspace/Decictor
# download data.zip from https://drive.google.com/file/d/1j4XVUTicmDR6x5YK_-Rj3ee3PpNRQuSi/view?usp=drive_link
unzip data.zip # note that, the data should be /workspace/Decictor/data
Note: Make sure you have installed Anaconda before running the following commands.
conda create -n decictor python=3.7.16 -y
conda activate decictor
pip install -r requirements.txt
# install pytorch
pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117
project_root and output_root are resolved automatically from the location of this
repository (project_root is the repository directory, output_root is <project_root>/outputs),
so config/common/project.yaml normally needs no editing.
apollo_root defaults to <project_root>/software/apollo. If your Apollo checkout lives
elsewhere, pass it on the command line or via the APOLLO_ROOT environment variable:
python main_fuzzer.py common.apollo_root=/path/to/apollo ...
# or
export APOLLO_ROOT=/path/to/apollo
Any of the three can still be overridden the same way, e.g. common.output_root=/path/to/outputs.
After setting up the configuration, execute the following command to start Decictor:
python main_fuzzer.py fuzzer=decictor fuzzer.run_hour=4 seed_name=scenario_1 map_name=sunnyvale_loop run_name=run_1
We provide the following seed corpus:
| Seed Name | Map Name |
|---|---|
| scenario_1 | sunnyvale_loop |
| scenario_2 | sunnyvale_loop |
| scenario_3 | sunnyvale_loop |
| scenario_4 | sunnyvale_loop |
| scenario_5 | sunnyvale_big_loop |
| scenario_6 | sunnyvale_big_loop |
We provide the following baselines you can use by setting the fuzzer parameter:
decictordrivefuzzeravfuzzerbehavexplorsamotadeepcollisionrandomrandom_delta
For convenience, experiment scripts are available in the scripts folder:
- To run experiments for RQ1:
bash scripts/rq1.sh - To run experiments for RQ2:
bash scripts/rq2.sh
These scripts locate the project root automatically; pass common.apollo_root=... (or set
APOLLO_ROOT) if your Apollo checkout is not at <project_root>/software/apollo.
Moreover, for the baselines DoppelTest and scenoRITA, we only modify the initial seeds in their original projects.
The modified projects are available on Zenodo. Please refer to their documentation and follow the provided instructions to set up the projects.
We provide a run_scenarios.sh script to run their projects under Decictor's settings. To use it, simply update the seed_root variable in the shell script.
@article{cheng2024evaluating,
title={Decictor: Towards Evaluating the Robustness of Decision-Making in Autonomous Driving Systems},
author={Cheng, Mingfei and Zhou, Yuan and Xie, Xiaofei and Wang, Junjie and Meng, Guozhu and Yang, Kairui},
journal={arXiv preprint arXiv:2402.18393},
year={2024}
}
Contact: Mingfei Cheng
We warmly welcome contributions and suggestions to enhance this project and promote research in industrial-grade ADS testing. Your contributions are highly valued and greatly appreciated.
Distributed under the GPL-3.0 License. See LICENSE for more information.
We thanks for the following open-source projects: