We implemented the deep reinforcement learning paper One Policy to Control Them All with the OpenAI gym environment BipedalWalker. The main idea of the paper is to use the same Shared Modular Policy (SMP) for all actuators of an kinetic agent and let the actuators communicate with each other in order to enable global coordination. We are the first, who use a SMP for the Bidpeal Walker environment. Our implementation uses - and is forked from - the ReAlly framework from the University Osnabrueck lecture Deep Reinforcement Learning.
For a quick overview about our results, have a look at this video. For a detailed graphical conceptual overview about the paper and a scientific comparision between the baseline and shared modular policy approach, have a look at our report.
All relevant scripts are in the smp package.
We recommend executing the scripts from the ReAlly root directory using commands like:
python smp/smp.py --batch_size 32 --epochs 150 --gamma 0.98 --hidden_units 128 --learning_rate 0.0004 --policy_noise 0.5 --buffer_size 100000 --sample_size 256 --saving_dir "smp_results"
Note that command line arguments are needed. For further information check the -h option.
baseline.pytrains a monolithic model as a baseline for our expepriment.smp.pytrains our experimental model using a shared modular policy approach.smp_test.pyloads a trained model an executes some test episodes to visualize the performance of the model within the environmentbase_models.pyholds some useful base classes for our models.smp_utils.pyholds some helper functions.
Each run was based on configuration 8 from our best grid search result which can be found in
results/grid_search/results_1.md.
grid_search/results_1.md: summarized results of our grid searchbaseline_results: Monolithic baseline modelsmp_results_config8: simple SMP run with best configuration from grid searchsmp_results_config8_bigger_bufferincreased buffer size to 100,000smp_results_moreepochsincreased number of epochs to 150smp_results_config8_withlidar: added lidar to states
Authors: Niklas Dettmer & Marko Duda
