Evolution strategy (ES) has been shown great promise in many challenging reinforcement learning (RL) tasks, rivaling other state-of-the-art deep RL methods. Yet, there are two limitations in the current ES practice that may hinder its otherwise further capabilities. First, most current methods rely on Monte Carlo type gradient estimators to suggest search direction, where the policy parameter is, in general, randomly sampled. Due to the low accuracy of such estimators, the RL training may suffer from slow convergence and require more iterations to reach optimal solution. Secondly, the landscape of reward functions can be deceptive and contains many local maxima, causing ES algorithms to prematurely converge and be unable to explore other parts of the parameter space with potentially greater rewards. In this work, we employ a Directional Gaussian Smoothing Evolutionary Strategy (DGS-ES) to accelerate RL training, which is well-suited to address these two challenges with its ability to i) provide gradient estimates with high accuracy, and ii) find nonlocal search direction which lays stress on large-scale variation of the reward function and disregards local fluctuation. Through several benchmark RL tasks demonstrated herein, we show that DGS-ES is highly scalable, possesses superior wall-clock time, and achieves competitive reward scores to other popular policy gradient and ES approaches. Our main contributions are
- Integrating the DGS-ES method into RL framework and demonstrating its superior performance on solving several benchmark RL problems provided in OpenAI Gym (https://gym.openai.com/) and PyBullet (https://pybullet.org/wordpress/).
- Scalable implementation of our DGS-ES method on distributed computing systems to reduce time-to-solution of reinforcement learning.
- Jiaxin Zhang, Hoang Tran, and Guannan Zhang, Accelerating Reinforcement Learning with a Directional Gaussian Smoothing Evolution Strategy, under review (https://arxiv.org/abs/2002.09077).
Significance and Impact
We reduced the training time from ~10 hours to ~15 minutes for a set of benchmark robotic control problems in OpenAI Gym on OLCF SUMMIT.
Last Updated: January 14, 2021 - 10:58 am