Header logo is am

Reinforcement learning of motor skills in high dimensions: A path integral approach

2010

Conference Paper

am


Reinforcement learning (RL) is one of the most general approaches to learning control. Its applicability to complex motor systems, however, has been largely impossible so far due to the computational difficulties that reinforcement learning encounters in high dimensional continuous state-action spaces. In this paper, we derive a novel approach to RL for parameterized control policies based on the framework of stochastic optimal control with path integrals. While solidly grounded in optimal control theory and estimation theory, the update equations for learning are surprisingly simple and have no danger of numerical instabilities as neither matrix inversions nor gradient learning rates are required. Empirical evaluations demonstrate significant performance improvements over gradient-based policy learning and scalability to high-dimensional control problems. Finally, a learning experiment on a robot dog illustrates the functionality of our algorithm in a real-world scenario. We believe that our new algorithm, Policy Improvement with Path Integrals (PI2), offers currently one of the most efficient, numerically robust, and easy to implement algorithms for RL in robotics.

Author(s): Theodorou, E. and Buchli, J. and Schaal, S.
Book Title: Robotics and Automation (ICRA), 2010 IEEE International Conference on
Pages: 2397-2403
Year: 2010
Month: May
Day: 3-7

Department(s): Autonomous Motion
Bibtex Type: Conference Paper (inproceedings)

Cross Ref: p10412
ISBN: 1050-4729
Note: clmc
URL: http://www-clmc.usc.edu/publications/T/theodorou-ICRA2010.pdf

BibTex

@inproceedings{Theodorou_RAIIC_2010,
  title = {Reinforcement learning of motor skills in high dimensions: A path integral approach},
  author = {Theodorou, E. and Buchli, J. and Schaal, S.},
  booktitle = {Robotics and Automation (ICRA), 2010 IEEE International Conference on},
  pages = {2397-2403},
  month = may,
  year = {2010},
  note = {clmc},
  crossref = {p10412},
  url = {http://www-clmc.usc.edu/publications/T/theodorou-ICRA2010.pdf},
  month_numeric = {5}
}