Header logo is am

Policy gradient methods for machine learning


Conference Paper



We present an in-depth survey of policy gradient methods as they are used in the machine learning community for optimizing parameterized, stochastic control policies in Markovian systems with respect to the expected reward. Despite having been developed separately in the reinforcement learning literature, policy gradient methods employ likelihood ratio gradient estimators as also suggested in the stochastic simulation optimization community. It is well-known that this approach to policy gradient estimation traditionally suffers from three drawbacks, i.e., large variance, a strong dependence on baseline functions and a inefficient gradient descent. In this talk, we will present a series of recent results which tackles each of these problems. The variance of the gradient estimation can be reduced significantly through recently introduced techniques such as optimal baselines, compatible function approximations and all-action gradients. However, as even the analytically obtainable policy gradients perform unnaturally slow, it required the step from ÔvanillaÕ policy gradient methods towards natural policy gradients in order to overcome the inefficiency of the gradient descent. This development resulted into the Natural Actor-Critic architecture which can be shown to be very efficient in application to motor primitive learning for robotics.

Author(s): Peters, J. and Theodorou, E. and Schaal, S.
Book Title: Proceedings of the 14th INFORMS Conference of the Applied Probability Society
Pages: 97-98
Year: 2007

Department(s): Autonomous Motion, Empirical Inference
Bibtex Type: Conference Paper (inproceedings)

Address: Eindhoven, Netherlands, July 9-11, 2007
Cross Ref: p2674
Note: clmc


  title = {Policy gradient methods for machine learning},
  author = {Peters, J. and Theodorou, E. and Schaal, S.},
  booktitle = {Proceedings of the 14th INFORMS Conference of the Applied Probability Society},
  pages = {97-98},
  address = {Eindhoven, Netherlands, July 9-11, 2007},
  year = {2007},
  note = {clmc},
  crossref = {p2674}