Sample efficient learning of manipulation skills poses a major challenge in robotics. While recent approaches demonstrate impressive advances in the type of task that can be addressed and the sensing modalities that can be incorporated, they still require large amounts of training data. Especially with regard to learning actions on robots in the real world, this poses a major problem due to the high costs associated with both demonstrations and real-world robot interactions. To address this challenge, we introduce BOpt-GMM, a hybrid approach that combines imitation learning with own experience collection. We first learn a skill model as a dynamical system encoded in a Gaussian Mixture Model from a few demonstrations. We then improve this model with Bayesian optimization building on a small number of autonomous skill executions in a sparse reward setting. We demonstrate the sample efficiency of our approach on multiple complex manipulation skills in both simulations and real-world experiments.

In this work, we consider a sparse reinforcement learning setting, in which a policy $\pi_\theta$ processing observations $s_t$, produces an action $a_t$, and receives a reward $r_t \in \R$ in return. The objective of the policy is to accumulate the maximum possible reward $R$ for an episode. We assume the reward to be sparse, only given out at the end of an episode for either success or failure, i.e. $R \in \set{0, 1}$. We assume the policy $\pi_\theta$ to be parameterized under a space $\Theta \subseteq \R^m$ and assume the existence of an update function $\oplus$ which can be used to derive an updated policy $\pi_{\theta,i} = \pi_\theta \oplus \Delta \theta_i$. We strive to find the optimal update $\Delta\theta^*$ which will yield the optimal policy $\pi^*=\pi_\theta \oplus \Delta \theta^*$. Performance is assessed by a non-deterministic evaluation function $h_{\theta}(\Delta\theta, j) \rightarrow \R$ which executes the policy yielded by the update for $j$ episodes and averages the rewards obtained by the executions. The overall objective is \[ \Delta\theta^* = \argmax_{\Delta\theta}\ h_{\theta}(\Delta\theta, j), \] where $j$ is constant. Each evaluation of $h_\theta$ yields a data point $(\Delta\theta_i, R_i)$, which is collected in a dataset $\D = \set{(\Delta\theta_1, R_1), \ldots, (\Delta\theta_I, R_I)}$. This dataset can be used in the search for $\Delta \theta^*$. Please refer to our paper for more details.

Adrian Röfer, Iman Nematollahi, Tim Welschehold, Wolfram Burgard, and Abhinav Valada
Bayesian Optimization for Sample-Efficient Policy Improvement in Robotic Manipulation
Under review, 2024.

@article{roefer2024boptgmm,
  title={Bayesian Optimization for Sample-Efficient Policy Improvement in Robotic Manipulation},
  author={Röfer, Adrian and Nematollahi, Iman and Welschehold, Tim and Burgard, Wolfram and Valada, Abhinav},
  journal={arXiv preprint arXiv:2403.14305},
  year={2024}
}