Goal-Based Reinforcement Learning via Experts Hints

In this paper the new approach for accelerating reinforcement learning algorithms is proposed. The key concept of the new method is to divide a learning process of the agent into two stages. During the first stage the agent learns several locally optimal policies (optimal for the subset of possible initial states) using goal-based learning, where each goal (particular state of the environment) is generated by some expert —— another agent (one or more) that acts optimally from the subset of the environment's states. In the second stage of the learning process the agent tries to build the globally optimal policy by combining learned during the first stage policies. For many simple environments policy-gradient method often stucks in locally optimal (suboptimal) policy, whereas proposed in this work way of learning provides a better exploration of the environment, especially when the reward signal is sparse, and does not stuck in insufficient suboptimal policy due to separating learning process into two stages and intrinsic motivation coming from experts. We show that even if we know about experts almost nothing a priori, for example, we do not know expert's policies and the optimality area for each expert, we can extract useful information from experts hints for enhancing the learning process of the agent. The detailed experimental comparison between vanilla policy gradient with baseline and proposed method is provided in this work. Moreover, we show how to deal with sparse reward using concepts of proposed method.