Deep Exploration in Reinforcement Learning

This work studies count-based exploration methods and their generalization to environments with continious state spaces. A literature review results in a couple of counters approximating methods noticed. They are pseudo-counters and E-counters. Both of these methods are studied experimentally. Pseudo-counters based on online learning of a probabilistic model turn out to be inapplicable as it is with a variational autoencoder as a probabilistic model. E-counters are shown to be useful as an approximation of counters in count-based exploration methods.There are experiments conducted on a synthetic environment, requiring the directed exploration, as well as on different environments from the toolkit for developing and comparing reinforcement learning algorithms called OpenAI Gym. There are also two reinforcement learning algorithms used in experiments, namely Q-learning and TRPO. It turns out that usage of count-based exploration methods based on E-counters results in effective exploration of an environment in deep reinforcement learning. This work also compares two different types of count-based exploration methods: reward bonus and upper confidence bound ones. There is an extensive review of recent trends in the field of environment exploration in reinforcement learning. Key words: exploration, state-action visitation approximation, E-counters, reward bonus, upper confidence bound.