How to avoid overfitting in Reinforcement Learning

Question

I have implemented a RL model based on Deep Q-Learning for learning how to play a 2D game, like the ones in the OpenAI Gym. For testing the model, unlike most people, I have chosen to evaluate its performance on different levels from the ones used for training. This way, I can assess if the knowledge learnt by the model generalizes well to previously unseen levels.

After doing some testing and plotting the results, I have noticed an issue. Although the model generalizes well, it ends up overfitting the training set (levels used for training) if the training process continues for too long. This makes sense since early stopping is a common technique used to prevent overfitting. The problem is that the longer the training lasts, the more samples the agent is trained on, which should improve its test performance. This doesn't happen, however, due to overfitting, as I have explained.

I would like to run the model and plot the performance (reward obtained) obtained on the test set as the number of samples in the experience replay increases. The plot I want to obtain should show how the performance increases as the number of samples grows, but overfitting causes that, after a point, the performance decreases, even if more samples are used for training. I want to avoid this. These are the solutions I can think of:

• Use techniques to prevent overfitting. Right now I'm using Dropout but it still occurs.
• Use an exponential decaying learning rate. Right now I'm using a constant one, since I don't want the newer samples to impact the training less than the older ones.
• Give up and change the number of training steps depending on how many samples I want to use for training. Then, to obtain the plots, I will have to repeat the training once per each point (number of samples) of the plot, in order to obtain the best performance for each different size of the experience replay. Of course, this is my last resort.

What do you think I should do?

Edit: I have just tried L2 regularization. It works more or less the same as Dropout.

Answer 1

In the end, I have simply solved this problem by maintaining a constant number of training iterations independently of the size of the training dataset used. This way, I am able to test how the performance varies as the size of the training dataset increases, without also increasing the number of training iterations and having the risk of training for too long and overfitting the model. I tested and found a more or less suitable number of iterations for both small and big datasets so as to prevent overfitting using early-stopping (not using too many training iterations).

Answer 2

In RL this is known as the exploration-exploitation trade-off, so i don't think you can avoid this regularizing the neural network, but rather the action policy it learns. The agent you are training is learning to solve a specific task and its policy is "over-fitting". You could tackle this in several ways, one of which is to encourage exploration by using a distribution over the Q Values such as Softmax

$$p(a|s) = \frac{\exp(\beta Q(a,s)}{\sum_{a'} \exp(\beta Q(a',s))},$$

or by applying a $\epsilon$-greedy action selection, where you choose a random action with probability $\epsilon$ and otherwise the max Q Value. There are a lot of other ways to achieve this, just to quick "exploration strategies for dqn" search on google scholar.