3
$\begingroup$

I have implemented a RL model based on Deep Q-Learning for learning how to play a 2D game, like the ones in the OpenAI Gym. For testing the model, unlike most people, I have chosen to evaluate its performance on different levels from the ones used for training. This way, I can assess if the knowledge learnt by the model generalizes well to previously unseen levels.

After doing some testing and plotting the results, I have noticed an issue. Although the model generalizes well, it ends up overfitting the training set (levels used for training) if the training process continues for too long. This makes sense since early stopping is a common technique used to prevent overfitting. The problem is that the longer the training lasts, the more samples the agent is trained on, which should improve its test performance. This doesn't happen, however, due to overfitting, as I have explained.

I would like to run the model and plot the performance (reward obtained) obtained on the test set as the number of samples in the experience replay increases. The plot I want to obtain should show how the performance increases as the number of samples grows, but overfitting causes that, after a point, the performance decreases, even if more samples are used for training. I want to avoid this. These are the solutions I can think of:

  • Use techniques to prevent overfitting. Right now I'm using Dropout but it still occurs.
  • Use an exponential decaying learning rate. Right now I'm using a constant one, since I don't want the newer samples to impact the training less than the older ones.
  • Give up and change the number of training steps depending on how many samples I want to use for training. Then, to obtain the plots, I will have to repeat the training once per each point (number of samples) of the plot, in order to obtain the best performance for each different size of the experience replay. Of course, this is my last resort.

What do you think I should do?

Edit: I have just tried L2 regularization. It works more or less the same as Dropout.

$\endgroup$
1
  • $\begingroup$ Try to use the regulatory function that best suits via cross-validation alongside your loss function. $\endgroup$ Apr 30, 2020 at 20:52

2 Answers 2

1
$\begingroup$

In the end, I have simply solved this problem by maintaining a constant number of training iterations independently of the size of the training dataset used. This way, I am able to test how the performance varies as the size of the training dataset increases, without also increasing the number of training iterations and having the risk of training for too long and overfitting the model. I tested and found a more or less suitable number of iterations for both small and big datasets so as to prevent overfitting using early-stopping (not using too many training iterations).

$\endgroup$
1
  • $\begingroup$ There is no "overfitting" in RL, at least not as you may think of it. You may be thinking of neural network overfitting, which is a different topic, separate from RL. Manually finding a number of iterations which work may work for this specific example of yours, but this is not the general solution. See nbro's answer ai.stackexchange.com/questions/20127/…. $\endgroup$ Dec 9, 2021 at 10:33
0
$\begingroup$

In RL this is known as the exploration-exploitation trade-off, so i don't think you can avoid this regularizing the neural network, but rather the action policy it learns. The agent you are training is learning to solve a specific task and its policy is "over-fitting". You could tackle this in several ways, one of which is to encourage exploration by using a distribution over the Q Values such as Softmax

$$p(a|s) = \frac{\exp(\beta Q(a,s)}{\sum_{a'} \exp(\beta Q(a',s))},$$

or by applying a $\epsilon$-greedy action selection, where you choose a random action with probability $\epsilon$ and otherwise the max Q Value. There are a lot of other ways to achieve this, just to quick "exploration strategies for dqn" search on google scholar.

$\endgroup$
1
  • $\begingroup$ I already use a technique to balance exploration and exploitation, although it is a bit different. In my case, I use offline training, i.e., train the model on a fixed training dataset I have obtained. I obtain this dataset by randomly sampling the actions, which would correspond to a ϵ-greedy exploration with ϵ=1 (100% of choosing a random action). However, this doesn't solve overfitting. The problem of generalizing RL to new, unseen tasks/levels is still unsolved, and new approaches are proposed such as papers.nips.cc/paper/6046-value-iteration-networks $\endgroup$
    – Aeryan
    May 12, 2020 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.