4
$\begingroup$

If I would like to use DQN to train my Reinforcement-Learning agent, how do I select the number of neurons?

In Supervised Learning, selecting too few or too many can result in either a too low potential (lack of neurons) or in a poor generalization due to overfitting (for example, due to too many neurons). We can use Genetic Algorithm to determine the best combination of hyperparameters instead of trying by hand.

However, in RL - is generalization even a thing? After all, we are training a spider to run by moving its limbs in a most efficient way, an arm to grab a box in the fastest way, etc.

Once learnt, are we even allowed to change the environment and expect agent to perform well, generalizing to new environment?

And thus, is selecting neurons even a thing for RL, or can we just give it a maximum possible number of neurons and layers, letting it learn correct movement of creature's limbs?

$\endgroup$
7
+50
$\begingroup$

Deep reinforcement learning (DRL) methods such as DQN involve deep learning as a large memory that is not expensive. The neural network learns in a supervised way to provide good estimates; and DQN is not an exception.

Hyper-parameters optimization for the neural network in DQN is similar to that of fully supervised learning. you should try various hyper-parameters[ number of layers, neurons,...etc] until obtaining a good solution.

Evolutionary algorithms can help you find appropriate hyper-parameters. Recently there are some published papers reported using evolutionary algorithms to train the neural network in RL.

  • are we allowed to change the environment and expect agent to perform well, generalizing to new environment?

Yes,the main idea of DQN is to allow generalization, see DeepMind publications such as [1]. And generalization to unseen environments is possible : see [2], [3]

As you can see in the literature, the boundaries of the generalization differ form one work to another. Generalization might mean the ability to perform well for different experiments at the same environment. Some researcher work on building agents that generalize to unseen environments which is more difficult and in this case the design of the neural network and the reinforcement learning algorithm play important role in determining the generalization ability.

  • is selecting neurons even a thing for RL, can we just give it a max possible number of neurons and layers, letting it learn correct movement of creature's limbs?

Generalization is an issue, large number of neurons can overfit (not generalize well) DQN involves neural network in order to allow generalization. The ability of the neural network to generalize to new cases (and new experiments/environments) depends on the approximation that the neural network provides which depends on the neural networks parameters and hyper-parameters.

For building a huge neural network : as a concept you can increase the depth of the neural network to get better results. However, the question is do you have enough data? in RL language, you need more different examples for the larger network to train. Theoretically, having infinite neural networks with infinite number of different experiments/environments will lead to the best results. however, we are not patient enough to wait for very long training time and most probably you don't have large number of different environments/experiments to teach the neural network. hence the standard way to do this is by starting with a small network (few neurons and 1 or 2 layers) and train it then expand it gradually step by step and stop when the network can't generalize well.

you can think of the size of the neural network as a water tank as long you have large amounts of water(experiments) you need larger tank to occupy it. A large network without sufficient number of learning experiments leads to worse results specially when the the agent face new unseen experiments.

:[1] https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf

:[2] https://www.semanticscholar.org/paper/Virtual-to-real-deep-reinforcement-learning%3A-of-for-Tai-Paolo/494af0cedf1abb2454d457d0a89e21b983233276

:[3] https://arxiv.org/pdf/1612.05533.pdf

$\endgroup$
2
  • $\begingroup$ Thank you Ahmed, curently this doesn't answer my question fully - in particular, are "selecting as little number of neurons as possible". To me, "until obtaining a good solution" would mean the best possible generalization - but is generalization an important criterion in RL? Some references would be appreciated $\endgroup$ – Kari Jun 15 '18 at 3:17
  • $\begingroup$ Hi @Kari plz see edited answer $\endgroup$ – ahmed Jun 15 '18 at 8:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.