We use Convolutional neural network because it by design learns features that generalize over spatial location , so when using conv operation it reduce image size and that what we hope to have so we can have more simplicity in the following layers , but when doing zero padding the image return to its original size , so what is the benefit of doing this ? the same problem return back since image size will be still large . Also, CNN works well for classify images; so how it can be used in games to detect best next move, I mean how it decide the Q values of the actions from the image itself even though understanding image doesn't give any indication of the next move ?
2
-
$\begingroup$ May you give a link to where exactly did you find someone using zero padding after a CNN convolution ? And for your second question, last time I saw CNN used in reinforcement learning, it was with its outputs (features) being fed into a deep policy optimization network, CNN is just being used as an encoder in this case. Overall, can you try to give more details about the example / code you are thinking of ? This should help to give a more accurate answer. $\endgroup$– UbikuityMay 7, 2021 at 9:22
-
$\begingroup$ I am trying to understand how CNN and Q-learning is being used to build a game agent , so what I understand from your comment that CNN is being used to encode image then we use the policy network to detect best next move ?! but the question here when we say a policy network , is that mean the CNN is part of this network ?.Regarding the padding question , I need just to understand what is the benefit of zero padding , I mean what is the effect in next layers . $\endgroup$– user117272May 7, 2021 at 9:36
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
For padding in CNN's there is a useful answer in this link:https://stats.stackexchange.com/questions/246512/convolutional-layers-to-pad-or-not-to-pad
In Q-learning an action is taken in current state, and a next state is obtained. Then state-action value of the current state-action pair is updated by using best state-action value of obtained next state. If you have discrete state-action space, or a few states you can simply record each state-action pair in a table, and then update it by Q-learning, but if the space is continuous, or there are so many state-action pairs then you should use function approximation methods like neural networks. For example by extracting current state's visual features by CNN, and then updating weights to better approximate value of this perceived state, and taken action pair.
$\begingroup$
$\endgroup$
We only use zero padding on the edge of the image to be able to compute our convolution on 'edgy' pixels. It is usually just to keep 'round' values of dimension.
For example if your input is 64x64, with strides = 2x2, you'd expect a 32x32 output, but without padding, you would get 31x31. Padding is really not a big deal and you can remove it without having much consequences on your results (at least according to my experience), it's more a reason of 'good looking' dimensions (multiples of 2).
To get back to the use of CNN in policy decisions, your agent uses the state value to decide which action to take. CNN is used to transform a 256x256 input into a 30 or 20 output features that is concatenated into the state of the network. So a corresponding state could be [agent_position, agent _speed, features_from_image] instead of [agent_position, agent _speed, image], and then your network / Q algorithm takes its decision from this state. So CNN is a kind of subpart of the network.
