I am trying to train a computer to play with LEGO bricks (simulated) using DQN. My input is an image with 4 color channels (RGB and depth) and the output of the neural network is the coordinates of the best position where the next brick should be put.

Instead of having $m_x * m_y * m_z$ (where $m_k$ is the size of positions along the k axis) different outputs, I want to have 3 output layers, each of which returns the x, y or z coordinates of the best position, so I have only $m_x + m_y + m_z$ outputs.

My idea is replacing the fully-connected output layer by three fully-connected output layers, but only have one reward value (labeled value) for three layers. Can I do this? Will the three layers returns different outputs if the reward is same?


You decide what the output layer represents, so it should be OK (and probably easier to implement) to have a single fully connected layer of size $m_x + m_y + m_z$ and to interpret the output in your own code. E.g. $n_x = N_{(x)}, n_y = N_{(m_x + y)}, n_z = N_{(m_x + m_y + z)}$, where $N_{(i)}$ is the output of the $i^{th}$ neuron. In the case of regression, this should have very similar behaviour to 3 separate fully connected layers for each dimension.

However, once you have either structure, it is not clear that you will be estimating reward directly any more. There is definitely no guarantee that the rewards will be the same between $n_x, n_y, n_z$, in fact they will not be in most cases. You will need to define a function $\hat{r}(x, y, z)$ that combined the outputs of the network and have your loss and error gradient calculated from that. The function could just be $\hat{r}(x, y, z) = n_x + n_y + n_z$ or maybe $\hat{r}(x, y, z) = n_x \times n_y \times n_z$

Clearly this approximation has limits built in - it will likely over-estimate and under-estimate rewards due to its structure. If the simplified structure matches your problem, then this will be fine - it may even speed convergence. But if not, it will limit performance.

An alternative, if you don't want to build a network that has a large number of outputs, is to have a single reward estimate as the output, and have the action choice as an input. That has the advantage of leaving the details of the approximation to the neural network. It has the disadvantage that you would need to run the network up to $m_x * m_y * m_z$ times in order to decide the policy in each state.

Because of the large number of possible actions in your case, you may want to look at a policy-based learning method, rather than DQN. There are deep NN-based versions of those algorithms. A recently-published algorithm called Asynchronous Advantage Actor-Critic (A3C) has performed well in computer gaming tasks and might be appropriate for your problem.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.