DQN - how is it possible to train separate outputs for each action?

Question

I'm trying to implement a Deep Q Network, but I'm stuck on how you train a network to predict multiple action-values when you can only collect data on a single action.

In the paper it recommends using a different output for each action

We instead use an architecture in which there is a separate output unit for each possible action, and only the state representation is an input to the neural network. The outputs correspond to the predicted Q-values of the individual actions for the input state.

Since we can only visit one action, we only know the loss for that action (ie. a single output). But as far as I'm aware, we need to have values for all of the outputs in order to train the network. What black magic can you use to get the other output values?

It seems like a bad idea to get the network to predict the other action-values and feed them back, as it would affect the optimizer. And if you tried to ignore the other outputs and train it as if there were only the one you were currently focused on, you would still affecting the others as they would share edges.

---

DQN Paper

Answer 1

The output of your network should be a Q value for every action in your action space (or at least available at the current state). Then you can use softmax or epsilon-greedy (or other strategies) to select the final action. The network will learn to predict which action should return the maximum reward from your current state. Also we update the network after we have collected a specific amount of experience and we use batches from that experience buffer in order to update the network. We do not feed back any value in the vanilla DQN (no recurrence).

There are many good implementations online that can help you understand the algorithm.

Answer 2

The other values are maintained from the existing q values. If you are using deep Q network, your replay memory will have a records of <s,a,r,s'> you are right the state is enough for all inputs. Bur the action and reward in the replay memory belong to a single output. To train it for all using the minibatch, first you fed the neural network with each state. Then, you record the Q values of the fed states for all outputs. Then, leaving the other Q values intact, you only update the Q values of the selected action with its reward using the bellman equation. Finally, you will have a data set of states and Q values for all outputs. You then train the neural network as the usual supervised learning.