I'm trying to implement a Deep Q Network, but I'm stuck on how you train a network to predict multiple action-values when you can only collect data on a single action.
In the paper it recommends using a different output for each action
We instead use an architecture in which there is a separate output unit for each possible action, and only the state representation is an input to the neural network. The outputs correspond to the predicted Q-values of the individual actions for the input state.
Since we can only visit one action, we only know the loss for that action (ie. a single output). But as far as I'm aware, we need to have values for all of the outputs in order to train the network. What black magic can you use to get the other output values?
It seems like a bad idea to get the network to predict the other action-values and feed them back, as it would affect the optimizer. And if you tried to ignore the other outputs and train it as if there were only the one you were currently focused on, you would still affecting the others as they would share edges.