I'm trying to implement deep Q-learning, but I do not know what to put into the cost function.
My net has 8 scalar inputs, 4 scalar outputs (from 0-1) and no hidden layers. To calculate the cost I use MSE.
What goes into it? The first term is obviously the network's predictions, but what about the second? If I put in the predictions again, it'll just return 0, and there aren't labels for each possible state.
Do I just put in the previous network predictions? How would that help calculate the gradient and then take a step in the right direction?