2
$\begingroup$

From what I understand, a DQN agent has as many outputs as there are actions (for each state). If we consider a scalar state with 4 actions, that would mean that the DQN would have a 4 dimensional output.

However, when it comes to the target value for training the agent, it is usually described as a scalar value = reward + discount*best_future_Q.

How could a scalar value be used to train a Neural Network having a vector output?

For example see image in https://towardsdatascience.com/deep-q-learning-tutorial-mindqn-2a4c855abffc

DQN

$\endgroup$
7
  • $\begingroup$ can you provide the reference of the statement: "..a DQN agent has as many outputs as there are actions (for each state)" ? $\endgroup$ – Nikos M. Mar 28 at 17:39
  • $\begingroup$ See the image on analyticsvidhya.com/blog/2019/04/… for example. The DQN outputs match the number of actions. $\endgroup$ – Dhoop Mar 28 at 18:18
  • $\begingroup$ I doubt the image reflects this, as it has a single node as output. My guess is that it simply shows the possible outputs (of that single node) $\endgroup$ – Nikos M. Mar 28 at 18:20
  • $\begingroup$ Are you sure you are looking at the right image? I have no way to paste the image here. Also look at many other blogs such as towardsdatascience.com/… If you dont agree with this statement, then can you state how many outputs you think a DQN agent has? $\endgroup$ – Dhoop Mar 28 at 18:23
  • $\begingroup$ I have edited your post to add the image, unfortunately it has to await approval $\endgroup$ – Nikos M. Mar 28 at 18:25
0
$\begingroup$

I am of the opinion that this architecture is only one among others that can solve the same problem (for example one may have only 2 outputs one for the chosen action and one for the $Q$ value of that action, but I will not elaborate further on this).

What this architecture does is to output the whole function $Q(a)$, that is the $Q$ value as a function of action $a$. So each output node represents the $Q$ value for a certain action $a$ corresponding to that node (so node 1 corresponds to the $Q$ value for action $a_1$, node 2 to $Q$ value for action $a_2$ and so on..)

BUT this is NOT a vector output, in the usual sense of the term. This is a functional output that represents the whole function $Q(a)$ for each action $a$.

As far as the learning/decision rule is concerned things are as usual. Hope this is clear.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.