3
$\begingroup$

I am going through the deep learning book by Goodfellow. In the RNN section I am stuck with the following:

RNN is defined like following:

enter image description here

And the equations are :

enter image description here

enter image description here

Now the $O^{(t)}$ above is considered as unnormalized log probability. But if this is true, then the value of $O^{(t)}$ must be negative because,

Probability is always defined as a number between 0 and 1, i.e, $P\in[0,1]$, where brackets denote closed interval. And $log(P) \le 0$ on this interval. But in the equations above, nowhere this condition that $O^{(t)} \le 0$ is explicitly enforced.

What am I missing!

$\endgroup$
5
$\begingroup$

You are right in a sense that it is better to be called log of unnormalized probability. This way, the quantity could be positive or negative. For example, $\text{log}(0.5) < 0$ and $\text{log}(12) > 0$ are both valid log of unnormalized probabilities. Here, in more detail:

  1. Probability: $P(i) = e^{o_i}/\sum_{k=1}^{K}e^{o_k}$ (using softmax as mentioned in Figure 10.3 caption, and assuming $\mathbf{o}=(o_1,..,o_K)$ is the output of layer before softmax),

  2. Unnormalized probability: $\tilde{P}(i) = e^{o_i}$, which can be larger than 1,

  3. Log of unnormalized probability: $\text{log}\tilde{P}(i) = o_i$, which can be positive or negative.

$\endgroup$
3
$\begingroup$

You are right, nothing stop $o_k^{(t)}$ from being nonnegative, the keyword here is "unnormalized".

If we let $o_k^{(t)}=\ln q_k^{(t)}$

$$\hat{y}^{(t)}_k= \frac{\exp(o^{(t)}_k)}{\sum_{k=1}^K \exp(o^{(t)}_k) }= \frac{q_k^{(t)}}{\sum_{k=1}^K q_k^{(t)}}$$

Here $q_k^{(t)}$ can be any positive number, they will be normalized to be sum to $1$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.