Unnormalized Log Probability - RNN

Question

I am going through the deep learning book by Goodfellow. In the RNN section I am stuck with the following:

RNN is defined like following:

And the equations are :

Now the $O^{(t)}$ above is considered as unnormalized log probability. But if this is true, then the value of $O^{(t)}$ must be negative because,

Probability is always defined as a number between 0 and 1, i.e, $P\in[0,1]$, where brackets denote closed interval. And $log(P) \le 0$ on this interval. But in the equations above, nowhere this condition that $O^{(t)} \le 0$ is explicitly enforced.

What am I missing!

Answer 1

You are right in a sense that it is better to be called log of unnormalized probability. This way, the quantity could be positive or negative. For example, $\text{log}(0.5) < 0$ and $\text{log}(12) > 0$ are both valid log of unnormalized probabilities. Here, in more detail:

1. Probability: $P(i) = e^{o_i}/\sum_{k=1}^{K}e^{o_k}$ (using softmax as mentioned in Figure 10.3 caption, and assuming $\mathbf{o}=(o_1,..,o_K)$ is the output of layer before softmax),

2. Unnormalized probability: $\tilde{P}(i) = e^{o_i}$, which can be larger than 1,

3. Log of unnormalized probability: $\text{log}\tilde{P}(i) = o_i$, which can be positive or negative.

Answer 2

You are right, nothing stop $o_k^{(t)}$ from being nonnegative, the keyword here is "unnormalized".

If we let $o_k^{(t)}=\ln q_k^{(t)}$

$$\hat{y}^{(t)}_k= \frac{\exp(o^{(t)}_k)}{\sum_{k=1}^K \exp(o^{(t)}_k) }= \frac{q_k^{(t)}}{\sum_{k=1}^K q_k^{(t)}}$$

Here $q_k^{(t)}$ can be any positive number, they will be normalized to be sum to $1$.