# Unnormalized Log Probability - RNN

I am going through the deep learning book by Goodfellow. In the RNN section I am stuck with the following:

RNN is defined like following: And the equations are :  Now the $$O^{(t)}$$ above is considered as unnormalized log probability. But if this is true, then the value of $$O^{(t)}$$ must be negative because,

Probability is always defined as a number between 0 and 1, i.e, $$P\in[0,1]$$, where brackets denote closed interval. And $$log(P) \le 0$$ on this interval. But in the equations above, nowhere this condition that $$O^{(t)} \le 0$$ is explicitly enforced.

What am I missing!

You are right in a sense that it is better to be called log of unnormalized probability. This way, the quantity could be positive or negative. For example, $$\text{log}(0.5) < 0$$ and $$\text{log}(12) > 0$$ are both valid log of unnormalized probabilities. Here, in more detail:

1. Probability: $$P(i) = e^{o_i}/\sum_{k=1}^{K}e^{o_k}$$ (using softmax as mentioned in Figure 10.3 caption, and assuming $$\mathbf{o}=(o_1,..,o_K)$$ is the output of layer before softmax),

2. Unnormalized probability: $$\tilde{P}(i) = e^{o_i}$$, which can be larger than 1,

3. Log of unnormalized probability: $$\text{log}\tilde{P}(i) = o_i$$, which can be positive or negative.

You are right, nothing stop $$o_k^{(t)}$$ from being nonnegative, the keyword here is "unnormalized".

If we let $$o_k^{(t)}=\ln q_k^{(t)}$$

$$\hat{y}^{(t)}_k= \frac{\exp(o^{(t)}_k)}{\sum_{k=1}^K \exp(o^{(t)}_k) }= \frac{q_k^{(t)}}{\sum_{k=1}^K q_k^{(t)}}$$

Here $$q_k^{(t)}$$ can be any positive number, they will be normalized to be sum to $$1$$.