Softmax classifier never allows for 100% probability in LSTM?

When working with LSTM I am using a softmax classifier and a one-hot encoded vector approach. The softmax looks like this:

$$S(h_i) = \frac{e^{h_i}}{\sum e^{h_{total}}}$$

notice, LSTM's result is a $h=tanh(c) \circ \sigma(p)$

Where c is the cell state passed through a tanh as well, and $\circ$ is the component-wise product of two vectors.

Recall that tanh never goes beyond -1 and never goes above 1; The $\sigma$ never goes below 0 and never above 1

Does this mean that if we have a 4-neurons on the output - the best guess a network can make will be 1, -1, -1, -1?

When softmaxed, this will produce $\frac{2.72}{3.83}, \frac{0.37}{3.83}, \frac{0.37}{3.83}, \frac{0.37}{3.83}$ which are these probabilities:

0.7,

0.1,

0.1,

0.1

and we can never get ~100% certainty, (Edit: or in this example even above 70%) no matter the learning rate? Is there a way to combat it without destroying its derivative working nicely with Cross-entropy?

Edit

As a solution, I've tried to determine how close to $e^{−1}$ and $e^1$ the $e^x$ actually lies. Basically an 'inverse lerp', and it indeed puts values to 0 and 1 range. However, I am not sure if it affects derivatives & if it no longer will work out of the box when using Cross Entropy.

Here is the usual way to compute softmaxed-vector:

float totalSum = 0;
Vec softmaxedVec;
for(int i=0; i< tanhVec.Length; ++i){
softmaxedVec[i] =  exp( tanhVec[i]);
totalSum += softmaxedVec[i];
}

for(int i=0; i<softmaxedVec.Length; ++i){
softmaxedVec[i] = softmaxedVec[i] / totalSum;
}


and here is what I am now trying to do:

float totalSum = 0;
Vec softmaxedVec;
for(int i=0; i< tanhVec.Length; ++i){
softmaxedVec[i] =  inverse_lerp(0.3678,  2.71828,  exp( tanhVec[i]));
totalSum += softmaxedVec[i];
}

for(int i=0; i<softmaxedVec.Length; ++i){
softmaxedVec[i] = softmaxedVec[i] / totalSum;
}


//returns percentage: 0 the value sits on min, 1.0 the value sits on max
float inverse_lerp(float min, float max, float currValue){
return (currValue - min) / (max - min);
}


However, with this I am afraid that I might have lost that gorgeous $\frac{\partial E}{\partial W} = expected - target$ (curtesy of softmax & cross entropy working together)

..or if I basically ruined the benefit of softmax

2 Answers

If you feed the output of the LSTM directly into a softmax, you probably won't get good results.

If you use a softmax layer after a tanh layer, bad stuff happens. As you say, the confidence will never get near 100%. For instance, if there are two classes, you can never get above about 88% confidence. If there are $k$ classes, you can never get confidence above $e/(e + (k-1)/e)) = e^2/(e^2 + k-1)$.

So, rather than directly feeding the output of the LSTM directly into softmax, you can instead use the output of the LSTM as the input to one or more (fully-connected) layers of neural network.

• I also wonder if I can feed the non-activated cell state into the softmax. However it still will have sigma and tanh from the 2 gates: input and data – Kari Dec 10 '17 at 17:09

If want you mean by 100% certainty is:

1,

0,

0,

0,

as by your example, you cannot obtain this type of output using soft-max. For this to happen you will need that the softmax equals 0 for those i that are not correct, and this simply cannot happen because:

$$e^{h_i} \neq 0$$ $$\forall h_i \in \mathbb{R}$$

See this link for better explanation. Note that $e^x$ can be close to 0 but it will never be 0. The kind of output that you want is the one that the max function would output, sadly the max function is not differentiable so a soft version is needed hence softmax.

I think that the problem is that you are using softmax after tanh, and this so you must eliminate one of the "squashing" functions, see this discussion

The reason to use softmax is if the classes are mutually exclusives, also one of the advantages of using softmax is that it reduces sensitivity to outliers by giving at least some probability to the other values. See this links for a more detailed explanation 1 and 2.

• Thank you @feynman410, can you please show if there is a way to combat the issue? I understand that softmax is assymptotic & will never reach zero, however in my question I have issue where it's nowhere near zero. As a solution, I've tried to determine how close to $e^{-1}$ and $e^1$ the $e^x$ actually lies. Basically an 'inverse lerp', and it indeed puts values to 0 and 1 range. However, I am not sure if it affects derivatives & if it no longer will work out of the box when using Cross Entropy – Kari Dec 10 '17 at 12:50
• I updated my answer, I think your problem is that you are using softmax right after tanh – feynman410 Dec 10 '17 at 13:55