I was wondering why there is no precise picture of the softmax activation function on the internet. Is it difficult for the plot or what is the reason behind that since I want to compare it with a sigmoid function?

I just found the following funny pictures from dataaspirant which is not available anymore: img

Can anyone illustrate a Softmax function graph? and explain why one is faster than another one due to the above picture?


Softmax is a multivariable function, generally. You wouldn't take a softmax of a single variable just like you wouldn't take a maximum of a single variable. It's difficult to plot functions of more than 2 variables because our eyes see in 3 dimensions.

A sigmoid is a 1 dimensional function. In your picture, they are applying a sigmoid to every variable separately.

| improve this answer | |

The softmax function is used in the last layer of CNN network. Softmax is an activation function like tanh and ReLU, the difference is that this technique can interpret the incoming inputs as output probabilities. The method guarantees that the output probabilities will be in a range of 0 and 1, and the sum of them is 1, thus the scores are interpretable as a percentage rate for each class. The function uses this formula, also you can use these lines of code to compute softmax:

logits = [2.0, 1.0, 0.1]
exps = [np.exp(i) for i in logits]
sum_of_exps = sum(exps)
softmax = [j/sum_of_exps for j in exps]
| improve this answer | |
  • $\begingroup$ would you modify your code so that it could be possible to plot it and present the picture for further discussion? $\endgroup$ – Mario Aug 5 '19 at 18:32

Softmax isn't a continuous mathematical function such as logistic(sigmoid), tanh or ReLU. softmax is used to map outputs of the last layer of a Neural Network into a probability distribution. i.e. summation of softmax squashed layer's outputs will be 1 (unity).

Unlike other activation functions softmax takes a list/array of input and maps them into probability distribution. Example : softmax([ 2.0 , 1.0 , 0.1 ]) will return [ 0.7 , 0.2 , 0.1 ]

And 0.7 + 0.2 + 0.1 = 1

so if we pass softmax list with only one element its probability of occurrence will be 1(unity).

i.e. softmax([ any_value ]) will return 1

Therefore softmax is helpful only when multiple outputs have to be squashed.

function softmax(list){
// list is an array of outputs of last layer neurons of a Neural Network 
// numerators is an array of Math.exp() applied to each element of list array
    var numerators = list.map(function(e){ return Math.exp(e); });
// denominator is summation of each element of numerators array
    var denominator = numerators.reduce(function(p, c){ return p + c; });
// returning numerators array after dividing each element by denominator
    return numerators.map(function(e){ return e / denominator; });

softmax is helpful in classification problems

| improve this answer | |
  • $\begingroup$ that's right, is it possible to demonstrate it or not in general? why it is shown in the picture softmax is faster than sigmoid? $\endgroup$ – Mario Aug 5 '19 at 18:33
  • $\begingroup$ Softmax isn't continuous?? $\endgroup$ – Ben Reiniger Aug 5 '19 at 20:59
  • 1
    $\begingroup$ Also it's not a function of 1 variable so it cannot be plotted on a regular chart. It's not faster / slower than sigmoid, it's different. A sigmoid activation returns a single value between 0 and 1, used when you want to predict if the input is true or false (i.e. is a cat / is not a cat). A softmax is used if you want to predict multiple classes, i.e. for an image does it include a person (yes/no) and car (yes/no) a cat (yes/no) etc. $\endgroup$ – David Waterworth Aug 6 '19 at 1:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.