In this post, https://cs231n.github.io/linear-classify/#webdemo it mentions "Softmax classifier interprets the scores inside the output vector f as the unnormalized log probabilities." Aren't we taking the log afterward and f is equal to just Wx+b?


No, that is not what is happening, the definition of the Softmax classifier is: $$z \in \mathbb{R}^K; \sigma(z)_j = \frac{e^{z_j}}{\sum_{k = 1}^{K} e^{z_k}} $$ Let's talk first about what the softmax function aims to do, but before we can understand that, we need to understand the aim of classification in general. Classifiers are functions which take a vector in $\mathbb{R^n}$ and output one of $k$ classes. The softmax function takes in a vector $z$ and outputs a value in $\mathbb{R^+}$ whose values sum to one.

Now let's get to your question, if I were to just take $\log(Wx + b)$. It is actually almost exactly that, we just take the exponent so that our values range from $0$ to $1$. However, these values are not normalized and in order to normalize it we need to divide by the sum and we take the exponents to maintain a positive value.

| improve this answer | |
  • $\begingroup$ What is z here? $\endgroup$ – Abhishek Bhatia Jun 18 '17 at 14:53
  • $\begingroup$ It is the equivalent of $f$ in the equation you posted, or more precisely, the output of $\log (Wx + b)$ $\endgroup$ – PSub Jun 18 '17 at 15:20
  • $\begingroup$ In the post, it is Wx+b. "In the Softmax classifier, the function mapping f(xi;W)=Wxif(xi;W)=Wxi " $\endgroup$ – Abhishek Bhatia Jun 18 '17 at 15:28
  • $\begingroup$ Can you state the objection function here as well. $\endgroup$ – Abhishek Bhatia Jun 18 '17 at 17:49
  • $\begingroup$ In canonical optimization, the objective function is the function you're looking to minimize/maximize. Generally this varies, sometimes people use Mean Squared Error to denote the loss for a machine learning algorithm, sometimes it's mean cross entropy, etc. Also as I hope I pointed out in my answer, it's "almost" what's happening, except you're normalizing the values. You're very close to understanding exactly what it is. Also a mapping is usually from a domain to a co-domain. In your example, you're considering the probability, not the actual function (which is what I stated in the question) $\endgroup$ – PSub Jun 19 '17 at 1:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.