8
$\begingroup$

"One common mistake that I would make is adding a non-linearity to my logits output."

What does the term "logit" means here or what does it represent ?

$\endgroup$
11
$\begingroup$

Logits interpreted to be the unnormalised (or not-yet normalised) predictions (or outputs) of a model. These can give results, but we don't normally stop with logits, because interpreting their raw values is not easy.

Have a look at their definition to help understand how logits are produced.

Let me explain with an example:

We want to train a model that learns how to classify cats and dogs, using photos that each contain either one cat or one dog. You build a model give it some of the data you have to approximate a mapping between images and predictions. You then give the model some of the unseen photos in order to test its predictive accuracy on new data. As we have a classification problem (we are trying to put each photo into one of two classes), the model will give us two scores for each input image. A score for how likely it believes the image contains a cat, and then a score for its belief that the image contains a dog.

Perhaps for the first new image, you get values out of 16.917 for a cat and then 0.772 for a dog. Higher means better, or ('more likely'), so you'd say that a cat is the answer. The correct answer is a cat, so the model worked!

For the second image, the model may say the scores are 1.004 for a cat and 0.709. So once again, our model says we the image contains a cat. The correct answer is once again a cat, so the model worked again!

Now we want to compare the two result. One way to do this is to normalise the scores. That is, we normalise the logits! Doing this we gain some insight into the confidence of our model.

Let's using the softmax, where all results sum to 1:

$$\sigma (\mathbf {z} )_{j}={\frac {e^{z_{j}}}{\sum _{k=1}^{K}e^{z_{k}}}} \hspace{20mm} for \hspace{5mm} j = 1, …, K.$$

For the first test image, we get

$$prob(cat) = \frac{exp(16.917)}{exp(16.917) + exp(0.772)} = 0.9999$$ $$prob(dog) = \frac{exp(0.772)}{exp(16.917) + exp(0.772)} = 0.0001$$

If we do the same for the second image, we get the results:

$$prob(cat) = \frac{exp(1.004)}{exp(1.004) + exp(0.709)} = 0.5732$$ $$prob(dog) = \frac{exp(0.709)}{exp(1.004) + exp(0.709)} = 0.4268$$

The model was not really sure about the second image, as it was very close to 50-50 - a guess!

The last part of the quote from your question likely refers to a neural network as the model. The layers of a neural network commonly take input data, multiply that by some parameters (weights) that we want to learn, then apply a non-linearity function, which provides the model with the power to learn non-linear relationships. Without this non-linearity, a neural network would simply be a list of linear operations, performed on some input data, which means it would only be able to learn linear relationships. This would be a massive constraint, meaning the model could always be reduced to a basic linear model. That being said, it is not considered helpful to apply a non-linearity to the logit outputs of a model, as you are generally going to be cutting out some information, right before a final prediction is made. Have a look for related comments in this thread.

$\endgroup$
8
$\begingroup$

Logits is the unnormalized final scores of your model. You apply softmax to it to get a probability distribution over your classes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.