What does Logits in machine learning mean?

Question

"One common mistake that I would make is adding a non-linearity to my logits output."

What does the term "logit" means here or what does it represent ?

Answer 1

Logits interpreted to be the unnormalised (or not-yet normalised) predictions (or outputs) of a model. These can give results, but we don't normally stop with logits, because interpreting their raw values is not easy.

Have a look at their definition to help understand how logits are produced.

Let me explain with an example:

We want to train a model that learns how to classify cats and dogs, using photos that each contain either one cat or one dog. You build a model give it some of the data you have to approximate a mapping between images and predictions. You then give the model some of the unseen photos in order to test its predictive accuracy on new data. As we have a classification problem (we are trying to put each photo into one of two classes), the model will give us two scores for each input image. A score for how likely it believes the image contains a cat, and then a score for its belief that the image contains a dog.

Perhaps for the first new image, you get logit values out of 16.917 for a cat and then 0.772 for a dog. Higher means better, or ('more likely'), so you'd say that a cat is the answer. The correct answer is a cat, so the model worked!

For the second image, the model may say the logit values are 1.004 for a cat and 0.709 for a dog. So once again, our model says we the image contains a cat. The correct answer is once again a cat, so the model worked again!

Now we want to compare the two result. One way to do this is to normalise the scores. That is, we normalise the logits! Doing this we gain some insight into the confidence of our model.

Let's using the softmax, where all results sum to 1 and so allow us to think of them as probabilities:

$$\sigma (\mathbf {z} )_{j}={\frac {e^{z_{j}}}{\sum _{k=1}^{K}e^{z_{k}}}} \hspace{20mm} for \hspace{5mm} j = 1, …, K.$$

For the first test image, we get

$$prob(cat) = \frac{exp(16.917)}{exp(16.917) + exp(0.772)} = 0.9999$$ $$prob(dog) = \frac{exp(0.772)}{exp(16.917) + exp(0.772)} = 0.0001$$

If we do the same for the second image, we get the results:

$$prob(cat) = \frac{exp(1.004)}{exp(1.004) + exp(0.709)} = 0.5732$$ $$prob(dog) = \frac{exp(0.709)}{exp(1.004) + exp(0.709)} = 0.4268$$

The model was not really sure about the second image, as it was very close to 50-50 - a guess!

The last part of the quote from your question likely refers to a neural network as the model. The layers of a neural network commonly take input data, multiply that by some parameters (weights) that we want to learn, then apply a non-linearity function, which provides the model with the power to learn non-linear relationships. Without this non-linearity, a neural network would simply be a list of linear operations, performed on some input data, which means it would only be able to learn linear relationships. This would be a massive constraint, meaning the model could always be reduced to a basic linear model. That being said, it is not considered helpful to apply a non-linearity to the logit outputs of a model, as you are generally going to be cutting out some information, right before a final prediction is made. Have a look for related comments in this thread.

Answer 2

Logits is the unnormalized final scores of your model. You apply softmax to it to get a probability distribution over your classes.

Answer 3

As other answers state, "logits" refers to unnormalized log-probabilities. However, what does that mean?

The term "logit" is used in machine learning models that output probabilities, that is, numbers between 0 and 1. The most prominent ones are classification models, either binary classification or multi-class classification:

• Binary classification models tell whether the input belongs or not to the positive class, that is, they generate a single number between 0 and 1 representing the probability of the input belonging to the positive class, that is, they model a Bernoulli distribution conditioned on the input.

  Normally, the model generates an unbounded real number that is then "squashed" into the $(0, 1)$ range with the sigmoid function: $\sigma(x) = \frac{1}{1 + e^{-x}}$. The unbounded real number (i.e. the unnormalized log-probability) is the logit.

  Note that, at inference time, in order to know if the probability is greater than 0.5, we don't need to compute the sigmoid: by the nature of the sigmoid, if the logit is greater than 0, then the probability computed with the sigmoid will be greater than 0.5.

• Multiclass classification models tell to which class the input belongs, among a set of pre-defined N classes, that is, they generate N probabilities, each between 0 and 1, and they all add up to 1. This way, they model a categorical distribution (sometimes referred to as "multinomial") conditioned on the input.

  Normally, the model generates N unbounded real numbers whose range and sum is normalized to $(0, 1)$ by means of the softmax function: $\sigma (z)_i = \frac{e^{z_i}}{\sum_{j=1}^N e^{z_j}}$. The N unbounded real numbers (i.e. the unnormalized probabilities) are the logits.

  At inference time we do not have to calculate the softmax to know which class has the highest probability. The argmax of the logits will match the argmax after computing the softmax.

Also, note that in some deep learning frameworks, when computing loss functions associated to probabilities, normally the logits are used as input instead of the normalized probabilities, e.g. cross-entropy loss in PyTorch:

The input is expected to contain raw, unnormalized scores for each class