Return to Answer

As other answers state, "logits" refers to unnormalized log-probabilities. However, what does that mean?

The term "logit" is used in machine learning models that output probabilities, that is, numbers between 0 and 1. The most prominent ones are classification models, either binary classification or multi-class classification:

• Binary classification models tell whether the input belongs or not to the positive class, that is, they generate a single number between 0 and 1 representing the probability of the input belonging to the positive class, that is, they model a Bernoulli distribution conditioned on the input.

  Normally, the model generates an unbounded real number that is then "squashed" into the $(0, 1)$ range with the sigmoid function: $\sigma(x) = \frac{1}{1 + e^{-x}}$. The unbounded real number (i.e. the unnormalized log-probability) is the logit.

  Note that, at inference time, in order to know if the probability is greater than 0.5, we don't need to compute the sigmoid: by the nature of the sigmoid, if the logit is greater than 0, then the probability computed with the sigmoid will be greater than 0.5.

• Multiclass classification models tell to which class the input belongs, among a set of pre-defined N classes, that is, they generate N probabilities, each between 0 and 1, and they all add up to 1. This way, they model a categorical distribution (sometimes referred to as "multinomial") conditioned on the input.

  Normally, the model generates N unbounded real numbers whose range and sum is normalized to $(0, 1)$ by means of the softmax function: $\sigma (z)_i = \frac{e^{z_i}}{\sum_{j=1}^N e^{z_j}}$. The N unbounded real numbers (i.e. the unnormalized probabilities) are the logits.

  To know which class has the highest probability, we do not have to calculate the softmax at inference time: the argmax of the logits will match the argmax after computing the softmax.

Also, note that in some deep learning frameworks, when computing loss functions associated to probabilities, normally the logits are used as input instead of the normalized probabilities, e.g. cross-entropy loss in PyTorch:

The input is expected to contain raw, unnormalized scores for each class

As other answers state, "logits" refers to unnormalized log-probabilities. However, what does that mean?

The term "logit" is used in machine learning models that output probabilities, that is, numbers between 0 and 1. The most prominent ones are classification models, either binary classification or multi-class classification:

• Binary classification models tell whether the input belongs or not to the positive class, that is, they generate a single number between 0 and 1 representing the probability of the input belonging to the positive class, that is, they model a Bernoulli distribution conditioned on the input.

  Normally, the model generates an unbounded real number that is then "squashed" into the $(0, 1)$ range with the sigmoid function: $\sigma(x) = \frac{1}{1 + e^{-x}}$. The unbounded real number (i.e. the unnormalized log-probability) is the logit.

  Note that, at inference time, in order to know if the probability is greater than 0.5, we don't need to compute the sigmoid: by the nature of the sigmoid, if the logit is greater than 0, then the probability computed with the sigmoid will be greater than 0.5.

• Multiclass classification models tell to which class the input belongs, among a set of pre-defined N classes, that is, they generate N probabilities, each between 0 and 1, and they all add up to 1. This way, they model a categorical distribution (sometimes referred to as "multinomial") conditioned on the input.

  Normally, the model generates N unbounded real numbers whose range and sum is normalized to $(0, 1)$ by means of the softmax function: $\sigma (z)_i = \frac{e^{z_i}}{\sum_{j=1}^N e^{z_j}}$. The N unbounded real numbers (i.e. the unnormalized probabilities) are the logits.

  At inference time we do not have to calculate the softmax to know which class has the highest probability. The argmax of the logits will match the argmax after computing the softmax.

Also, note that in some deep learning frameworks, when computing loss functions associated to probabilities, normally the logits are used as input instead of the normalized probabilities, e.g. cross-entropy loss in PyTorch:

The input is expected to contain raw, unnormalized scores for each class

As other answers state, "logits" refers to unnormalized log-probabilities. However, what does that mean?

The term "logit" is used in machine learning models that output probabilities, that is, numbers between 0 and 1. The most prominent ones are classification models, either binary classification or multi-class classification:

• Binary classification models tell whether the input belongs or not to the positive class, that is, they generate a single number between 0 and 1 representing the probability of the input belonging to the positive class, that is, they model a Bernoulli distribution conditioned on the input.

  Normally, the model generates an unbounded real number that is then "squashed" into the $(0, 1)$ range with the sigmoid function: $\sigma(x) = \frac{1}{1 + e^{-x}}$. The unbounded real number (i.e. the unnormalized log-probability) is the logit.

  Note that, at inference time, in order to know if the probability is greater than 0.5, we don't need to compute the sigmoid: by the nature of the sigmoid, if the logit is greater than 0, then the probability computed with the sigmoid will be greater than 0.5.

• Multiclass classification models tell to which class the input belongs, among a set of pre-defined N classes, that is, they generate N probabilities, each between 0 and 1, and they all add up to 1. This way, they model a categorical distribution (sometimes referred to as "multinomial") conditioned on the input.

  Normally, the model generates N unbounded real numbers whose range and sum is normalized to $(0, 1)$ by means of the softmax function: $\sigma (z)_i = \frac{e^{z_i}}{\sum_{j=1}^K e^{z_j}}$. The N unbounded real numbers (i.e. the unnormalized probabilities) are the logits.

  Note that, at inference time, to know which class has the highest probability, we don't need to compute the softmax: by the nature of the softmax, we can simply take the argmax of the logits, and it will match the argmax after computing the softmax.

Also, note that in some deep learning frameworks, when computing loss functions associated to probabilities, normally the logits are used as input instead of the normalized probabilities, e.g. cross-entropy loss in PyTorch:

The input is expected to contain raw, unnormalized scores for each class

As other answers state, "logits" refers to unnormalized log-probabilities. However, what does that mean?

The term "logit" is used in machine learning models that output probabilities, that is, numbers between 0 and 1. The most prominent ones are classification models, either binary classification or multi-class classification:

• Binary classification models tell whether the input belongs or not to the positive class, that is, they generate a single number between 0 and 1 representing the probability of the input belonging to the positive class, that is, they model a Bernoulli distribution conditioned on the input.

  Normally, the model generates an unbounded real number that is then "squashed" into the $(0, 1)$ range with the sigmoid function: $\sigma(x) = \frac{1}{1 + e^{-x}}$. The unbounded real number (i.e. the unnormalized log-probability) is the logit.

  Note that, at inference time, in order to know if the probability is greater than 0.5, we don't need to compute the sigmoid: by the nature of the sigmoid, if the logit is greater than 0, then the probability computed with the sigmoid will be greater than 0.5.

• Multiclass classification models tell to which class the input belongs, among a set of pre-defined N classes, that is, they generate N probabilities, each between 0 and 1, and they all add up to 1. This way, they model a categorical distribution (sometimes referred to as "multinomial") conditioned on the input.

  Normally, the model generates N unbounded real numbers whose range and sum is normalized to $(0, 1)$ by means of the softmax function: $\sigma (z)_i = \frac{e^{z_i}}{\sum_{j=1}^N e^{z_j}}$. The N unbounded real numbers (i.e. the unnormalized probabilities) are the logits.

  To know which class has the highest probability, we do not have to calculate the softmax at inference time: the argmax of the logits will match the argmax after computing the softmax.

Also, note that in some deep learning frameworks, when computing loss functions associated to probabilities, normally the logits are used as input instead of the normalized probabilities, e.g. cross-entropy loss in PyTorch:

The input is expected to contain raw, unnormalized scores for each class

As other answers state, "logits" refers to unnormalized probabilities. However, what does that mean?

The term "logit" is used in machine learning models that output probabilities, that is, numbers between 0 and 1. The most prominent ones are classification models, either binary classification or multi-class classification:

• Binary classification models tell whether the input belongs or not to the positive class, that is, they generate a single number between 0 and 1 representing the probability of the input belonging to the positive class, that is, they model a Bernoulli distribution conditioned on the input.

  Normally, the model generates an unbounded real number that is then "squashed" into the $(0, 1)$ range with the sigmoid function. The unbounded real number (i.e. the unnormalized probability) is the logit.

  Note that, at inference time, in order to know if the probability is greater than 0.5, we don't need to compute the sigmoid: by the nature of the sigmoid, if the logit is greater than 0, then the probability computed with the sigmoid will be greater than 0.5.

• Multiclass classification models tell to which class the input belongs, among a set of pre-defined N classes, that is, they generate N probabilities, each between 0 and 1, and they all add up to 1. This way, they model a categorical distribution (sometimes referred to as "multinomial") conditioned on the input.

  Normally, the model generates N unbounded real numbers whose range and sum is normalized to $(0, 1)$ by means of the softmax function. The N unbounded real numbers (i.e. the unnormalized probabilities) are the logits.

  Note that, at inference time, to know which class has the highest probability, we don't need to compute the softmax: by the nature of the softmax, we can simply take the argmax of the logits, and it will match the argmax after computing the softmax.

Also, note that in some deep learning frameworks, when computing loss functions associated to probabilities, normally the logits are used as input instead of the normalized probabilities, e.g. cross-entropy loss in PyTorch:

The input is expected to contain raw, unnormalized scores for each class

As other answers state, "logits" refers to unnormalized log-probabilities. However, what does that mean?

The term "logit" is used in machine learning models that output probabilities, that is, numbers between 0 and 1. The most prominent ones are classification models, either binary classification or multi-class classification:

• Binary classification models tell whether the input belongs or not to the positive class, that is, they generate a single number between 0 and 1 representing the probability of the input belonging to the positive class, that is, they model a Bernoulli distribution conditioned on the input.

  Normally, the model generates an unbounded real number that is then "squashed" into the $(0, 1)$ range with the sigmoid function: $\sigma(x) = \frac{1}{1 + e^{-x}}$. The unbounded real number (i.e. the unnormalized log-probability) is the logit.

  Note that, at inference time, in order to know if the probability is greater than 0.5, we don't need to compute the sigmoid: by the nature of the sigmoid, if the logit is greater than 0, then the probability computed with the sigmoid will be greater than 0.5.

• Multiclass classification models tell to which class the input belongs, among a set of pre-defined N classes, that is, they generate N probabilities, each between 0 and 1, and they all add up to 1. This way, they model a categorical distribution (sometimes referred to as "multinomial") conditioned on the input.

  Normally, the model generates N unbounded real numbers whose range and sum is normalized to $(0, 1)$ by means of the softmax function: $\sigma (z)_i = \frac{e^{z_i}}{\sum_{j=1}^K e^{z_j}}$. The N unbounded real numbers (i.e. the unnormalized probabilities) are the logits.

  Note that, at inference time, to know which class has the highest probability, we don't need to compute the softmax: by the nature of the softmax, we can simply take the argmax of the logits, and it will match the argmax after computing the softmax.

Also, note that in some deep learning frameworks, when computing loss functions associated to probabilities, normally the logits are used as input instead of the normalized probabilities, e.g. cross-entropy loss in PyTorch:

The input is expected to contain raw, unnormalized scores for each class