1
$\begingroup$

As far as I know, logistic regression can be denoted as:

$$ f(x) = \sigma(w \cdot x + b) $$

A perceptron can be denoted as:

$$ f(x) = \operatorname{sign} (w \cdot x + b) $$

It seems that the only difference between logistic regression and a perceptron model is the activation function.

Is this correct?

$\endgroup$
1
$\begingroup$

LT;DR: Yes and No: they're both similar decision function models but there's more to each model than their main formulation

One could use the logit function as the activation function of a perceptron and consider the output a probability. Yet, that value would likely need a probability calibration.

As with most ML models several things are very similar from model to model, on the other hand varying tiny parameters can result in a different model. Let's take on both sides:

Logit Regression and Perceprton similarities

The logit function is used in logit regression for its properties of being an S-curve, by default valued between 0 and 1. The sign activation function in the perceptron is also shaped like an "S-curve" (with very rough edges - so mathematically not an S-curve by its definition but with similar properties) valued "between" -1 and 1. Another activation function often used with perceptrons is hyperbolic tangent (tanh), which is another S-curve - very similar to the sign function but with a rounded shape (and also valued between -1 and 1). We can say that tanh is similar to sign because:

$$ \texttt{sign}(x) \approx \texttt{tanh}(kx) \qquad \text{for k >> 0}\\ or\\ \texttt{sign}(x) = \texttt{tanh}(kx) \qquad k \to \infty $$

So it makes sense to compare tanh with logit as an analogy to compare sign with logit.

Now the logit function is (the one used in logistic regression that is - not the statistical $ln(x/(1- x))$ one):

$$ \texttt{logit}(x) = \frac{L}{1 + e^{-k(x - b)}} $$

Where $L$, $k$ and $b$ are parameters that we can steer. (Since $b$ is a bias term, an unknown constant, it does not matter if we write $- b$ or $b$)

And the hyperbolic tangent is:

$$ \texttt{tanh}(x) = \frac{e^{2x} - 1}{e^{2x} + 1} $$

But wait, if we set the parameters of the logit as $L = 1$, $k = 1$ and $b = 0$, then:

$$ 2 \cdot \texttt{logit}(2x) - 1 = 2 \cdot \frac{1}{1 + e^{-2x}} -1 = \frac{2e^{2x}}{e^{2x} + 1} - 1 = \frac{2e^{2x}}{e^{2x} + 1} - \frac{e^{2x} + 1}{e^{2x} + 1} = \frac{2e^{2x} - e^{2x} - 1}{e^{2x} + 1} = \frac{e^{2x} - 1}{e^{2x} + 1} = \texttt{tanh}(x) $$

So tanh, which is more-or-less a rounded sign, is a scaled and shifted special case of logit. Not only the model differ by the activation function alone but the activation functions are very similar to each other.

Logit Regression and Perceprton differences

Log probabilities (this is probably the major difference and the most important one)

We looked at the logit function above but in reality the logit regression takes the logarthm of the logit instead of plain values for the probabilities. Or, more exactly, in statistics the logit is defined as the natural logarithm of what in most ML implementations is defined as a logit decision function. This is very different from the perceptron which performs the output directly from the activation function.

Regularization

Both models (logit regression and perceptron) are often trained with some form of gradient descent optimizer (be it SGD, have it momentum, or even something else). This training will optimize all parameters ($w$ in your representation or $k$ in mine for logit and the weights in a perceptron) but the model itself will also be given a bunch of hyperparameters. And in terms of hyperparameters things do differ:

The perceptron will be trained by optimizing the weights (including a bias) and can be regularized with $L_2$ or $L_1$ (or a combination of both). One may or may not add a bias term.

The actual model of the logistic regression will optimize the logarithm but there's more: most implementations will include a scaling hyperparameter ($C$) which will multiply the log probabilities. One can use $C$ to regularize the model apart from $L_1$ and/or $L_2$.

Multiclass

The perceptron is always a binary classifier. One sets an output threshold and that works as the decision function (for the sign or tanh functions the threshold is often $0$). For multiclass classification one must build One-Vs-Rest or One-Vs-One groups of models.

Logistic Regression can be used as a binary classifier and in this case can be used for multiclass classification with One-Vs-Rest and One-Vs_one methods. But, there exist a formulation of logistic regression for direct multiclass classification:

$$ \texttt{multinomial logit}(x|y) = \frac{e^{-k_y(x - b)}}{\sum^{K}_{k=0} e^{-c(x - b)}} \qquad \text{for y in }\{0,1,\dots\} $$

i.e. whether $x$ belongs to class $y$. This is performed for each class and the result can be passed to a softmax function. This multiclass formulation can be performed because logit regression deals with log probabilities instead of direct inputs (contrary to a perceptron).

$\endgroup$
  • $\begingroup$ thanks for your answer. what does LT;DR stands for? $\endgroup$ – Jay Jun 9 at 5:10
  • $\begingroup$ @Jay - to confuse the shit out of people. I wrote that at 4AM for me :). I have clarified a good deal and added sentences to deal with some inaccuracies. The general idea is that both models are very similar alright: both are binary classifiers (by default) with a decision function - but since logit has a logarithm that can be fully interpreted as probability then we can do a bunch of tricks with it. $\endgroup$ – grochmal Jun 9 at 10:55
  • $\begingroup$ you are so kind, thank you so much. I guess LT;DR at the beginning of your answer means TL;DR, that is, Too long; but please read. Do I understand this right? $\endgroup$ – Jay Jun 9 at 11:43
  • $\begingroup$ @Jay - oh that! LT;DR -> too long; didn't read - on SO/SE it is used to indicate that the answer is long so here is a summary sentence - then i'll expand on it. (therefore someone may read just that sentence and be blissfully ignorant of the details if he so wants). $\endgroup$ – grochmal Jun 9 at 11:48
  • 1
    $\begingroup$ @Jay - You are correct. I guess the avoidance of writing S-curve by writing S curve is not a good enough distinction. Clarified properly and added the s-curve notation. And yep, sign(x) is definitely not differentiable. $\endgroup$ – grochmal Jun 9 at 13:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.