Is the activation function the only difference between logistic regression and perceptron?

Question

As far as I know, logistic regression can be denoted as:

$$ f(x) = \sigma(w \cdot x + b) $$

A perceptron can be denoted as:

$$ f(x) = \operatorname{sign} (w \cdot x + b) $$

It seems that the only difference between logistic regression and a perceptron model is the activation function.

Is this correct?

Answer 1

TL;DR: Yes and No; they're both similar decision function models but there's more to each model than their main formulation

One could use the logit function as the activation function of a perceptron and consider the output a probability. Yet, that value would likely need a probability calibration.

As with most ML models several things are very similar from model to model, on the other hand varying tiny parameters can result in a different model. Let's take on both sides:

Logit Regression and Perceprton similarities

The logit function is used in logit regression for its properties of being an S-curve, by default valued between 0 and 1. The sign activation function in the perceptron is also shaped like an "S-curve" (with very rough edges - so mathematically not an S-curve by its definition but with similar properties) valued "between" -1 and 1. Another activation function often used with perceptron is hyperbolic tangent (tanh), which is another S-curve - very similar to the sign function but with a rounded shape (and also valued between -1 and 1). We can say that tanh is similar to sign because:

$$ \texttt{sign}(x) \approx \texttt{tanh}(kx) \qquad \text{for k >> 0}\\ or\\ \texttt{sign}(x) = \texttt{tanh}(kx) \qquad k \to \infty $$

So it makes sense to compare tanh with logit as an analogy to compare sign with logit.

Now the logit function is (the one used in logistic regression that is - not the statistical $ln(x/(1- x))$ one):

$$ \texttt{logit}(x) = \frac{L}{1 + e^{-k(x - b)}} $$

Where $L$, $k$ and $b$ are parameters that we can steer. (Since $b$ is a bias term, an unknown constant, it does not matter if we write $- b$ or $b$)

And the hyperbolic tangent is:

$$ \texttt{tanh}(x) = \frac{e^{2x} - 1}{e^{2x} + 1} $$

But wait, if we set the parameters of the logit as $L = 1$, $k = 1$ and $b = 0$, then:

$$ 2 \cdot \texttt{logit}(2x) - 1 = 2 \cdot \frac{1}{1 + e^{-2x}} -1 = \frac{2e^{2x}}{e^{2x} + 1} - 1 = \frac{2e^{2x}}{e^{2x} + 1} - \frac{e^{2x} + 1}{e^{2x} + 1} = \frac{2e^{2x} - e^{2x} - 1}{e^{2x} + 1} = \frac{e^{2x} - 1}{e^{2x} + 1} = \texttt{tanh}(x) $$

So tanh, which is more-or-less a rounded sign, is a scaled and shifted special case of logit. Not only the model differ by the activation function alone but the activation functions are very similar to each other.

Logit Regression and Perceprton differences

Log probabilities (this is probably the major difference and the most important one)

We looked at the logit function above but in reality the logit regression takes the logarithm of the logit instead of plain values for the probabilities. Or, more exactly, in statistics the logit is defined as the natural logarithm of what in most ML implementations is defined as a logit decision function. This is very different from the perceptron which performs the output directly from the activation function.

Regularization

Both models (logit regression and perceptron) are often trained with some form of gradient descent optimizer (be it SGD, have it momentum, or even something else). This training will optimize all parameters ($w$ in your representation or $k$ in mine for logit and the weights in a perceptron) but the model itself will also be given a bunch of hyper-parameters. And in terms of hyper-parameters things do differ:

The perceptron will be trained by optimizing the weights (including a bias) and can be regularized with $L_2$ or $L_1$ (or a combination of both). One may or may not add a bias term.

The actual model of the logistic regression will optimize the logarithm but there's more: most implementations will include a scaling hyper-parameter ($C$) which will multiply the log probabilities. One can use $C$ to regularize the model apart from $L_1$ and/or $L_2$.

Multi-class

The perceptron is always a binary classifier. One sets an output threshold and that works as the decision function (for the sign or tanh functions the threshold is often $0$). For multi-class classification one must build One-Vs-Rest or One-Vs-One groups of models.

Logistic Regression can be used as a binary classifier and in this case can be used for multi-class classification with One-Vs-Rest and One-Vs_one methods. But, there exist a formulation of logistic regression for direct multi-class classification:

$$ \texttt{multinomial logit}(x|y) = \frac{e^{-k_y(x - b)}}{\sum^{K}_{k=0} e^{-c(x - b)}} \qquad \text{for y in }\{0,1,\dots\} $$

i.e. whether $x$ belongs to class $y$. This is performed for each class and the result can be passed to a softmax function. This multi-class formulation can be performed because logit regression deals with log probabilities instead of direct inputs (contrary to a perceptron).