Is the prediction algorithm absolutely the same for all linear classifiers?

Question

Is the prediction algorithm absolutely the same for all linear classifiers and linear regression algorithms?

As known, any linear classifier can be described as: y = w1*x1 + w2*x2 + ... + c

There are two broad classes of methods for determining the parameters of a linear classifier (generative and discriminative): https://en.wikipedia.org/wiki/Linear_classifier

1. Linear Discriminant Analysis (or Fisher's linear discriminant), Naive Bayes classifier
2. Logistic regression, Perceptron, Support vector machine

---

Question: Is it true that linear classifiers differ only in the Learning algorithm, but do they do the same during Prediction y = w1*x1 + w2*x2 + ... + c?

• If I used one method for Training (for example SVM with linear kernel function), then can I use other method for prediction (for example Perceptron) with the same output result?

• And can I do the same, if I used SVR (Support Vector Regression with linear kernel function) for Training, then can I use Perceptor as linear regression method for Prediction value?

Answer 1

Is it true that linear classifiers differ only in the Learning algorithm, but do they do the same during Prediction y = w1*x1 + w2*x2 + ... + c?

Yes, all parametric linear classifiers try to predict the weights for the same equation $y = w1*x1 + w2*x2 + ... + c$ and differ in how they try to optimize the cost function.

If I used one method for Training (for example SVM with linear kernel function), then can I use other method for prediction (for example Perceptron) with the same output result?

You will not be able to achieve this using standard libraries. But, if you code your models from scratch and extract weights from one model after training and use those weights to make a prediction with a different model, you'll get a sufficiently close answer. This is due to the fact that once you have learned the weights $(w1, w2,...., wn)$, you know the nature of data and can predict the target variable using the equation $y = w1*x1 + w2*x2 + ... + c$. Note that, this will be true only in the case of parametric linear models as only then the above equation would be applicable.

And can I do the same, if I used SVR (Support Vector Regression with linear kernel function) for Training, then can I use Perceptor as linear regression method for Prediction value? Well, yes again with the same restrictions as above. This is because you will get a decision boundary as the output which depends on the data. Note that, there will be minor changes in the output as the error of different models is different.

Also, any other differences in the output will be due to the difference in the learning algorithm and how it operates on the data. You can verify this by training two models on the same data and extracting the weights after training. If the weights are close enough (which means you have learned the correct underlying pattern), you can use any of the two models on test data to predict and you will get similar values.

Answer 2

If I understand well, you are asking if all the linear classifier algorithm are equivalent. The answer is no. Otherwise, what would be the point of having different algorithm if all of them enable you to predict the same thing. If you want to convince yourself about that have a try with scikit learn

Answer 3

In general case it is not true. The linear equation parameters estimations depend on the mathematical model of classifier and assumptions made on data (and in practice, on the optimization algorithm applied to solve the learning problem). The following function \begin{equation} f(\sum_i x_iw_i + c), \end{equation} has a strong influence on the parameters of the linear equation. For example, using the sigmoid function in perceptron could result in getting the model parameters close to the logistic regression ones. But another activation function would result in getting linear equation parameters estimation different from those.

Answer 4

No. I will add another point to what is already said, and go in some details to justify my answer.

Algorithms are not the same because:

1. the regulatization parameters are different. In scikit-learn object models we have the ones which use alpha and the one wich use C. The more alpha is high the more the computed model will be simple. For C, that is the contrary: the more C is low the more the computed model will be simple.
2. Ridge Regression uses the same formula as Linear Regression (Ordinary Least Squares in scikit-learn) but there is a constraint added: all $w$ must be the lowest possible. Meaning the closest to 0. Even Lasso uses the same formula, even follows the same constraint as Ridge. But it admits some $w$ will be exactly 0, due to a specific regularization called L1.

So even close modelisations, shown above, have different algorithms. And I am not considering the different times of execution to make the computation with a same dataset.

Their simalarities are above all the pattern of model: $y=w1∗x1+w2∗x2+...+c$ the linear relation between the variables, and their methods in one library: for instance fit() or predict() available on all scikit-learn object models.