What are the different estimators used by different loss functions in sklearn's SGDClassifier?

Question

I am new to Data Science and was curious to learn about sklearn package. While digging up details on SGDClassifier I found in the documentation that SGDClassifier uses different estimators for different loss functions and have given a example stating when SGDClassifier(loss='log') then Logistic Regression is used. Can I ask what are the estimators for other loss functions like hinge, huber, etc...?

Link of the documentation for quick reference: https://scikit-learn.org/stable/modules/sgd.html#mathematical-formulation

Answer 1

I think it's important to clarify the different terms used here.

1. An estimator refers to the type of model that is being used (e.g. logistic regression, linear regression, support vector machines).

2. The loss function refers to the function you use to quantify how much your model's predictions differ from the target values they're trying to predict (e.g. log loss, mean squared error, cross entropy loss)

3. An optimisation technique is a method for updating the parameters of your model to maximise its performance with respect to some loss function - often referred to as minimising the loss. (e.g Stochastic gradient descent, maximum likelihood estimation).

In the case of SGDClassifier() you are first picking an optimisation method - stochastic gradient descent, and then picking a loss function to optimise, by specifying the loss argument. With this information, scikit-learn then 'autocompletes' the model (estimator) to be used.

The page you linked to answers part of your question about which models correspond to which loss functions:

• Perceptron = Perceptron
• Least-squares = Linear regression
• Hinge = SVM classification
• Epsilon-insensitive = SVM Regression

As for Huber and modified Huber, I believe those are also equivalent to variations of SVM (see here).

Note that the SGDClassifier()/SGDRegressor() classes will not always give you the same results as the model-specific scikit-learn (e.g. LogisticRegressor()), since this former will use Stochastic Gradient Descent for optimisation, while the latter will usually have a different default optimisation method.