# What optimization algorithms are used in scikit-learn?

Gradient descent is one of the well-known optimisation algorithms. However, are the regression algorithms in scikit-learn implemented with gradient descent or some other techniques?

## 1 Answer

There are multiple approaches to optimization in scikit-learn. I will focus on generalized linear models, where a vector of coefficients needs to be estimated:

• LinearRegression and Ridge use closed-form solution $\beta=(X^TX+I\lambda)^{-1}X^TY$, but Ridge can also use stochastic gradient descent or method of conjugate gradients
• Lasso and ElasticNet use coordinate descent
• OrthogonalMatchingPursuit uses a greedy algorithm with the same name, that has $L_0$ penalty on coefficients
• ARDRegression and BayesianRidge use something like EM algorithm
• SGDRegressor and PassiveAggressiveRegressor use guess what! Stochastic gradient descent.
• HuberRegressor uses BFGS (a second-order optimization method)