Gradient descent is one of the well-known optimisation algorithms. However, are the regression algorithms in scikit-learn implemented with gradient descent or some other techniques?


1 Answer 1


There are multiple approaches to optimization in scikit-learn. I will focus on generalized linear models, where a vector of coefficients needs to be estimated:

  • LinearRegression and Ridge use closed-form solution $\beta=(X^TX+I\lambda)^{-1}X^TY$, but Ridge can also use stochastic gradient descent or method of conjugate gradients
  • Lasso and ElasticNet use coordinate descent
  • OrthogonalMatchingPursuit uses a greedy algorithm with the same name, that has $L_0$ penalty on coefficients
  • ARDRegression and BayesianRidge use something like EM algorithm
  • SGDRegressor and PassiveAggressiveRegressor use guess what! Stochastic gradient descent.
  • HuberRegressor uses BFGS (a second-order optimization method)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.