# Linear Regression Optimization

I am learning linear regression right now. In the most of the examples of implementation of this method, which I found, gradient descent is used.

Is there a better way to optimize linear regression than gradient descent?

• For vanilla Ordinary Least Squares (OLS) regression, most software packages do NOT use gradient descent but instead QR decomposition
– Jon
Jul 8 '18 at 22:34
• – Jon
Jul 8 '18 at 22:35

Is there a better way to optimize linear regression than gradient descent?

If by better you mean finding the better separator, no you can't due to the fact that the cost function for linear regression is convex which means there is just one optimal point.

If you want to optimize using different algorithms, there are different kinds of solutions. Gradient-based algorithms like Adam and RMSProp are of those. You also can use normal equation.

• I would consider this an incomplete answer. This solution works only through the perspective of neural networks. For a typical problem, gradient descent would be outperformed by other solutions like QR decomposition or least squares estimation.
– Jon
Jul 8 '18 at 23:33
• I am learning linear regression right now. In the most of the examples of implementation of this method .... The question is about something else, you could add extra details but they do not have any relation to the question. Jul 9 '18 at 1:11
• You might want to review linear regression. The role of gradient descent is for parameter estimation. For parameter estimation there are already well established methods that outperform in most problems. GD is not guaranteed to land on the unbiased estimation, where other methods almost always will (as long as you have >30 samples).
– Jon
Jul 9 '18 at 1:52
• What do you mean by outperform? Linear regression just has one optimal point. By using second order optimizations you can accelerate its convergance. Jul 9 '18 at 1:55