I am trying to learn linear regression using ordinary least squares and gradient descent from scratch.

I read the documentation for the Scikit learn function and I do not see a means to adjust the learning rate or the epoch with the sklearn.linear_model.LinearRegression class.

Is there a standard learning rate for the linear regression model?

Epochs I am assuming are determined by the change in the error function and are thus dependent on the dataset, with some predetermined maximum amount before convergence.

Are there any other good packages that have a linear regression model that I can test my own model against with adjustable learning rates and epochs?


A linear regression model $y=\beta X+u$ can be solved in one "round" by using $(X'X)^{-1}X'y=\hat{\beta}$. It can also be solved using gradient descent but there is no need to adjust something like a learning rate or the number of epochs since the solver (usually) converges without much trouble.

Here is a minimal example in R:

x0 <- c(1,1,1,1,1) 
x1 <- c(1,2,3,4,5)
x2 <- c(8,4,3,1,8)
x <- as.matrix(cbind(x0,x1,x2))
y <- as.matrix(c(3,7,5,11,14))


# (X'X)^-1 X'y
beta1 = solve(t(x)%*%x) %*% t(x)%*%y 

# R's regression command
beta2 = summary(lm(y ~ x[, 2:3]))

# Gradient decent
m <- nrow(y)
grad <- function(x, y, theta) {
  gradient <- (1/m)* (t(x) %*% ((x %*% t(theta)) - y))

# define gradient descent update algorithm
grad.descent <- function(x, maxit){
  theta <- matrix(c(0, 0, 0), nrow=1) # Initialize the parameters
  alpha = .05 # set learning rate
  for (i in 1:maxit) {
    theta <- theta - alpha  * grad(x, y, theta)   

# results without feature scaling
| improve this answer | |
  • $\begingroup$ Thank you, I wrongly assumed that the scikitlearn system uses gradient descent by default. Ordinary least squares only works with a matrix that you can invert right? Otherwise it won't be solvable? $\endgroup$ – chrisper Sep 6 at 9:17
  • $\begingroup$ I‘m not sure what solver is used with scikit. However OLS can be solved by the standard formula, which requires that X can be inverted. To see the difference between OLS and regression with regulation (as in the SGD case), see ISL, Chapter 6.2., i.e. equation 6.5. Regulation is introduced by an additional penalty term on top of the standard OLS loss function. So the results OLS vs. Lasso etc. are different faculty.marshall.usc.edu/gareth-james/ISL/… $\endgroup$ – Peter Sep 6 at 9:43

You can use SGDRegressor available in scikit learn for adjusting learning rate. It has a variety of parameters you can adjust.

| improve this answer | |
  • $\begingroup$ This is a lasso/ridge/elastic net regressor, not OLS as specified in the question. $\endgroup$ – Peter Sep 5 at 10:36
  • $\begingroup$ @Peter OP's first line says OLS and Gradient Descent and he never said anything about a specific algorithm. OP just wants a Linear Regression model with adjustable learning rate, so I suggested SGD regression. $\endgroup$ – Ankit Seth Sep 6 at 5:47
  • $\begingroup$ OLS is not lasso/ridge/elastic net since the loss function is different. $\endgroup$ – Peter Sep 6 at 8:37
  • $\begingroup$ @Ankit Seth thank you. I will try out the sgd regressor and see how the results compare. $\endgroup$ – chrisper Sep 6 at 9:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.