# Why linear regression feature coefficients become super large?

## Introduction

I've implemented linear regression using sklearn and after all calculations I've got results like this:

Feature: 0, coef: -9985335237.46533
Feature: 1, coef: 417387013140.39661
Feature: 2, coef: -2.85809
Feature: 3, coef: 1.50522
Feature: 4, coef: -1.07076


## Data

My data is based on user visits in gym. All data normalized 0 <= x <= 1. Data set has 10k observations.

X:

• feature_0: gym's rating
• feature_1: gym's review(rating) count
• feature_2: gym's one visit price
• feature_3: gym's unlimited subscription price
• feature_4: distance to gym from user's home | calculated min(x / 30, 1.0), because mean is 15.17

Y: user's visit count to that gym

Data sample

## Code

from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from matplotlib import pyplot

# define dataset
# define the model
model = LinearRegression()
# fit the model
model.fit(x, y)
# get importance
importance = model.coef_
# summarize feature importance
for i,v in enumerate(importance):
print('Feature: %0d, coef: %.5f' % (i,v))


## Question

Why linear regression feature coefficients become super large? Is it okay?

Feature: 0, coef: -9985335237.46533
Feature: 1, coef: 417387013140.39661
...


P.S: I'm new to this "part" of StackExchange and ML\DS at all, so please if I do something wrong or I have to provide more information, let me know! Any help would be appreciated. Thanks in advance!

• How big is your dataset? – nwaldo Apr 22 '20 at 22:10
• @nwaldo 10k observations – えるまる Apr 22 '20 at 22:12
• Have you normalized the data before performing regression. – Shubham Panchal Apr 23 '20 at 2:53
• @BenReiniger I’ve just used cross validation dividing to 5 parts and it shows -0.1, seems very bad. – えるまる Apr 23 '20 at 5:40
• A phenomenon called "(multi-)collinearity". – Michael M Apr 23 '20 at 19:36

Large coefficients in linear regression are not necessarily a problem. They can be large becuase some variable was rescaled. You mentionned that you do some rescaling, but provide no details. Therefore it is not possible to tell what exactly is going on.

Here is a (general) example that explains how coefficients can get "large" (in R). Assume we want to model "visits" ($$y$$) contingent on "rating" ($$x$$):

# Data
df = data.frame(c(1,3,5,3,7,5,8,9,7,10),c(34,54,31,45,65,78,56,87,69,134))
colnames(df)<-c("rating","visits")

# Regression 1
reg1 = lm(visits~rating,data=df)
summary(reg1)


The regression results are:

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)   19.452     15.273   1.274   0.2385
rating         7.905      2.379   3.322   0.0105 *


This tells us, that visits increase by about 7.9 when rating increases by one unit. This is basically a linear function with intercept 19.45 and slope 7.9. Since our model is $$y = \beta_0 + \beta_1 x + u ,$$ the corresponding (estimated) linear function would look like: $$f(x) = 19.45 + 7.9 x .$$

We can predict and plot our model. The results are just as expected, a positive linear function.

# Predict and plot
pred1 = predict(reg1,newdata=df)
plot(df$$rating,df$$visits,xlab="Rating",ylab="Visits")
lines(df$rating,pred1)  Now comes the interesting part: I do a linear transformation on $$x$$. Namely, I divide $$x$$ by some "large" number and I run the same regression as before: # Transform x large_integer = 10000000 df$$rating2 = df$$rating/large_integer df rating visits rating2 1 1 34 1e-07 2 3 54 3e-07 3 5 31 5e-07 4 3 45 3e-07 5 7 65 7e-07 6 5 78 5e-07 7 8 56 8e-07 8 9 87 9e-07 9 7 69 7e-07 10 10 134 1e-06 # Regression 2 (with transformed x) reg2 = lm(visits~rating2,data=df) summary(reg2)  The results are: Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 1.945e+01 1.527e+01 1.274 0.2385 rating2 7.905e+07 2.379e+07 3.322 0.0105 *  As you see, the coefficient for rating is rather large now. However, when I predict and plot, I get basically the same results as before. The only thing that has changed is the "scale" of $$x$$ (the way $$x$$ is expressed). Let's compare the coefficient for rating in both regressions. In the first case it was: # Relevant coefficient "rating" from reg1 (the "small" one) reg1$coefficients[2]

rating
7.904762


In the second case it was:

# Relevant coefficient "rating2" from reg2 (the "large" one)
reg2$coefficients[2] rating2 79047619  However, when I divide the coefficient rating2 by the same "large" number as I did to "rescale" the data, I get: # "Rescale" large coefficient reg2$coefficients[2]/large_integer

rating2
7.904762


As you can see, the "rescaled" coefficient rating2 is exactly the same as the original coefficient for rating.

What can you do to check your regression:

• Run the regression without any rescaling and see if the results make sense

• Make a prediction from the regression

• Rescale your data (i.e. "standardise"), which should contribute to get better predictions because data are less "wonky" in this case. However, coefficients have no natural interpretation any more

• Compare standardised data to non-standardised to see how your data changed. Based on the discussion above, you should get a good idea if very small or large coefficients can make sense after standardisation

• Make a prediction, compare to the prediction from above

• Peter, thanks for ur answer! Let me try :) then I will come with some feedback – えるまる Apr 23 '20 at 19:52
• Thanks for your answer, now I have very bad training score(-0.10), can you please share some links on how to improve my prediction? – えるまる Apr 25 '20 at 17:16
• what did you do exactly to end up with this score, and what kind of score is it? R2? – Peter Apr 26 '20 at 21:59