1
$\begingroup$

Introduction

I've implemented linear regression using sklearn and after all calculations I've got results like this:

Feature: 0, coef: -9985335237.46533
Feature: 1, coef: 417387013140.39661
Feature: 2, coef: -2.85809
Feature: 3, coef: 1.50522
Feature: 4, coef: -1.07076

Data

My data is based on user visits in gym. All data normalized 0 <= x <= 1. Data set has 10k observations.

X:

  • feature_0: gym's rating
  • feature_1: gym's review(rating) count
  • feature_2: gym's one visit price
  • feature_3: gym's unlimited subscription price
  • feature_4: distance to gym from user's home | calculated min(x / 30, 1.0), because mean is 15.17

Y: user's visit count to that gym

Data sample


Code

from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from matplotlib import pyplot
from numpy import loadtxt

# define dataset
x = loadtxt('formatted_data_x.txt')
y = loadtxt('formatted_data_y.txt')
# define the model
model = LinearRegression()
# fit the model
model.fit(x, y)
# get importance
importance = model.coef_
# summarize feature importance
for i,v in enumerate(importance):
    print('Feature: %0d, coef: %.5f' % (i,v))

Question

Why linear regression feature coefficients become super large? Is it okay?

Feature: 0, coef: -9985335237.46533
Feature: 1, coef: 417387013140.39661
...

P.S: I'm new to this "part" of StackExchange and ML\DS at all, so please if I do something wrong or I have to provide more information, let me know! Any help would be appreciated. Thanks in advance!

$\endgroup$
12
  • $\begingroup$ How big is your dataset? $\endgroup$
    – nwaldo
    Apr 22, 2020 at 22:10
  • $\begingroup$ @nwaldo 10k observations $\endgroup$ Apr 22, 2020 at 22:12
  • $\begingroup$ Have you normalized the data before performing regression. $\endgroup$ Apr 23, 2020 at 2:53
  • $\begingroup$ @BenReiniger I’ve just used cross validation dividing to 5 parts and it shows -0.1, seems very bad. $\endgroup$ Apr 23, 2020 at 5:40
  • 1
    $\begingroup$ A phenomenon called "(multi-)collinearity". $\endgroup$
    – Michael M
    Apr 23, 2020 at 19:36

1 Answer 1

4
$\begingroup$

Large coefficients in linear regression are not necessarily a problem. They can be large becuase some variable was rescaled. You mentionned that you do some rescaling, but provide no details. Therefore it is not possible to tell what exactly is going on.

Here is a (general) example that explains how coefficients can get "large" (in R). Assume we want to model "visits" ($y$) contingent on "rating" ($x$):

# Data
df = data.frame(c(1,3,5,3,7,5,8,9,7,10),c(34,54,31,45,65,78,56,87,69,134))
colnames(df)<-c("rating","visits")

# Regression 1
reg1 = lm(visits~rating,data=df)
summary(reg1)

The regression results are:

Coefficients:
            Estimate Std. Error t value Pr(>|t|)  
(Intercept)   19.452     15.273   1.274   0.2385  
rating         7.905      2.379   3.322   0.0105 *

This tells us, that visits increase by about 7.9 when rating increases by one unit. This is basically a linear function with intercept 19.45 and slope 7.9. Since our model is $$ y = \beta_0 + \beta_1 x + u ,$$ the corresponding (estimated) linear function would look like: $$f(x) = 19.45 + 7.9 x .$$

We can predict and plot our model. The results are just as expected, a positive linear function.

# Predict and plot
pred1 = predict(reg1,newdata=df)
plot(df$rating,df$visits,xlab="Rating",ylab="Visits")
lines(df$rating,pred1)

enter image description here

Now comes the interesting part: I do a linear transformation on $x$. Namely, I divide $x$ by some "large" number and I run the same regression as before:

# Transform x
large_integer = 10000000
df$rating2 = df$rating/large_integer
df

   rating visits rating2
1       1     34   1e-07
2       3     54   3e-07
3       5     31   5e-07
4       3     45   3e-07
5       7     65   7e-07
6       5     78   5e-07
7       8     56   8e-07
8       9     87   9e-07
9       7     69   7e-07
10     10    134   1e-06

# Regression 2 (with transformed x)
reg2 = lm(visits~rating2,data=df)
summary(reg2)

The results are:

Coefficients:
             Estimate Std. Error t value Pr(>|t|)  
(Intercept) 1.945e+01  1.527e+01   1.274   0.2385  
rating2     7.905e+07  2.379e+07   3.322   0.0105 *

As you see, the coefficient for rating is rather large now. However, when I predict and plot, I get basically the same results as before. The only thing that has changed is the "scale" of $x$ (the way $x$ is expressed).

enter image description here

Let's compare the coefficient for rating in both regressions.

In the first case it was:

# Relevant coefficient "rating" from reg1 (the "small" one)
reg1$coefficients[2]

  rating 
7.904762 

In the second case it was:

# Relevant coefficient "rating2" from reg2 (the "large" one)
reg2$coefficients[2]

 rating2 
79047619

However, when I divide the coefficient rating2 by the same "large" number as I did to "rescale" the data, I get:

# "Rescale" large coefficient
reg2$coefficients[2]/large_integer

 rating2 
7.904762

As you can see, the "rescaled" coefficient rating2 is exactly the same as the original coefficient for rating.

What can you do to check your regression:

  • Run the regression without any rescaling and see if the results make sense

  • Make a prediction from the regression

  • Rescale your data (i.e. "standardise"), which should contribute to get better predictions because data are less "wonky" in this case. However, coefficients have no natural interpretation any more

  • Compare standardised data to non-standardised to see how your data changed. Based on the discussion above, you should get a good idea if very small or large coefficients can make sense after standardisation

  • Make a prediction, compare to the prediction from above

$\endgroup$
3
  • $\begingroup$ Peter, thanks for ur answer! Let me try :) then I will come with some feedback $\endgroup$ Apr 23, 2020 at 19:52
  • $\begingroup$ Thanks for your answer, now I have very bad training score(-0.10), can you please share some links on how to improve my prediction? $\endgroup$ Apr 25, 2020 at 17:16
  • $\begingroup$ what did you do exactly to end up with this score, and what kind of score is it? R2? $\endgroup$
    – Peter
    Apr 26, 2020 at 21:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.