Why linear regression feature coefficients become super large?

Question

Introduction

I've implemented linear regression using sklearn and after all calculations I've got results like this:

Feature: 0, coef: -9985335237.46533
Feature: 1, coef: 417387013140.39661
Feature: 2, coef: -2.85809
Feature: 3, coef: 1.50522
Feature: 4, coef: -1.07076

---

Data

My data is based on user visits in gym. All data normalized 0 <= x <= 1. Data set has 10k observations.

X:

• feature_0: gym's rating
• feature_1: gym's review(rating) count
• feature_2: gym's one visit price
• feature_3: gym's unlimited subscription price
• feature_4: distance to gym from user's home | calculated min(x / 30, 1.0), because mean is 15.17

Y: user's visit count to that gym

Data sample

---

Code

from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from matplotlib import pyplot
from numpy import loadtxt

# define dataset
x = loadtxt('formatted_data_x.txt')
y = loadtxt('formatted_data_y.txt')
# define the model
model = LinearRegression()
# fit the model
model.fit(x, y)
# get importance
importance = model.coef_
# summarize feature importance
for i,v in enumerate(importance):
    print('Feature: %0d, coef: %.5f' % (i,v))

---

Question

Why linear regression feature coefficients become super large? Is it okay?

Feature: 0, coef: -9985335237.46533
Feature: 1, coef: 417387013140.39661
...

---

P.S: I'm new to this "part" of StackExchange and ML\DS at all, so please if I do something wrong or I have to provide more information, let me know! Any help would be appreciated. Thanks in advance!

Answer 1

Large coefficients in linear regression are not necessarily a problem. They can be large becuase some variable was rescaled. You mentionned that you do some rescaling, but provide no details. Therefore it is not possible to tell what exactly is going on.

Here is a (general) example that explains how coefficients can get "large" (in R). Assume we want to model "visits" ($y$) contingent on "rating" ($x$):

# Data
df = data.frame(c(1,3,5,3,7,5,8,9,7,10),c(34,54,31,45,65,78,56,87,69,134))
colnames(df)<-c("rating","visits")

# Regression 1
reg1 = lm(visits~rating,data=df)
summary(reg1)

The regression results are:

Coefficients:
            Estimate Std. Error t value Pr(>|t|)  
(Intercept)   19.452     15.273   1.274   0.2385  
rating         7.905      2.379   3.322   0.0105 *

This tells us, that visits increase by about 7.9 when rating increases by one unit. This is basically a linear function with intercept 19.45 and slope 7.9. Since our model is $$ y = \beta_0 + \beta_1 x + u ,$$ the corresponding (estimated) linear function would look like: $$f(x) = 19.45 + 7.9 x .$$

We can predict and plot our model. The results are just as expected, a positive linear function.

# Predict and plot
pred1 = predict(reg1,newdata=df)
plot(df$rating,df$visits,xlab="Rating",ylab="Visits")
lines(df$rating,pred1)

Now comes the interesting part: I do a linear transformation on $x$. Namely, I divide $x$ by some "large" number and I run the same regression as before:

# Transform x
large_integer = 10000000
df$rating2 = df$rating/large_integer
df

   rating visits rating2
1       1     34   1e-07
2       3     54   3e-07
3       5     31   5e-07
4       3     45   3e-07
5       7     65   7e-07
6       5     78   5e-07
7       8     56   8e-07
8       9     87   9e-07
9       7     69   7e-07
10     10    134   1e-06

# Regression 2 (with transformed x)
reg2 = lm(visits~rating2,data=df)
summary(reg2)

The results are:

Coefficients:
             Estimate Std. Error t value Pr(>|t|)  
(Intercept) 1.945e+01  1.527e+01   1.274   0.2385  
rating2     7.905e+07  2.379e+07   3.322   0.0105 *

As you see, the coefficient for rating is rather large now. However, when I predict and plot, I get basically the same results as before. The only thing that has changed is the "scale" of $x$ (the way $x$ is expressed).

Let's compare the coefficient for rating in both regressions.

In the first case it was:

# Relevant coefficient "rating" from reg1 (the "small" one)
reg1$coefficients[2]

  rating 
7.904762 

In the second case it was:

# Relevant coefficient "rating2" from reg2 (the "large" one)
reg2$coefficients[2]

 rating2 
79047619

However, when I divide the coefficient rating2 by the same "large" number as I did to "rescale" the data, I get:

# "Rescale" large coefficient
reg2$coefficients[2]/large_integer

 rating2 
7.904762

As you can see, the "rescaled" coefficient rating2 is exactly the same as the original coefficient for rating.

What can you do to check your regression:

• Run the regression without any rescaling and see if the results make sense

• Make a prediction from the regression

• Rescale your data (i.e. "standardise"), which should contribute to get better predictions because data are less "wonky" in this case. However, coefficients have no natural interpretation any more

• Compare standardised data to non-standardised to see how your data changed. Based on the discussion above, you should get a good idea if very small or large coefficients can make sense after standardisation

• Make a prediction, compare to the prediction from above