5
$\begingroup$

I wanted to put together an example notebook to demonstrate how regularization makes an impact for such a simple model as a simple linear regression. When executing the below script though, I notice that the LinearRegression() and Ridge() models both return the same coef_ and intercept_ and hence, the same regression plots, no matter how I change the regularization strength.

The data is a small dataset containing GDP per capita values in U.S. dollars for 2021 and life satisfaction scores, for a set of countries (n=40). The dataset plotted with the regression line(s) looks like this:

enter image description here

In Aurélien Géron's Hands-On Machine Learning with Scikit-Learn and TensorFlow book (page 27), I read the following:

Constraining a model to make it simpler and reduce the risk of overfitting is called regularization. For example, the linear model we defined earlier has two parameters, θ0 and θ1. This gives the learning algorithm two degrees of freedom to adapt the model to the training data: it can tweak both the height (θ0) and the slope (θ1) of the line. (...) If we allow the algorithm to modify θ1 but we force it to keep it small, then the learning algorithm will produce a simpler model than with two degrees of freedom, but more complex than with just one.

Based on this, I had the understanding that by including the alpha hyperparameter (i.e., the regularization lambda), I am essentially putting an additional constraint on the squared sum of coef_ and intercept_, thereby forcing the model to adjust them during training, since the new cost function now includes the lambda-term. In my current setting, coef_ (slope) is in the range of 1e-5 while intercept_ is around 5.75. Their squared sum is around 33. I would expect the regularized model to decrease the intercept_ because that has a much higher impact on the squared sum, than coef_ does. Yet, when running Ridge(alpha=...) with any alpha value, the learned parameters (and hence the regression plot) are identical to those of the unregularized simple linear regression.

Do I misunderstand something fundamental here? How could I see the effects of regularization on such a model?

My full code:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression, Ridge

X = np.c_[df['GDP_2021']]
y = np.c_[df['LifeSatisf']]

model1 = LinearRegression()
model1.fit(X, y)

model2 = Ridge(alpha=100)
model2.fit(X, y)

print("Coefs model 1: ", model1.coef_, model1.intercept_)
print("Coefs model 2: ", model2.coef_, model2.intercept_)

x_new = np.linspace(0, 140000, 10000)[:, np.newaxis]
y_1 = model1.predict(x_new).flatten().tolist()
y_2 = model2.predict(x_new).flatten().tolist()

plt.figure(figsize=(10, 3))
ax = plt.axes()
ax.scatter(df['GDP_2021'], df['LifeSatisf']);
ax.plot(x_new, y_1, c='r', linestyle='--');
ax.plot(x_new, y_2, c='b', linestyle='--');
plt.title("Life satisfaction vs. GDP per capita regression models");
plt.xlabel("GDP per capita (2021) [US$]");
plt.ylabel("Life satisfaction [-]");

Dataset as a comma separated file:

Country,LifeSatisf,GDP_2021
Australia,7.3,62618.59
Austria,7.1,53793.37
Belgium,6.9,50412.71
Canada,7.4,52791.23
Czech Republic,6.7,25806.38
Denmark,7.6,67919.59
Finland,7.6,53522.57
France,6.5,45028.27
Germany,7.0,50787.86
Greece,5.4,19827.16
Hungary,5.6,18527.59
Iceland,7.5,68843.65
Ireland,7.0,102394.02
Italy,6.0,35584.88
Japan,5.9,40704.3
Korea,5.9,35195.52
Luxembourg,6.9,131301.6
Mexico,6.5,9967.39
Netherlands,7.4,57714.88
New Zealand,7.3,48348.99
Norway,7.6,82244.23
Poland,6.1,17318.5
Portugal,5.4,24457.14
Slovak Republic,6.2,21383.29
Spain,6.3,30536.86
Sweden,7.3,58639.19
Switzerland,7.5,93515.48
Turkey,5.5,9406.58
United Kingdom,6.8,46200.26
United States,6.9,69375.38
Brazil,6.4,7741.15
Chile,6.5,16799.37
Estonia,5.7,27100.74
Israel,7.2,49840.25
Latvia,5.9,19538.9
Russia,5.8,11273.24
Slovenia,5.9,28939.27
South Africa,4.7,6861.17
Colombia,6.3,5892.14
Lithuania,5.9,22411.65
$\endgroup$

1 Answer 1

5
$\begingroup$

Your understanding of regularization is completely correct, it just seems that the value for alpha you are using is too low for this example to have any meaningful impact. Increasing the value for alpha to something much larger (10e9 for the example below) does show the impact of regularization on the parameters of a model: enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.