I wanted to put together an example notebook to demonstrate how regularization makes an impact for such a simple model as a simple linear regression. When executing the below script though, I notice that the LinearRegression() and Ridge() models both return the same coef_ and intercept_ and hence, the same regression plots, no matter how I change the regularization strength.
The data is a small dataset containing GDP per capita values in U.S. dollars for 2021 and life satisfaction scores, for a set of countries (n=40). The dataset plotted with the regression line(s) looks like this:
In Aurélien Géron's Hands-On Machine Learning with Scikit-Learn and TensorFlow book (page 27), I read the following:
Constraining a model to make it simpler and reduce the risk of overfitting is called regularization. For example, the linear model we defined earlier has two parameters, θ0 and θ1. This gives the learning algorithm two degrees of freedom to adapt the model to the training data: it can tweak both the height (θ0) and the slope (θ1) of the line. (...) If we allow the algorithm to modify θ1 but we force it to keep it small, then the learning algorithm will produce a simpler model than with two degrees of freedom, but more complex than with just one.
Based on this, I had the understanding that by including the alpha hyperparameter (i.e., the regularization lambda), I am essentially putting an additional constraint on the squared sum of coef_ and intercept_, thereby forcing the model to adjust them during training, since the new cost function now includes the lambda-term. In my current setting, coef_ (slope) is in the range of 1e-5 while intercept_ is around 5.75. Their squared sum is around 33. I would expect the regularized model to decrease the intercept_ because that has a much higher impact on the squared sum, than coef_ does. Yet, when running Ridge(alpha=...) with any alpha value, the learned parameters (and hence the regression plot) are identical to those of the unregularized simple linear regression.
Do I misunderstand something fundamental here? How could I see the effects of regularization on such a model?
My full code:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression, Ridge
X = np.c_[df['GDP_2021']]
y = np.c_[df['LifeSatisf']]
model1 = LinearRegression()
model1.fit(X, y)
model2 = Ridge(alpha=100)
model2.fit(X, y)
print("Coefs model 1: ", model1.coef_, model1.intercept_)
print("Coefs model 2: ", model2.coef_, model2.intercept_)
x_new = np.linspace(0, 140000, 10000)[:, np.newaxis]
y_1 = model1.predict(x_new).flatten().tolist()
y_2 = model2.predict(x_new).flatten().tolist()
plt.figure(figsize=(10, 3))
ax = plt.axes()
ax.scatter(df['GDP_2021'], df['LifeSatisf']);
ax.plot(x_new, y_1, c='r', linestyle='--');
ax.plot(x_new, y_2, c='b', linestyle='--');
plt.title("Life satisfaction vs. GDP per capita regression models");
plt.xlabel("GDP per capita (2021) [US$]");
plt.ylabel("Life satisfaction [-]");
Dataset as a comma separated file:
Country,LifeSatisf,GDP_2021
Australia,7.3,62618.59
Austria,7.1,53793.37
Belgium,6.9,50412.71
Canada,7.4,52791.23
Czech Republic,6.7,25806.38
Denmark,7.6,67919.59
Finland,7.6,53522.57
France,6.5,45028.27
Germany,7.0,50787.86
Greece,5.4,19827.16
Hungary,5.6,18527.59
Iceland,7.5,68843.65
Ireland,7.0,102394.02
Italy,6.0,35584.88
Japan,5.9,40704.3
Korea,5.9,35195.52
Luxembourg,6.9,131301.6
Mexico,6.5,9967.39
Netherlands,7.4,57714.88
New Zealand,7.3,48348.99
Norway,7.6,82244.23
Poland,6.1,17318.5
Portugal,5.4,24457.14
Slovak Republic,6.2,21383.29
Spain,6.3,30536.86
Sweden,7.3,58639.19
Switzerland,7.5,93515.48
Turkey,5.5,9406.58
United Kingdom,6.8,46200.26
United States,6.9,69375.38
Brazil,6.4,7741.15
Chile,6.5,16799.37
Estonia,5.7,27100.74
Israel,7.2,49840.25
Latvia,5.9,19538.9
Russia,5.8,11273.24
Slovenia,5.9,28939.27
South Africa,4.7,6861.17
Colombia,6.3,5892.14
Lithuania,5.9,22411.65
