# Ridge regression model creation using grid-search and cross validation

I created python code for ridge regression.For that I used cross validation and grid-search technique in together. i got output result. I want check whether my regression model building steps correct or not? can some one explain it?

from sklearn.linear_model import Ridge
ridge_reg = Ridge()
from sklearn.model_selection import GridSearchCV
params_Ridge = {'alpha': [1,0.1,0.01,0.001,0.0001,0] , "fit_intercept": [True, False], "solver": ['svd', 'cholesky', 'lsqr', 'sparse_cg', 'sag', 'saga']}
Ridge_GS = GridSearchCV(ridge_reg, param_grid=params_Ridge, n_jobs=-1)
Ridge_GS.fit(x_train,y_train)
Ridge_GS.best_params_


output - {'alpha': 1, 'fit_intercept': True, 'solver': 'cholesky'}

Ridgeregression = Ridge(random_state=3, **Ridge_GS.best_params_)
from sklearn.model_selection import cross_val_score
all_accuracies = cross_val_score(estimator=Ridgeregression, X=x_train, y=y_train, cv=5)
all_accuracies


output - array([0.93335508, 0.8984485 , 0.91529146, 0.89309012, 0.90829416])

print(all_accuracies.mean())


output - 0.909695864130532

Ridgeregression.fit(x_train,y_train)
Ridgeregression.score(x_test,y_test)


output - 0.9113458623386644

Is 0.9113458623386644 my ridge regression accuracy(R squred) ? if it is, then what is meaning of 0.909695864130532 value.

Is 0.9113458623386644 my ridge regression accuracy(R squred) ? if it is, then what is meaning of 0.909695864130532 value.

The first score is the cross-validation score on the training set, and the second is your test set score. The first is perhaps a little biased, since those models are built using hyperparameters selected while using that dataset (but a different cv-split I think). The second score should be unbiased, and should probably be your reported test score. (The second score is a bit better than what should be a optimistically-biased score; perhaps just randomly, perhaps because of the train/test split, and perhaps because that model has seen more data.)

I want check whether my regression model building steps correct or not? can some one explain it?

Things look OK to me.

You could save yourself some code and training time; by default GridSearchCV refits a model on the entire training set using the identified hyperparameters, so you don't need to fit in the last code block. It also has the cv_results_ and best_score_ attributes to provide you with cross-validation scores, but since you've used the them in selecting the optimal hyperparameters, the best score is no longer an unbiased estimator for future performance.

Note too that there's a builtin for tuning the regularization parameter, but it uses a different CV approach and has RMSE as default scorer.

• That means 0.909695864130532 value came from only training data set and model created using training data then predict using training data and give R score as 0.909695864130532. is this correct? (you mean "The first is perhaps a little biased" because model training and testing did for same data, then it give bias result is this correct?) Jan 14, 2020 at 8:04
• It's not quite as bad as that; a model that was actually trained on all of x_train and then scored on x_train would be very bad. The 0.909 number is the average of cross-validation scores, so each individual model was scored on a subset of x_train that it was not trained on. However, you did use x_train for the GridSearch, so the hyperparameters you chose (alpha=1, fit_intercept=True) were chosen based on x_train. Any model you fit on x_train or its subsets with those hyperparameters has a little bit of an edge. Jan 14, 2020 at 12:45
• Optimizing RMSE and $R^2$ (minimization and maximization, respectively) result in the same model, barring numerical technicalities arising from doing math on a computer. $R^2$ might have the easier interpretation, but the model is the same.
– Dave
Aug 8, 2020 at 13:01