2
$\begingroup$

I created python code for ridge regression.For that I used cross validation and grid-search technique in together. i got output result. I want check whether my regression model building steps correct or not? can some one explain it?

from sklearn.linear_model import Ridge
ridge_reg = Ridge()
from sklearn.model_selection import GridSearchCV
params_Ridge = {'alpha': [1,0.1,0.01,0.001,0.0001,0] , "fit_intercept": [True, False], "solver": ['svd', 'cholesky', 'lsqr', 'sparse_cg', 'sag', 'saga']}
Ridge_GS = GridSearchCV(ridge_reg, param_grid=params_Ridge, n_jobs=-1)
Ridge_GS.fit(x_train,y_train)
Ridge_GS.best_params_

output - {'alpha': 1, 'fit_intercept': True, 'solver': 'cholesky'}

Ridgeregression = Ridge(random_state=3, **Ridge_GS.best_params_)
from sklearn.model_selection import cross_val_score
all_accuracies = cross_val_score(estimator=Ridgeregression, X=x_train, y=y_train, cv=5)
all_accuracies

output - array([0.93335508, 0.8984485 , 0.91529146, 0.89309012, 0.90829416])

print(all_accuracies.mean())

output - 0.909695864130532

Ridgeregression.fit(x_train,y_train)
Ridgeregression.score(x_test,y_test)

output - 0.9113458623386644

Is 0.9113458623386644 my ridge regression accuracy(R squred) ? if it is, then what is meaning of 0.909695864130532 value.

$\endgroup$
1
$\begingroup$

Is 0.9113458623386644 my ridge regression accuracy(R squred) ? if it is, then what is meaning of 0.909695864130532 value.

These are both R^2 values.

The first score is the cross-validation score on the training set, and the second is your test set score. The first is perhaps a little biased, since those models are built using hyperparameters selected while using that dataset (but a different cv-split I think). The second score should be unbiased, and should probably be your reported test score. (The second score is a bit better than what should be a optimistically-biased score; perhaps just randomly, perhaps because of the train/test split, and perhaps because that model has seen more data.)

I want check whether my regression model building steps correct or not? can some one explain it?

Things look OK to me.

You could save yourself some code and training time; by default GridSearchCV refits a model on the entire training set using the identified hyperparameters, so you don't need to fit in the last code block. It also has the cv_results_ and best_score_ attributes to provide you with cross-validation scores, but since you've used the them in selecting the optimal hyperparameters, the best score is no longer an unbiased estimator for future performance.

Note too that there's a builtin for tuning the regularization parameter, but it uses a different CV approach and has RMSE as default scorer.

| improve this answer | |
$\endgroup$
  • $\begingroup$ That means 0.909695864130532 value came from only training data set and model created using training data then predict using training data and give R score as 0.909695864130532. is this correct? (you mean "The first is perhaps a little biased" because model training and testing did for same data, then it give bias result is this correct?) $\endgroup$ – randunu galhena Jan 14 at 8:04
  • $\begingroup$ It's not quite as bad as that; a model that was actually trained on all of x_train and then scored on x_train would be very bad. The 0.909 number is the average of cross-validation scores, so each individual model was scored on a subset of x_train that it was not trained on. However, you did use x_train for the GridSearch, so the hyperparameters you chose (alpha=1, fit_intercept=True) were chosen based on x_train. Any model you fit on x_train or its subsets with those hyperparameters has a little bit of an edge. $\endgroup$ – Ben Reiniger Jan 14 at 12:45
  • 1
    $\begingroup$ Optimizing RMSE and $R^2$ (minimization and maximization, respectively) result in the same model, barring numerical technicalities arising from doing math on a computer. $R^2$ might have the easier interpretation, but the model is the same. $\endgroup$ – Dave Aug 8 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.