1
$\begingroup$

I am performing regularization (Ridge regression) using cross validation. I understand that for a certain value of the regularization parameter $\lambda$, we first fit on the training-set and then we use the resulting parameters to calculate the error on the test-set and the optimal $\lambda$ is the one that minimizes the test-set error.
My question is: for the calculation of the test-set error do we include the penalty term (for the value of $\lambda$ that we used in the training set), or not (i.e. use $\lambda =0$)?
On one hand, I see the need to treat the training and test sets "on an equal footing" and use the same $\lambda$ for the test-set error calculation.
On the other hand, the parameter values that we get from fitting the training-set have already been penalized, and we want to see how these values perform on the test-set, so why use a penalty term again?

$\endgroup$

1 Answer 1

1
$\begingroup$

You should use the same value of λ on both training and test data sets.

The value of λ influences the values of the parameters. The values of the parameters should be consistent across training and test data sets, thus the value of λ should be the same.

$\endgroup$
1
  • $\begingroup$ Thanks Brian. I would agree that consistency is a good enough reason. $\endgroup$
    – dimitris
    Dec 2, 2022 at 10:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.