3
$\begingroup$

I have learned regularization for linear and logistic regression but when I implement that algorithm to my code generally my estimates not changing.I mean,it looks like ineffective.I know,it's for overfitting.So if I use it in my code every time ,could it be a problem? or is this a good thing?

$\endgroup$
5
$\begingroup$

Normally you use regularization. The exception is if you know the data generating process and can model it exactly. Then you merely estimate the model parameters. In general you will not know the process, so you will have to approximate with a flexible enough model. If the model is not flexible enough you will not need to regularize but you won't approximate well anyway. If you use a more flexible model, you will get closer on average (low bias) but you will have more variance, thus the need for increased regularization. In other words, the amount of regularization you need depends on the model. This is related to the bias-variance trade-off.

Welcome to DataScience.SE.

$\endgroup$
3
$\begingroup$

Adding a few more specifics to the previous two responses which both contain useful insight and perspective:

Regularization is used to control overfitting (more formally high variance) scenarios. You should view any model as a careful balance of bias and variance. Thus a model that is unresponsive to regularization might be too underfit to begin with. Its tough to guess what is occurring without seeing your cross validation results.

Have you cross validated your model? Are you using good metrics to assess your model? How do the the training metrics compare to the testing metrics as you adjust the regularization?

The ideal model is slightly overfit and then regularization is applied to balance the bias and the variance. One can engage in a bias-variance decomposition, but with a little more experience you can just compare the single metrics for the training and testing set and test the effect of increasing and decreasing the regularization.

Hope this helps!

$\endgroup$
0
$\begingroup$

Regularization is useful in preventing overtraining, however, if your regularization parameter is too aggressive, then it will dominate the learning function, and not allow you to properly tune variables. Try reducing the regularization parameter.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.