I have a question about ridge regression and about its benefits (relative to OLS) when the datasets are big. Do the benefits of ridge regression disappear when the datasets are larger (e.g. 50,000 vs 1000)? When the dataset is large enough, wouldn't the normal OLS model be able to determine which parameters are more important, thus reducing the need for the penalty term? Ridge regression makes sense when the data sets are small and there is scope for high variance, but do we expect its intended benefits (relative to OLS) to disappear for large datasets?
0
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
One thing that you might be overlooking with your reasoning is the fact that increased predictors does not necessarily lead to a better model. In this case, the more predictors you have the higher the risk of collinearity that there is between them - thus increasing the utility of ridge regression.
In fact, a second point:
The larger the number of predictors that you have in your model the more useful techniques like ridge regression are. This is because with so many predictors it is very difficult to determine if collinearity or other relationships exist between predictors. When you have a smaller model with say 5 predictors, you could verify this fairly easily. With 1,000 predictors this would be much more difficult.
In the case of the Lasso regression, the benefits are even more obvious. Since predictor coefficients values can be shrunk fully to zero this acts as a form of feature selection. Here, you are potentially removing predictors that capture redundant information.
Also - in answer to the second part of your question - OLS does not perform any form of feature selection. Therefore, it will not be able to pick out which parameters are the most important by adding more. The more parameters you add to your model the higher the variance gets. Why is this? You have more estimations to make and therefore collectively your model is less reliable. Consequently, your model will likely inflate the coefficients on the predictors giving undue "weight" or importance to certain predictors.
-
$\begingroup$ Thanks for this response. When I said 'larger datasets', I was referring to number of data points rather than parameters. You are correct that OLS doesn't have feature selection. Generally speaking, for a dataset of size 50,000 (with ~10 parameters) would we still expect ridge regression to outperform as significantly as it might to with a dataset of size 1,000? Thanks for the help! $\endgroup$ Commented Dec 4, 2020 at 20:01
-
$\begingroup$ I think a generalization in this regard is somewhat difficult to draw as it will depend on the data that comprises each of the data sets. I would say that depends. If you have a data set or 50k intuition would suggest that we might get a better fit, however since OLS is at high risk of being effected by outliers perhaps if there are many present in that data set the fit will be worse. If the dataset of 1k however follows a linear pattern almost exactly the fit might be better. $\endgroup$– EthanCommented Dec 4, 2020 at 20:07
-
$\begingroup$ Consequently, it would be difficult to say whether ridge regression would or would not perform better in either of these cases without trying it both ways to see. $\endgroup$– EthanCommented Dec 4, 2020 at 20:08
-
1$\begingroup$ Okay thank you very much! (I would upvote the post but I do not have enough reputation) $\endgroup$ Commented Dec 4, 2020 at 20:13