Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of overfitting the data.
Do I have this right?
It would be very convenient for my use case to be able to skip lengthy cross-validation procedures.
There is no free beer in such matters so "automatically takes care of the danger of overfitting the data" is not true. When you optimize for a loss error function and also for simplicity the immediate question is how do you trade off simplicity on loss error. There is no answer to that, other than in very specific cases (but those cases are so constraint so they can't provide a generic answer).
This is the reason why many models which allows you to do this trade off will give you a parameter or more to use as knobs for this trade off. Examples are logistic regression with regularization, neural nets with dropouts, decision trees with max tree levels, boosting models with learning rates, and how many more. Even if the model does this trade off automatically, it does not mean it is the best trade off, mostly it is some trade off incorporated into the model structure and its assumptions.
The second aspect of your question is that you do not consider the whole scope of cross validation and of similar medthods in general. CV is used to assess the performance of different models, either if they have the same structure but with different parameters, either they are from different families. CV allows you to have a reliable hind regarding which will be the error of a model if used on similarly distributed new data. This is not covered by regularization at all.
No - Adding model complexity penalty not does remove the usefulness of cross-validation.
A model complexity penalty encourages parsimonious models. Cross-validation provides numerical estimate of overfitting.
Cross-validation is one of the best ways to evaluate generalization. It provides an empirical estimate of model performance on data the model was not trained on.
They are best used together. Cross-validation is a way of finding the best level of penalty for model complexity.
