Request: Confirmation on my understanding of overfitting and regularization concepts

Question

Overfitted models tend to have largely different (some very high, some comparatively low) coefficients/weights for different feature values. So, this means the model (when drawn as graph) will have high variation in slopes and even a small change in training data value (feature value) can lead to large change in output. To smoothen the overfitted model/curve that has high slope variation, we use regularization (example: L1/L2).

L1 regularization removes unnecessary/less influential features from the model making the model less complex. It does so by changing the weights/coefficients of such features to 0. So, this regularization is useful when we have many unnecessary features and is also considered useful for feature selection.

L2 regularization shrinks/adjusts extreme weights and results in a set of weights that are more evenly distributed. Unlike L1 Regularization, it does not result in weights of features being 0. So, this regularization is a little better when we know that all/a majority of the features are useful for the model.

Answer 1

In general your understanding is correct.

Overfitted model parameters also means that you have captured the training sets distribution to specifically and that the model does not generalize to the true distribution of the variable.

The general explanations you gave for the two forms of regularization are correct. However, you need to also consider the cost to solving the regularizations problems. In linear regression, there is a closed form solution for L2 regularization but not for L1, this means you will need to use an optimization algorithm to find an optima which can be costly.

For some other details about regularization I have answered a similar question here: Regularization in simple math explained