I would like to understand regularization/shrinkage in the light of MLE/Gradient Descent. I know both concepts but I do not know/understand whether both are used to determine coefficients of a linear model. If so, what are the steps followed?

To further elaborate, regularization is used to reduce variance which is accomplished through penalizing coefficients of a linear model. The tuning parameter, lambda, is determined through cross-validation. Once, lambda is determined the coefficients are automatically determined, right? Hence, why do we need to minimize (RSS + regularization term) to find coefficients? Are the steps the following:

  1. Find lambda through cross-validation
  2. Minimize (RSS + regularization) through MLE or GD
  3. Find coefficients
  4. Penalize coefficients to decrease variance
  5. We are left with a small subset of coefficients
  • 2
    $\begingroup$ You want to minimize RSS + reg.term because you (1) want to fit the model to the training data which is likely from the distribution of the process you want to model, (2) and not overfit. $\endgroup$ – Vladislavs Dovgalecs Sep 2 '15 at 21:04

The fitting procedure is the one that actually finds the coefficients of the model. The regularization term is used to indirectly find the coefficients by penalizing big coefficients during the fitting procedure. A simple (albeit somewhat biased/naive) example might help illustrate this difference between regularization and gradient descent:

X, y <- read input data
for different values of lambda L
    for each fold of cross-validation using X,y,L
        theta <- minimize (RSS + regularization using L) via MLE/GD
        score <- calculate performance of model using theta on the validation set 
    if average score across folds for L is better than the current best average score 
        L_best <- L

As you can see, the fitting procedure (MLE or GD in our case) finds the best coefficients given the specific value of lambda.

As a side note, I would look at this answer here about tuning the regularization parameter, because it tends a little bit murky in terms of bias.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.