2
$\begingroup$

I am familiar with regularization, where we add a penalty in our cost function to force the model to behave a certain way. But is this a definition of regularization?

Typically we regularize to get a "simpler" model in some sense. But we could easily create a penalty function that forces a model to be more complex. Would this be considered regularization?

Most commonly it is a penalty on the size of our model parameters. If we add a penalty that is not a function of the model parameters, but rather the model output, would that still be considered regularization? Or is that just a modified objective function?

$\endgroup$
2
  • $\begingroup$ this is not the definition, the definition is what yourself stated in the beginning. The rest is simpl;y one of the ways to achieve that $\endgroup$
    – Nikos M.
    Sep 25 at 8:32
  • $\begingroup$ In mathematics, statistics, finance, computer science, particularly in machine learning and inverse problems, regularization is the process of adding information in order to solve an ill-posed problem or to prevent overfitting (wikipedia) $\endgroup$
    – Nikos M.
    Sep 25 at 9:17
1
$\begingroup$

According to wikipedia, the definition regularization is the process of adding information in order to solve an ill-posed problem or to prevent overfitting.

One common approach is to add a penalty term for large parameter values to the loss function. There are many other approaches to regularization. Here are a couple of other examples:

  • Increasing the amount of data (either by collecting more data or data augmentation of existing data)
  • Early stopping of the training process
  • Add a prior to the model
  • Dropout - randomly remove connections during training
  • Pruning - removing connections after training
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.