What does "regularization" actually refer to?

Question

I am familiar with regularization, where we add a penalty in our cost function to force the model to behave a certain way. But is this a definition of regularization?

Typically we regularize to get a "simpler" model in some sense. But we could easily create a penalty function that forces a model to be more complex. Would this be considered regularization?

Most commonly it is a penalty on the size of our model parameters. If we add a penalty that is not a function of the model parameters, but rather the model output, would that still be considered regularization? Or is that just a modified objective function?

Answer 1

According to wikipedia, the definition regularization is the process of adding information in order to solve an ill-posed problem or to prevent overfitting.

One common approach is to add a penalty term for large parameter values to the loss function. There are many other approaches to regularization. Here are a couple of other examples:

• Increasing the amount of data (either by collecting more data or data augmentation of existing data)
• Early stopping of the training process
• Add a prior to the model
• Dropout - randomly remove connections during training
• Pruning - removing connections after training