I am familiar with regularization, where we add a penalty in our cost function to force the model to behave a certain way. But is this a definition of regularization?
Typically we regularize to get a "simpler" model in some sense. But we could easily create a penalty function that forces a model to be more complex. Would this be considered regularization?
Most commonly it is a penalty on the size of our model parameters. If we add a penalty that is not a function of the model parameters, but rather the model output, would that still be considered regularization? Or is that just a modified objective function?