Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.
Regularization refers to the inclusion of additional components in the model fitting process that are used to prevent overfitting and/or stabilize parameter estimates.
Parametric approaches to regularization typically add terms to the training error or MLE objective function that penalize model complexity, in addition to the standard data misfit terms (e.g. Ridge Regression, LASSO). This penalty can be interpreted as arising from a prior on the parameter vector in the framework of Bayesian MAP estimation.
Non-parametric regularization techniques include dropout (used in deep learning) and truncated-SVD (used in linear least squares).
Synonyms include: penalization, shrinkage methods, and constrained fitting.
