SVMs were developed by Vapnik (1995,1998), based on the structural risk minimization principle (Vapnik, 1982) from statistical learning theory.
The complexity of the class of functions performing classification or regression and the algorithm’s generalizability is related. The Vapnik-Chervonenkis (VC) theory provides a general measure of complexity and proves bounds on errors as a function of complexity. Structural risk minimization is the minimization of these bounds, which depend on the empirical risk and the capacity of the function class
More Details with Math & Paper
More on SRM
In brief:
Empirical risk minimization (ERM) is a principle in statistical learning theory that defines a family of learning algorithms and is used to give theoretical bounds on their performance. The core idea is that we cannot know exactly how well an algorithm will work in practice (the true "risk") because we don't know the true distribution of data that the algorithm will work on, but we can instead measure its performance on a known set of training data (the "empirical" risk).
$$
R_{\mathrm{emp}}(h)=\frac{1}{m} \sum_{i=1}^{m} L\left(h\left(x_{i}\right), y_{i}\right)
$$
Note the $h$ here stands for the f — our classifier. This function is called Empirical Risk Minimization (ERM).
Structural risk minimization (SRM) is an inductive principle of use in machine learning. Commonly in machine learning, a generalized model must be selected from a finite data set, with the consequent problem of overfitting – the model becoming too strongly tailored to the particularities of the training set and generalising poorly to new data. The SRM principle addresses this problem by balancing the model's complexity against its success at fitting the training data.
$$
R_{\mathrm{srm}}(f)=\frac{1}{N} \sum_{i=1}^{N} L\left(y_{i}, f\left(x_{i}\right)\right)+\lambda J(f)
$$
which is called Structural Risk Minimization (SRM).
Where,
- $J(f)$ is the model's complexity, usually the bound of vector space.
- $λ≥0$ is the coefficient choosing the strength of the penalty term.
More Info at