3
$\begingroup$

I understand the meaning of empirical risk minimization as separate topic and was reading about structural risk minimization, it is hard for me to understand the difference between these two.

I read somewhere that perceptron uses Emperical risk minimization where as SVM uses structural.

I know that SVM takes into consideration model complexity so it is somewhat related to structural risk minimization put this is not clear to me whats the difference

$\endgroup$
1
  • $\begingroup$ Would it be possible to provide the link of the reference material which says "perceptron uses ERM and SVM uses structural" ? $\endgroup$ – avimehenwal Jan 22 '20 at 14:16
2
$\begingroup$

SVMs were developed by Vapnik (1995,1998), based on the structural risk minimization principle (Vapnik, 1982) from statistical learning theory.

The complexity of the class of functions performing classification or regression and the algorithm’s generalizability are related. The Vapnik-Chervonenkis (VC) theory provides a general measure of complexity and proves bounds on errors as a function of complexity. Structural risk minimization is the minimization of these bounds, which depend on the empirical risk and the capacity of the function class

More Details with Math & Paper

More on SRM

In brief:

Empirical risk minimization (ERM) is a principle in statistical learning theory that defines a family of learning algorithms and is used to give theoretical bounds on their performance. The core idea is that we cannot know exactly how well an algorithm will work in practice (the true "risk") because we don't know the true distribution of data that the algorithm will work on, but we can instead measure its performance on a known set of training data (the "empirical" risk).

enter image description here

Note the h here stands for the f — our classifier. This function is called Empirical Risk Minimization (ERM).

Structural risk minimization (SRM) is an inductive principle of use in machine learning. Commonly in machine learning, a generalized model must be selected from a finite data set, with the consequent problem of overfitting – the model becoming too strongly tailored to the particularities of the training set and generalizing poorly to new data. The SRM principle addresses this problem by balancing the model's complexity against its success at fitting the training data.

enter image description here

which is called Structural Risk Minimization (SRM). J(f) is the complexity of the model, usually can be the bound of vector space. λ≥0 is the coefficient choosing the strength of the penalty term.

More Info at

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.