2
$\begingroup$

I begin the post trying to say that I don't know if this post is in compliance with community rules, so pardon me for any abuse.

I studied statistical learning theory back at university. Specifically, PAC learning, VC Dimension, Uniform convergence, etc. Recently. I watched this talk, with Vapnik, in which he claims that deep learning is essentially "a bla bla interpretation" and also claims that "every problem can be solved with statistical learning theory."

I am very confused about this. I can't see how I can apply statistical learning theory to real problems.

Let's suppose I'm facing a new dataset with a clear task of binary classification, with many features and lots of training data. How am I supposed to check for example, if a hypothesis class H is PAC learnable, or in other words if it has a finite VC dimension? Don't take my example too literally, I just would like to know if someone can point me out to an article, blog, or some kind of answer that clearly shows how we can use these theorems and results in real analysis.

Thank You.

$\endgroup$

2 Answers 2

0
$\begingroup$
  1. the development of the basic calculus , necessary for both formulating problems and numerical techniques for finding the minimum of a function

  2. the theoretical development of theory of learning, such as the VC theory, in the same way it is used in statistics to do things like prove the central limit theorem

$\endgroup$
0
$\begingroup$

I have not seen the youtube video you post, but if you read "Understanding deep learning requires rethinking generalization" (Zhang, Bengio et al, 2016), you will see a very clear analysis of why this Vapnik's claim is overstating. In Statistical Learning the complexity-generalisation trade-off lead us to a pseudo-paradox: Models with more parameters than there are input data should not be able to generalise. In total disregard to the theory, in fact, they do. To sum up, VC will give you vacuous bounds for DNNs.

Recently, Dziugaite and Roy "Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks" have been able to give nonvacuous bounds for Deep Learning, but they relied on the characteristics of solutions found by SGD.

I recommend this video (https://www.youtube.com/watch?v=dHUH0hmKvs8) with Karoline Dziugaite explaining their effort, she explains very well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.