I learned about "the information bottleneck view of deep learning." But in a nutshell, what does this tell us?

I don't see what the role is of depth in this approach as long as it is larger than 2 or 3. Is there a rigorous theory? Or just some hypothesis or heuristic explanations on deep neural net?

I saw the author's talk on YouTube. But, probably my ignorance, I don't really get the main point and the implication is. I can see a lot of explanations on graphs on the video, but honestly, I don't get it.

Any comments, suggestions, opinions will be very appreciated.


Current Statistical Learning Theory treats a learning algorithm like a "black-box", analysing its input versus outputs. Besides, it is usually criticised for lack of non-vacuous bounds (Despite the non-vacuous bound proved by Diziugaite and Roy).

Information Bottleneck Theory brings an Information-Theoretic perspective to the learning problem that allows us to analyse what happens during training with information measures. When you do that, IBT predicts a phase transition between two distinct phases of training (a fitting phase, where the model rapidly fits to the data;and a compression phase when the model forgets irrelevant information of the dataset, trying to avoid overfitting).

It is not a proven rigorous theory as Tishby himself admits (see the video in DeepMath conference 2020) and this lack of rigour has provoked a great deal of criticism (see Saxe et al "On the IBT". They are not alone in the criticism). A more rigorous approach that can be seen as in the realm of IBT is taken by Stefano Soatto and Alessandro Achille in their research group in California (see Emergence of Invariance and Disentangling in Deep Representations).

Still, it is an "emerging field" where rigour is been built. The interesting aspect of IBT is that it gives new meaning (a narrative) for what is happening during training. In this narrative, there is no generalization paradox (see Zhang,Bengio, et al. "Understanding... rethink generalisation"), as what matters is not the number of parameters of a model, but the amount of information it has about the training dataset.

  • $\begingroup$ Thank you so much for your answer. I do really appreciate it. I have some clarification questions. Hope you can answer them. $\endgroup$ – induction601 Jul 16 at 17:30
  • $\begingroup$ You said there are two training phases. I can understand what is the fitting phase which we can see from the loss function. I am not sure how one can measure the compression. Is there any mathematical definition? $\endgroup$ – induction601 Jul 16 at 17:36
  • $\begingroup$ My background is in math. Since many ppl from different areas are doing the ML, I often got lost due to non-rigorous approaches and heuristic explanations from other communities. Also, I often found that some proofs are not rigorously written in conference papers. $\endgroup$ – induction601 Jul 16 at 17:36
  • $\begingroup$ The compression is measured in information about input variable X in Z. Computing the Shannon information is impossible due the need to know the distribution which we are learning. So, you can use a rough estimation by calculating the fisher information in the weights which is always lower than the true Shannon information. I would like to point you to my dissertation where I explain this in detail, but Still have not defended it. I will check if there is a problem in publishing it at Arxiv $\endgroup$ – Fred Guth Jul 19 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.