2
$\begingroup$

I am doing some research on machine learning algorithms in the context of a seminar, which focuses on Energy-Based Modeling vs Deep Learning Modeling specifically in working with images. Now I know these are broad topics, but I was wondering what some specific similarities and differences are. I have read countless papers that say "EBMs have properties that make them better than DL in certain tasks" but I haven't really found a comprehensive motivation

  • What is the problem that EBM is solving compared to DL?
  • Why are we doing this?

I hope this is the right site for this question (I also checked theoretical CS), This brought me here.

update Based on my continued reading I believe, that in simple terms my notion of ebm vs DL is wrong. It is a framework that can incorporate deep learning and thus helps improve interpret ability of deep learning models (fundamentally they are black boxes after all!). It is another point of view - a different pair of glasses to see the same thing, if you will. An example would be, that the probabilities generated by probabilistic EBMs are actual probabilities and allow us to make statements about uncertainty (in contrast to the regular soft Mac output). Furthermore their generative nature allows for better calibration in classification and better out of distribution detection.

Maybe this might help or interest someone perusing this modeling framework

Improve this question
$\endgroup$
3
  • $\begingroup$ A new introductory article about EBMs by relevant researchers just came out, maybe you are interested: arxiv.org/abs/2101.03288 . At least I find the docs on modern EBMs somewhat lacking. $\endgroup$
    – noe
    Jan 13, 2021 at 10:02
  • $\begingroup$ Also, you may find this article relevant to your question: arxiv.org/abs/1912.03263 . They reinterpret deep learning classifiers as EBMs. $\endgroup$
    – noe
    Jan 13, 2021 at 10:04
  • $\begingroup$ Thanks for the paper suggestions! I've already seen the second one and found it quite interesting because I found that it highlights the connections quite well. Somehow still missing what specific tasks deep learning can't do. Will check out the first paper now $\endgroup$
    – dohm
    Jan 13, 2021 at 11:14

1 Answer 1

Reset to default
1
$\begingroup$

I also encountered the concept of energy-based models recently and I am not sure if I got the gist: From my understanding, it was mostly about rephrasing the objective to a contrastive divergence via sampling instead of more classic MLE objectives. Is there anything else that is relevant to EBMs?

Also, I have a question from one Wikipedia article about EBMs: Why does it adapt without training?

EBM generators are implicitly defined by the probability distribution, and automatically adapt as the distribution changes (without training), allowing EBMs to address domains where generator training is impractical, as well as minimizing mode collapse and avoiding spurious modes from out-of-distribution samples.

I am new to this topic so apologies for my potentially confusing questions:)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.