I often stumble upon papers saying that the NN was trained using maximum likelihood principle. I think I generally understand what maximum likelihood is. But I have trouble understanding how to apply it to neural network training.

How are neural network trained using maximum likelihood principle?


This is pretty common.

The last step for a simple deep neural network would be to determine a way to choose values of $\theta$ that maximize the likelihood of your training data. So, what do you do? Typically, you write a log-likelihood function, and then find the values that maximize it.

In fact, it's relatively trivial to prove that minimizing the loss is equivalent to maximizing the likelihood of the input. Typically, in academic papers, you may see people prefer maximizing likelihood simply for the mere fact that some proofs are easier that way.


Consider the Binary Cross Entropy formula $$CE(y,x,\theta) = - \sum_{i=1}^{n} y_{i} log{p_{\theta}}(y \mid x_{i})+ (1-y_{i})log(1-p_{\theta})(y \mid x_{i})).$$ Naturally, you would optimize this with respect to $\theta$.

We can instead choose to take a probabilistic approach and maximize the likelihood of the data under some probabilistic model. A natural choice would be to use a Bernoulli distribution, such that $$p(y \mid \pi) = \prod_{i=1}^{y_{i}} \pi_{i}^{y_i}(1-\pi_{i})^{1-y_{i}}.$$

Now, the task here is to train a neural network to estimate $\pi$. The likelihood function is simply $$p(y \mid x, \theta) = \prod_{i=1}^{y_{i}} p_{\theta}(y \mid x_{i})^{y_i} (1-p_{\theta}(y \mid x_{i}))^{1-y_{i}}.$$ We want to maximize the function w.r.t $\theta$.

Now, if you take the log, and we find $$\sum_{i=1}^{n} y_{i} log{p_{\theta}}(y \mid x_{i})+ (1-y_{i})log(1-p_{\theta})(y \mid x_{i})).$$ Thus there is literally no difference in minimizing a cross entropy or maximizing the log likelihood.

Therefore, the only difference is the approach. At the end of the day, you're doing the same thing regardless what you choose to call it.

  • $\begingroup$ So what does it mean exactly? Just changing MSE cost function to log-like? Maybe you can give some example or code? $\endgroup$ – Mikhail Jan 4 '18 at 15:14
  • $\begingroup$ I don't feel like implementing a code solution but hopefully the above is helpful. $\endgroup$ – Tophat Jan 4 '18 at 15:38

If you have a probabilistic cost function (e.g log-loss), I'm pretty sure backprop is an estimator for the MLE. Consider the derivation here: http://ttic.uchicago.edu/~shubhendu/Pages/Files/Lecture3_flat.pdf

If you have articles on hand, I'd be interested to see an example of an experiment where they discussed fitting a network via MLE but explicitly were not using backprop. Otherwise, I think you can assume that the authors were describing backprop when they said "MLE". Maybe they thought "MLE" sounded more academic?

  • $\begingroup$ The latest paper I've seen was google's Tacotrton 2, where they say " To train the feature prediction network, we apply the standard maximum-likelihood training procedure". I understand that this should be more about specific cost function rather than learning method. But which cost function? $\endgroup$ – Mikhail Jan 4 '18 at 8:42
  • $\begingroup$ The only mention of MLE in the tacotron2 paper is: "To train the feature prediction network, we apply the standard maximum-likelihood training procedure (feeding in the correct output instead of the predicted output on the decoder side, also referred to as teacher-forcing)". If you're looking for an explanation of teacher forcing (MLE for RNNs), this blog post offers a simple explanation with some good citations: machinelearningmastery.com/… $\endgroup$ – David Marx Jan 4 '18 at 10:04
  • $\begingroup$ I understand the teacher-forcing part (that's why i haven't asked about it). I am interested in MLE part here, because eg. in this specific paper there's no other mentions of a cost function being used $\endgroup$ – Mikhail Jan 4 '18 at 12:37
  • 1
    $\begingroup$ I'm pretty sure sections 2.1 and 2.2 say they're minimizing MSE to encode the spectrogram, and then sec 2.3 says they minimized the negative log likelihood of a 10 component logistic mixture model to train the waveform generator. I have no idea what the theoretical basis is for that particular mixture of logistics, but it appears to be described in citation 27 if you want to dig deeper. $\endgroup$ – David Marx Jan 4 '18 at 12:53
  • $\begingroup$ Thanks for pointing to specific sections, seems that i've missed this during last re-reading of the paper. Still this was just an example, and i rather sure i saw "MLE training" in other papers as well $\endgroup$ – Mikhail Jan 4 '18 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.