I often stumble upon papers saying that the NN was trained using maximum likelihood principle. I think I generally understand what maximum likelihood is. But I have trouble understanding how to apply it to neural network training.
How are neural network trained using maximum likelihood principle?
This is pretty common.
The last step for a simple deep neural network would be to determine a way to choose values of $\theta$ that maximize the likelihood of your training data. So, what do you do? Typically, you write a log-likelihood function, and then find the values that maximize it.
In fact, it's relatively trivial to prove that minimizing the loss is equivalent to maximizing the likelihood of the input. Typically, in academic papers, you may see people prefer maximizing likelihood simply for the mere fact that some proofs are easier that way.
Consider the Binary Cross Entropy formula $$CE(y,x,\theta) = - \sum_{i=1}^{n} y_{i} log{p_{\theta}}(y \mid x_{i})+ (1-y_{i})log(1-p_{\theta})(y \mid x_{i})).$$ Naturally, you would optimize this with respect to $\theta$.
We can instead choose to take a probabilistic approach and maximize the likelihood of the data under some probabilistic model. A natural choice would be to use a Bernoulli distribution, such that $$p(y \mid \pi) = \prod_{i=1}^{y_{i}} \pi_{i}^{y_i}(1-\pi_{i})^{1-y_{i}}.$$
Now, the task here is to train a neural network to estimate $\pi$. The likelihood function is simply $$p(y \mid x, \theta) = \prod_{i=1}^{y_{i}} p_{\theta}(y \mid x_{i})^{y_i} (1-p_{\theta}(y \mid x_{i}))^{1-y_{i}}.$$ We want to maximize the function w.r.t $\theta$.
Now, if you take the log, and we find $$\sum_{i=1}^{n} y_{i} log{p_{\theta}}(y \mid x_{i})+ (1-y_{i})log(1-p_{\theta})(y \mid x_{i})).$$ Thus there is literally no difference in minimizing a cross entropy or maximizing the log likelihood.
Therefore, the only difference is the approach. At the end of the day, you're doing the same thing regardless what you choose to call it.
If you have a probabilistic cost function (e.g log-loss), I'm pretty sure backprop is an estimator for the MLE. Consider the derivation here: http://ttic.uchicago.edu/~shubhendu/Pages/Files/Lecture3_flat.pdf
If you have articles on hand, I'd be interested to see an example of an experiment where they discussed fitting a network via MLE but explicitly were not using backprop. Otherwise, I think you can assume that the authors were describing backprop when they said "MLE". Maybe they thought "MLE" sounded more academic?
