For training an autoencoder model whose outputs (and inputs) are parameters from a categorical distribution $[q_1, q_2, \ldots, q_n]$, I have to define a proper loss function measuring the distance between the ground truth $P$ and prediction $Q$.

Since the outputs are probabilistic, I want to use a loss function that amongst others satisfies that the absolute difference between a ground truth $P$ and prediction $Q$ is penalized more as $P$ increases. As an example, $[p,q] = [0.9, 0.8]$ should receive a higher penalty than $[p,q] = [0.8, 0.7]$.

My initial thoughts were to use the Kullback-Leibler divergence, which for $n$-dimensional categorical distributions $P$ and $Q$ is as follows: $$KL(P||Q) = \sum_{i=1}^{n}p_i\log\big(\frac{p_i}{q_i}\big)$$ A problem that arises when using this divergence is that it is only defined when $q_i=0$ implies $p_i=0$. This problem can be solved by using a 'fuzz factor' $\epsilon>0$: $$KL_{\epsilon}(P||Q) = \sum_{i=1}^{n}(p_i+\epsilon)\log\big(\frac{p_i+\epsilon}{q_i+\epsilon}\big) $$

I wonder if there are other good metrics that can be used as loss function for this case where the output is a set of parameters from a categorical distribution.

  • $\begingroup$ How's a formulation like L(P||Q) = |(p-q)|^(1/(1+p)) ? You panalize for higher p's and it is a continous function with no boundaries. $\endgroup$ – Maeaex1 Nov 20 '19 at 9:04
  • $\begingroup$ @Maeaex1 that does the job indeed. I do wonder however if there are widely used solutions which have a good mathematical background. $\endgroup$ – Rutger Mauritz Nov 20 '19 at 11:09
  • $\begingroup$ Ouch :-D "Good Mathematical background" that hurts - But no indeed - I was quickly looking up for a more generalized/recognized solution but didn't found something on the way. If not looked up already: loss functions for classification My approach should be conform with Bayes consistency. $\endgroup$ – Maeaex1 Nov 20 '19 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.