# Statistical loss function for categorical distributions

For training an autoencoder model whose outputs (and inputs) are parameters from a categorical distribution $$[q_1, q_2, \ldots, q_n]$$, I have to define a proper loss function measuring the distance between the ground truth $$P$$ and prediction $$Q$$.

Since the outputs are probabilistic, I want to use a loss function that amongst others satisfies that the absolute difference between a ground truth $$P$$ and prediction $$Q$$ is penalized more as $$P$$ increases. As an example, $$[p,q] = [0.9, 0.8]$$ should receive a higher penalty than $$[p,q] = [0.8, 0.7]$$.

My initial thoughts were to use the Kullback-Leibler divergence, which for $$n$$-dimensional categorical distributions $$P$$ and $$Q$$ is as follows: $$KL(P||Q) = \sum_{i=1}^{n}p_i\log\big(\frac{p_i}{q_i}\big)$$ A problem that arises when using this divergence is that it is only defined when $$q_i=0$$ implies $$p_i=0$$. This problem can be solved by using a 'fuzz factor' $$\epsilon>0$$: $$KL_{\epsilon}(P||Q) = \sum_{i=1}^{n}(p_i+\epsilon)\log\big(\frac{p_i+\epsilon}{q_i+\epsilon}\big)$$

I wonder if there are other good metrics that can be used as loss function for this case where the output is a set of parameters from a categorical distribution.

• How's a formulation like L(P||Q) = |(p-q)|^(1/(1+p)) ? You panalize for higher p's and it is a continous function with no boundaries. – Maeaex1 Nov 20 '19 at 9:04
• @Maeaex1 that does the job indeed. I do wonder however if there are widely used solutions which have a good mathematical background. – Rutger Mauritz Nov 20 '19 at 11:09
• Ouch :-D "Good Mathematical background" that hurts - But no indeed - I was quickly looking up for a more generalized/recognized solution but didn't found something on the way. If not looked up already: loss functions for classification My approach should be conform with Bayes consistency. – Maeaex1 Nov 20 '19 at 12:43