8
$\begingroup$

Given an original probability distribution P, I want to measure how much an approximation Q differs from the initial distribution. For that I calculate the KL-divergence via scipy.stats.entropy, which returns infinity due to the large difference. However, as with time the approximation becomes better, I still want to quantify the divergence between the two sets.

The question is, is there any hack to avoid inf values or should I circumvent the behaviour by using some other distance measure?

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
7
$\begingroup$

The infinity is due to divide by zero. Just replace any zero value with a very small value. This question is very common , if you search for it you will find many questions similar to it . Remember that kl is not a metric

Improve this answer
$\endgroup$
3
  • $\begingroup$ Even though I replaced all items in the array by 0.00001, it still returns infinity after the computation... However, I feed the function an unnormalized version so it can transform it into a normalized one. Might this be the problem? $\endgroup$
    – user17988
    Apr 21, 2016 at 13:13
  • $\begingroup$ This should be by default. Kl works on distributions. You need to convert to probability, replace any zero value with a very small value $\endgroup$
    – Bashar Haddad
    Apr 21, 2016 at 15:49
  • $\begingroup$ My output changes significantly based on whether I change 0's to 10E-20 or to 10E-300. Any idea on why such a large change? $\endgroup$
    – Jake Perret
    Aug 5, 2019 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy