# Distance between very large discrete probability distributions

I have 192 countries where each country has some value for 1 million attributes which sum up to 1 (a discrete probability distribution). For any one country most of the values for the attributes are 0.

Now I am trying to find the distance/similarity between those countries using these attributes. I know we can use Jensen Shannon Divergence between two discrete probability distributions to get a distance measure, but the caveat is that all the values have to be non-zero.

Given that there are zero valued attributes for the countries, is there any other suitable statistical distance measure that can help me to cluster these countries using these 1 million attributes?

• Use regularization so none of the probabilities are precisely zero. You should be using regularization anyway; it reduces variance. Welcome to the site.
– Emre
Jul 3, 2018 at 22:55
• Could you elaborate a bit? Do you mean replacing 0 with very small non-zero value? Jul 3, 2018 at 23:41
• For example, while ensuring the result still adds to unity. Read about Bayesian priors and conjugates.
– Emre
Jul 3, 2018 at 23:44
• You can use zero values in the JSD by invoking the limit $\lim \limits_{x\to 0^{+}}x\log (x)=0$ yesterday

Yes, plenty.

Get the book "encyclopedia of distances".

For example, you can use Histogram Intersection distance. Since your data is already normalized, that reduces to Manhattan distance, if I am not mistaken. Yes: this can be appropriate for distributions.

I suggest using some kind of smoothing to the probability distribution, e.g., Laplace smoothing (sometimes known as "add-one smoothing"). Then, you will be able to use the Jensen-Shannon distance.

Or, you can use some other distance metric, such as earth-mover's distance or total variation distance. Which metric is most suitable might depend on details of the application domain and the desired interpretation of these distances.

You could use Kullback–Leibler divergence, aka relative entropy, which is a measure of how one probability distribution diverges from a second, expected probability distribution. It handles attributes that have 0 value.

• Kullback–Leibler divergence only handles zeros through applying the limit $\lim \limits_{x\to 0^{+}}x\log (x)=0$. This limit also applies to the Jensen Shannon Divergence (the 'distance' suggested by the OP). So, in the sense of zeros, the KLD is no different to the JSD. yesterday