Given there are two matrices of dimensionality 100x2 with absolute values ranging from -50 to +50. Is it possible to determine the kl-divergence by applying the entropy algorithm from scipy.stats to the two flattened vectors of size 200?

  • $\begingroup$ As a follow-up question, does the following equation from scipy.stats.entropy calculate the symmetrical KL-divergence, usable as a metric (with pk and qk representing the two vectors)? S = sum(pk * log(pk / qk), axis=0) $\endgroup$ – user17988 Apr 19 '16 at 8:08
  • 1
    $\begingroup$ @piutu scipy.stats.entropy calculates the kullback leibler divergence for p and q, which is not symmetric. You can use scipy.stats.entropy to calculate the jensen-shannon divergence, which is symmetric and whose square root satisfies the triangle inequality (i.e. it's a metric): jsd(p, q) = 0.5*entropy(p, 0.5*(p+q)) + 0.5*entropy(q, 0.5*(p+q)) $\endgroup$ – stmax Apr 19 '16 at 14:08

I think you can. Just normalize both of the vectors to be sure they are distributions. Then you can apply the kl divergence . Note the following: - you need to use a very small value when calculating the kl-d to avoid division by zero. In other words , replace any zero value with ver small value - kl-d is not a metric . Kl(AB) does not equal KL(BA) . If you are interested in it as a metric you have to use the symmetric kl = (Kl(AB) +KL(BA) )/2

  • $\begingroup$ that's too naive.... $\endgroup$ – LKM Jan 13 at 18:29

Even though it is possible, it probably does not produce a meaningful measure. I would advice however to be careful with these points:

  • A probability distribution should represent probabilities. You can not take any arbitrary vector and consider them as probabilities distributions. The key here is to have a sound explanation for why outcome X has a say .2 (arbitrary) probability. The most simple way of "calculating" probabilities is considering it is the frequency of occurrences in an experiment comprised of N trials.

  • What your dimensions represent? To flatten a matrix only means you now have a vector. Having a vector might be enough for you function to run, but might not give you a meaningful measure.

  • stats.entropy normalizes your vector if it does not sums up to 1. Often you will want to use the same kind of normalization on your pipeline. Be sure to normalize it before passing to entropy() if necessary. Here is the normalization performed by entropy:

    pk = 1.0*pk / np.sum(pk, axis=0)
  • $\begingroup$ Basically I have two matrices where dimensionality is reduced via tSNE to two 100x2 matrices, and visualized via scatter plot. What I want to measure is how much the second distribution of points differs from the first distribution. So I assume the steps to be taken are: - normalize the values via pk = 1.0*pk / np.sum(pk, axis=0) for both matrices - flatten the matrices to dimensionality 200x1 - feed them to scipy.stats.entropy Is this correct or will the results not be meaningful? $\endgroup$ – user17988 Apr 22 '16 at 8:15
  • $\begingroup$ @Piutu, the steps you describe will produce a KL-divergence metric. You don't need to normalize if you want to use entropy() built-in normalization. You still should know what your matrix represents. I quickly read about tSNE implementation from SKlearn and I believe each row of your 100x2 matrix is a sample (as it is on a design matrix), so you should be calculating the KL-divergence between each row from your 2 matrices (you will have a 100x100 resulting matrix). Please confirm you actually have 100 samples in each matrix. If that is the case, then you should not flatten the matrix. $\endgroup$ – Rabbit Apr 22 '16 at 19:41
  • $\begingroup$ (I can't comment yet due to missing reputation) Yes, indeed each row represents a sample. Does it mean that I now need to feed each row of the two matrices to scipy.stats.entropy and sum over all 100 samples? $\endgroup$ – user17988 Apr 25 '16 at 7:56
  • $\begingroup$ tSNE can produce very different results for two sets of data, even if the data is similar. I don't think two runs of tSNE should be used to compare the similarity of the samples. $\endgroup$ – stmax Apr 25 '16 at 12:22
  • $\begingroup$ @Piutu, you will make a call to scipy.stats.entropy for each pair of rows that you have. The regular way to compare would be compare between 2 samples (two rows, one from each matrix). I am not sure if there is a way to summarize a set of divergences, though. Maybe you could average each dimension and then apply the KL-Divergence. Do you need one single value, is that it? What use do you intend to use from this similarity measure? $\endgroup$ – Rabbit Apr 26 '16 at 1:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.