I have a dataset with 500 000 samples, each sample contains 30 features. The values of the features are in the range 0.0 to 1.0.

I want to measure the distance between two points in the dataset. A simple thing to do could be to measure the euclidean distance between the two 30 dimensional points.

However, if I perform for e.g PCA with 3 components, and then measure the euclidan distance, should the measurements yield similar results?

My idea is that if PCA is performed, knowledge about the whole dataset is passed to the distance function, which might improve it? - Or is this just a simple way of throwing away information?

I would hope that I could gain some knowledge from the whole dataset instead of measuring the distance between two points, and ignoring the distribution of the dataset.


What I am looking for is. Can I take advantage of the whole data set when doing similarity measures between random pairs? E.g just plain euclidian distance between the raw pairs does just calculate the distance between two points. It ignores the relation for the whole dataset, two points might considered to be even more close, when all other points are more far away.


I would expect your original distance metric to be recapitulated reasonably well in your PCA-reduced distance metric, particularly if your first few PCs capture most of the variability in your data. The first PCs capture the greatest axes of variability in your data, so the last few PCs are relatively invariant across all samples, and will contribute less to the distance metric. You can imagine a case where you have 3 informative features, and 27 features that are invariant. Your first 3 PCs can perfectly represent your dataset, and will produce the exact same distance matrix as the original space.

If, however, your features are all highly variable and uncorrelated, your first few PCs may not capture a sufficient proportion of the variation, in which case your distance metric will look quite different.

  • $\begingroup$ But is there any information gained by doing this? What am looking for is, can I take any advantage of the whole dataset when doing similarity measures between samples? Or is plain euclidian of the raw pairs good enough? For e.g would I capture similarity better if I performed some sort of clustering, and then doing similarity measures? $\endgroup$ – Isbister Feb 7 '19 at 19:23

PCA usually involves scaling the data. Hence it will not preserve the original distances. If you'd only use the rotation then it should preserve Euclidean distances. The question then mostly is: why? 30 is not much. Just keep all 30?

There are other projections such as MDS (Multidimensional Scaling) that try to find a low dimensional embedding such that the original distances are best preserved as Euclidean distances now.

  • $\begingroup$ Thx for the answer. Maybe I tackled this in the wrong way. Basically what I want to do is, when measuring the distance between two points in a space, can I take advantage of the distribution of the whole dataset to capture similarities better? $\endgroup$ – Isbister Feb 11 '19 at 8:45
  • $\begingroup$ PCA is related to Mahalanobis distance. So that is, obviously, possible. The question whether it is the right thing to do is up to you... When the data is in 0:1, I'd assume it either has already been hammered before without care and damage has been done already, or the data natively has this range, and then it probably should not be transformed to lose this semantic property... $\endgroup$ – Has QUIT--Anony-Mousse Feb 11 '19 at 9:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.