Which dissimilarity/similarity measure use after a dimension reduction ( PCA / AutoEncoder / ... )?

Question

Each problem required its own similarity/dissimilarity measure. Imagine we are dealing with dataset composed with vector of real. I suppose that we mostly use the euclidean distance especially in low dimension. Unfortunately we often have to deal with huge dimensional dataset and we apply divers dimension reduction techniques to make the problem easier to solve.

I will be happy to know your opinion about different similarity/dissimilarity measure you use after made a dimensionality reduction and how they impact classification/regression/clustering metrics.

Is it better to use euclidean distance after a PCA or another distance measure ? Why ?

And if i prefer to use AutoEncoder to reduce my dataset dimensions, does it exists a special one that guaranty to offer better results with a specific similiarity/dissimilarity measure ?

Does it exist a dimensionality reduction technique that is optimal with a particular distance metrics an which is bounded in $O(n.log(n))$ in time ?

Answer 1

This question will invite a lot of opinions and it will be very specific to the problem that you are looking at. But there's no harm in laying out the popular options -

1. Start with the data

a. Images - If you are working with images and plan to use a classifier like ConvNet, PCA probably isnt a very good idea. You should just resize the image using popular libs like opencv

b. Unstructured Text - I have personally looked at a lot of free text problems, and my goto method is using Cosine Distance after doing a TruncatedSVD (dimensionality reduction) on the bag-of-words data (or any way of structuring for that matter)

c. Structured - Usually PCA is the go-to solution, but only if you are ok with losing the interpretability in your downstream predictive models. Scree Plots are a good way to evaluate how good a PCA exercise was / how many Principal Components you need.

2. Methods

a. Auto-encoders - These are neural-network based dimensionality reduction technique and I haven't looked at a single production instance where in a predictive modelling workflow Auto-encoders outperformed PCA. I guess we just have to wait for the right use-case

b. Distance metrics - Euclidean is no-doubt the most popular one, but Cosine catches up well to free text problems. You also have Hamming distance, Mahalanobis distance and countless others but its really upto you to try that. I haven't come across a case where these were performing better.

While we are talking about distance metrics, also look at DBScan for clustering.

Also, Silhouette Distance is another effective way of measuring how good your clusters did.