I (think I) understand the underlying principles of most dimensionality reduction methods (MDS, IsoMap, t-SNE, Spectral Embedding, Diffusion maps, etc...).

Some of the algorithms I use the most are Kernel PCA (with a gaussian kernel) and t-SNE. My question is, do you know some theoretical reasons when to use t-SNE or kernel PCA? Do you know which are their relative strengths/weaknesses are? Are there some known cases where one is better than the other? Do their results have different characteristics (I have noticed that KPCA tends to preserve the density of points, while t-SNE spreads the points with more or less uniform density)? This type of thing...


1 Answer 1


Kernel PCA

Kernel PCA is one of the variations of principal component analysis in which we use kernel methods to perform principal component analysis with nonlinearly separable datasets. The Kernel approach of Principal Component Analysis is very similar to the standard one but with a different processing step.

Principal Component Analysis

Principal Component Analysis is an unsupervised learning algorithm that is used for dimensionality reduction in machine learning. It is a statistical process that converts the observations of correlated features into a set of linearly uncorrelated features with the help of orthogonal transformation.

Both t-SNE and kernel PCA are popular dimensionality reduction methods that can be used to visualize high-dimensional data in two or three dimensions. However, they have different strengths and weaknesses and are appropriate for different types of data and applications.

One key difference between t-SNE and kernel PCA is how they measure the similarity between data points. t-SNE uses a non-parametric method based on the Kullback-Leibler divergence to measure the similarity between high-dimensional data points, while kernel PCA uses a parametric approach based on the kernel trick to measure similarity in the transformed space. This means that t-SNE is more flexible and can adapt to the structure of the data, but it can also be more computationally expensive and harder to optimize than kernel PCA.

Another important difference is the way they preserve structure in the data. t-SNE is designed to preserve the local structure of the data, such as clusters and manifolds, but it does not preserve global structure such as distances or densities. In contrast, kernel PCA is designed to preserve the global structure of the data, such as distances between points, but it may not preserve local structure as well as t-SNE.

In general, t-SNE is a better choice for data that has complex, non-linear structure, such as images or text, while kernel PCA is a better choice for data that has simpler, more linear structure, such as gene expression data or financial time series. However, there is no one-size-fits-all method, and the best choice of dimensionality reduction method depends on the specific data and application.

There are many other factors that can influence the choice of dimensionality reduction method, such as the size and complexity of the dataset, the computational resources available, and the specific goals of the analysis. Ultimately, the best method to use will depend on the characteristics of the data and the specific goals of the analysis.

It is often useful to try multiple methods and compare the results to see which one works best for a given dataset.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.