What does it mean by “t-SNE retains the structure of the data”?

Question

I was learning about t-SNE when I was told that t-SNE retains the structure of the data in the embeddings.

What exactly does this mean ?

How does the algorithm achieve this ?

So far I have understood this ->

t-SNE is an unsupervised learning algorithm that is used for dimensionality reduction and visualization of high dimentioanal data. The algorithm works by measuring the similarity between one point and all the other points using the t-curve. The width of the curve is dependant on the density of the cluster the point belongs to. t-SNE retains the structure of the initial data.

My question is what does it mean by retaining the structure of the data? Shouldn't there be some loss in the structure of the data seeing as it is transformed into a lower dimensional space ? Also what does "strucutre of the data mean"? Please ask for any further details that are required.

Answer 1

You should break this down one step further: retaining local structure and retaining global structure.

---

• Other well-understood methods, such as Principal Component Analysis are great at retaining global structure, because it looks at ways in which a dataset's variance is retained, globally, across the entire dataset.

• t-SNE works differently, by looking at locally appearing datapoints. It does this by computing a metric between each datapoint and a given number of neighbours - modelling them as being within a t-distributed distribution (hence the name: t-distributed Stochastic Neighbourhood Embedding). It then tries to find an embedding, such that neighbours in the original n-dimensional space, are also found close together in the reduced (embedded) dimensional space. It does this by minimising the KL-divergence between the before and after datapoint distributions, $\mathbb{P}$ and $\mathbb{Q}$ respectively.

This method has the benefit of retaining local structure - so clusters in the low dimensional space should be interpretable as datapoints that were also very similar in the high dimensional space. t-SNE works remarkedly well on many problem, however there are a few things to watch out for:

1. Because we know have some useful local structure retained, we essentially trade that off for ability in retaining global structure. This equates to you not being able to really compare e.g. 3 clusters in the final embedding, where 2 are close together and 1 is far away. This does not mean they were also far away from each other in the original space.

2. t-SNE can be very sensitive to its perplexity parameter. In fact, you might get different results with the three-cluster example in point 1, using an only slightly different perplexity value. This value can indeed be roughly equated to "how many points shall we inlude in the t-distribution to find neighbours of a datapoint" - it essentially gives the area which is encompassed in the t-distribution.

---

I would recommend watching this lecture by the author of t-SNE, Laurens van der Maaten, as well as getting some intuition for t-SNE and it's parameters using this great visual explanation.

There are also some good answers here on CrossValidated with a little more technical information.