How do I interpret low dimentional embeddings of high dimentional embeddings?

Question

I am trying to understand what I am supposed to learn about a problem when using dimensionality reduction methods. In particular, I am referring to methods like t-SNE and UMAP.

For the most part I am told that I should be using these methods to 'discover structure in the data' and I understand what is supposed to be happening is that, in the high dimensional space, there is some lower dimensional manifold that the data occupies and projecting that into lower dimensional space is supposed to help me understand something about the data. But examples of "good" embeddings look like nonsense to me. Furthermore, there is usually a small (3-5) number of hyperparameters that I have little idea what to do with.

For a concrete example, here is a parameter walk-through for UMAP. In the linked section on the parameter n_neighbors the author says

By the stage of n_neighbors=20 we have a fairly good overall view of the data showing how the various colors interelate to each other over the whole dataset.

When I look at that plot, I see different colors splattered all over the place. There are multiple groups of each color. At best, it appears that the more translucent points have collected in the upper right of the plot. I understand that this particular set of images is supposed to illustrate the effect of the n_neighbors parameter on the plot. But so what?

For another example, there is a webpage describing how to effectively use t-SNE. There are a number of great examples in there. But, so far as I can tell, the overall take-away is that you can more-or-less adjust the t-SNE parameters until you get an embedding you like for some reason. Again, so what? To what end?

As far as I can tell, these manifold methods, act as kind of a Rorschach test or a data science version of a cloud shape game. To reiterate my titular question: How do I interpret low dimentional embeddings of high dimentional embeddings? What am I supposed to be seeing in these plots that I am evidently not seeing?

Answer 1

With this kind of low dimensional embeddings, you confirm visually some kind of relationship between the data points that you suspected by intuition.

For instance, the following graph taken from the article From bilingual to multilingual neural-based machine translation by incremental training shows the UMAP 2D plot of the representations computed by a neural network for sentences in different languages. The plot shows that the different languages form clusters that are distant from each other, which makes us think that the neural network is not able to properly learn the same representation space for all language.

Another typical example is with MNIST, the handwritten digit dataset. Here you can see plots of the bottleneck representation of an autoencoder and from PCA from this blog post where each point is the 2D representation of a digit image from the dataset and its color is its label (i.e. which number it is):

In the figure, we can observe that, while PCA hardly separates the different numbers, the autoencoder indeed separates them.